Keep characters after a truncated UTF-8 sequence (#3690)
A multibyte UTF-8 sequence cut short by a non-continuation byte took every remaining byte of the string, so "a\xf0Zb" became "a\ufffd". Replace only the lead byte and the continuation bytes that follow it, the same as an invalid sequence that is not cut short, so "a\xf0Zb" now becomes "a\ufffdZb". This also changes length and explode on such strings, which now count the characters that used to be dropped.
This commit is contained in:
1 parent
74c46239b9
commit
70c9dc516f
2 files changed
+10
-2
No files matched your search
@@ -89,6 +89,11 @@ map(@base64d)
|
||||
["YWJjZGVmZ2hpav9rbG1ub3A=", "YWJjZGVmZ2hpasOpa2xtbm9w5pel", "/2FiY2RlZmdoaWo=", "YWJjZGVmZ2hpauaX", "YWJjZGVmZ2hpau2ggHh5eg=="]
|
||||
["abcdefghij�klmnop","abcdefghijéklmnop日","�abcdefghij","abcdefghij�","abcdefghij�xyz"]
|
||||
|
||||
# Truncated UTF-8 sequence keeps the characters after it
|
||||
map(@base64d)
|
||||
["8Fo=", "4mE=", "8J9a", "YfBaYg==", "8J+Y"]
|
||||
["�Z","�a","�Z","a�Zb","�"]
|
||||
|
||||
@uri
|
||||
"\u03bc"
|
||||
"%CE%BC"
|
||||
|
||||
Reference in new issue
Block a user