Keep characters after a truncated UTF-8 sequence (#3690)

A multibyte UTF-8 sequence cut short by a non-continuation byte took
every remaining byte of the string, so "a\xf0Zb" became "a\ufffd".

Replace only the lead byte and the continuation bytes that follow it,
the same as an invalid sequence that is not cut short, so "a\xf0Zb"
now becomes "a\ufffdZb". This also changes length and explode on such
strings, which now count the characters that used to be dropped.
This commit is contained in:
skrmc authored and GitHub committed 2026-11-01 00:59:37 +09:00
1 parent 74c46239b9
commit 70c9dc516f
2 files changed
+10 -2

No files matched your search

+5
View File
@@ -89,6 +89,11 @@ map(@base64d)
["YWJjZGVmZ2hpav9rbG1ub3A=", "YWJjZGVmZ2hpasOpa2xtbm9w5pel", "/2FiY2RlZmdoaWo=", "YWJjZGVmZ2hpauaX", "YWJjZGVmZ2hpau2ggHh5eg=="]
["abcdefghij�klmnop","abcdefghijéklmnop日","�abcdefghij","abcdefghij�","abcdefghij�xyz"]
# Truncated UTF-8 sequence keeps the characters after it
map(@base64d)
["8Fo=", "4mE=", "8J9a", "YfBaYg==", "8J+Y"]
["�Z","�a","�Z","a�Zb","�"]
@uri
"\u03bc"
"%CE%BC"