diff options
| author | Cliff Pruitt <cliff.pruitt@cliffpruitt.com> | 2019-07-26 11:02:40 -0400 |
|---|---|---|
| committer | Cliff Pruitt <cliff.pruitt@cliffpruitt.com> | 2019-07-26 12:18:01 -0400 |
| commit | 05633d02d8ac8aa1289c0a01872e13e9b2449cd5 (patch) | |
| tree | 952cb2760ca0877ce37b9449f930850ef67b5469 /actiontext | |
| parent | 369daa530dd9db7bbba79b8c75012e0fa84c9f48 (diff) | |
| download | rails-05633d02d8ac8aa1289c0a01872e13e9b2449cd5.tar.gz rails-05633d02d8ac8aa1289c0a01872e13e9b2449cd5.tar.bz2 rails-05633d02d8ac8aa1289c0a01872e13e9b2449cd5.zip | |
Handle GB18030 strings with invalid characters in transliterate
GB18030 is Unicode compatible and covers all Unicode code points so we can temporarily convert GB18030 strings to UTF-8 to perform the transliteration. After transliterating we want to convert back to GB18030.
In all cases of transcoding, we replace invalid or undefined characters with the default replacement character ("?"). This is in line with the behavior of tidy_bytes which is used on the UTF-8 string before transliterating.
Diffstat (limited to 'actiontext')
0 files changed, 0 insertions, 0 deletions