| Commit message (Collapse) | Author | Age | Files | Lines |
|
|
|
|
|
| |
GB18030 is Unicode compatible and covers all Unicode code points so we can temporarily convert GB18030 strings to UTF-8 to perform the transliteration. After transliterating we want to convert back to GB18030.
In all cases of transcoding, we replace invalid or undefined characters with the default replacement character ("?"). This is in line with the behavior of tidy_bytes which is used on the UTF-8 string before transliterating.
|
|
|
|
| |
US-ASCII is a subset of UTF-8 so we can temporarily convert US-ASCII strings to UTF-8 to perform the transliteration. After we've converted characters to ASCII representations, we can set the encoding back to US-ASCII to return the same encoding we accepted.
|
|
|
|
| |
Adds ArgumentErrors to `ActiveSupport::Inflector::transliterate` if a string is with ASCII-8BIT which will raise an error in `unicode_normalize`.
|
|
|
|
| |
`ActiveSupport::Inflector.transliterate`"
|
|
|
|
|
|
| |
It's noted in #34062 that String#parameterize will raise an `Encoding::CompatibilityError` if the string is not UTF-8 encoded. The error is raised as a result of passing the string to `.unicode_normalize`.
This PR raises a higher level `ArgumentError` if the provided string is not UTF-8 and updates documentation to note the encoding requirement.
|
| |
|
|
|
|
| |
Also add tests for parametrize and transliterate
|
|
|
|
|
|
| |
Parameterize is triggering I18n#transliterate. This method already
accepts a locale. It would be cleaner if similar to other string inflection
methods #parameterize also accepted 'locale' as a parameter.
|
| |
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
Since Rails 6.0 will support Ruby 2.4.1 or higher
`# frozen_string_literal: true` magic comment is enough to make string object frozen.
This magic comment is enabled by `Style/FrozenStringLiteralComment` cop.
* Exclude these files not to auto correct false positive `Regexp#freeze`
- 'actionpack/lib/action_dispatch/journey/router/utils.rb'
- 'activerecord/lib/active_record/connection_adapters/sqlite3_adapter.rb'
It has been fixed by https://github.com/rubocop-hq/rubocop/pull/6333
Once the newer version of RuboCop released and available at Code Climate these exclude entries should be removed.
* Replace `String#freeze` with `String#-@` manually if explicit frozen string objects are required
- 'actionpack/test/controller/test_case_test.rb'
- 'activemodel/test/cases/type/string_test.rb'
- 'activesupport/lib/active_support/core_ext/string/strip.rb'
- 'activesupport/test/core_ext/string_ext_test.rb'
- 'railties/test/generators/actions_test.rb'
|
|
|
| |
[ci skip]
|
|
|
|
| |
Trés -> Très
https://fr.wiktionary.org/wiki/tr%C3%A8s
|
| |
|
|
|
|
| |
This basically reverts 8da30ad6be34339124ba4cb4e36aea260dda12bc
|
|
|
|
|
|
|
| |
We have some indentation cops. But now there is a little inconsistent
params indentations. Enable `Layout/FirstParameterIndentation` cop to
prevent newly inconsistent indentation added and auto-correct to
existing violations.
|
| |
|
| |
|
| |
|
| |
|
| |
|
| |
|
| |
|
| |
|
|
|
|
|
| |
The current code base is not uniform. After some discussion,
we have chosen to go with double quotes by default.
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
Added test cases
Using kwargs instead of three seperate functions
Updated parameterize in transliterate.rb
Updated parameterize in transliterate.rb
Added deprecation warnings and updating RDoc+Guide
Misspelled separtor. Fixed.
Deprecated test cases and added support to parameterize with keyword parameters
Squashing commits.
Fixed test cases and added deprecated test cases
Small changes to Gemfile.lock and CHANGELOG
Update Gemfile.lock
|
|
|
|
|
|
|
|
|
| |
Saw this while doing a review of a patch:
* Normalize case and punctuation across comments.
* ascii -> ASCII
* Since I was on it, some blank lines that visually
add some clarity IMO.
|
| |
|
| |
|
|
|
|
|
|
| |
We can save a few objects by freezing the `replacement` string. We save a few more by down-casing the string in memory instead of allocating a new one. We save far more objects by checking for the default separator `"-"`, and using pre-generated regular expressions.
We will save 209,231 bytes and 1,322 objects.
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
I wrote a utility that helps find areas where you could optimize your program using a frozen string instead of a string literal, it's called [let_it_go](https://github.com/schneems/let_it_go). After going through the output and adding `.freeze` I was able to eliminate the creation of 1,114 string objects on EVERY request to [codetriage](codetriage.com). How does this impact execution?
To look at memory:
```ruby
require 'get_process_mem'
mem = GetProcessMem.new
GC.start
GC.disable
1_114.times { " " }
before = mem.mb
after = mem.mb
GC.enable
puts "Diff: #{after - before} mb"
```
Creating 1,114 string objects results in `Diff: 0.03125 mb` of RAM allocated on every request. Or 1mb every 32 requests.
To look at raw speed:
```ruby
require 'benchmark/ips'
number_of_objects_reduced = 1_114
Benchmark.ips do |x|
x.report("freeze") { number_of_objects_reduced.times { " ".freeze } }
x.report("no-freeze") { number_of_objects_reduced.times { " " } }
end
```
We get the results
```
Calculating -------------------------------------
freeze 1.428k i/100ms
no-freeze 609.000 i/100ms
-------------------------------------------------
freeze 14.363k (± 8.5%) i/s - 71.400k
no-freeze 6.084k (± 8.1%) i/s - 30.450k
```
Now we can do some maths:
```ruby
ips = 6_226k # iterations / 1 second
call_time_before = 1.0 / ips # seconds per iteration
ips = 15_254 # iterations / 1 second
call_time_after = 1.0 / ips # seconds per iteration
diff = call_time_before - call_time_after
number_of_objects_reduced * diff * 100
# => 0.4530373333993266 miliseconds saved per request
```
So we're shaving off 1 second of execution time for every 220 requests.
Is this going to be an insane speed boost to any Rails app: nope. Should we merge it: yep.
p.s. If you know of a method call that doesn't modify a string input such as [String#gsub](https://github.com/schneems/let_it_go/blob/b0e2da69f0cca87ab581022baa43291cdf48638c/lib/let_it_go/core_ext/string.rb#L37) please [give me a pull request to the appropriate file](https://github.com/schneems/let_it_go/blob/b0e2da69f0cca87ab581022baa43291cdf48638c/lib/let_it_go/core_ext/string.rb#L37), or open an issue in LetItGo so we can track and freeze more strings.
Keep those strings Frozen
![](https://www.dropbox.com/s/z4dj9fdsv213r4v/let-it-go.gif?dl=1)
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
Many methods in ActiveSupport::Inflector were actually documenting
the String methods with the same name.
For instance the doc for `camelize` said:
> If the argument to +camelize+ is set to <tt>:lower</tt>...
while it should say:
> If the +uppercase_first_letter+ parameter is set to false
[ci skip]
|
| |
|
| |
|
|
|
|
|
|
|
| |
This reverts commit e8feaff60b9c04d34ad234f7d17b5d2ad9cc7a24, reversing
changes made to 9adf28c026070afb78b80027521a4ddddd68d697.
Reason: This broke the actionmailer tests
|
|
|
|
|
| |
- access & filters don't use multibyte ext
- transliterate requires only AS::Multibyte but not multibyte ext
|
| |
|
| |
|
|
|
|
|
|
|
|
|
|
| |
Makes String#mb_chars on Ruby 1.9 return an instance of ActiveSupport::Multibyte::Chars to work around 1.9's lack of Unicode case folding.
Refactors class methods from ActiveSupport::Multibyte::Chars into new Unicode module, adding other related functionality for consistency.
[#4594 state:resolved]
Signed-off-by: Jeremy Kemper <jeremy@bitsweat.net>
|
| |
|
|
|
|
| |
Signed-off-by: Xavier Noria <fxn@hashref.com>
|
|
|
|
|
|
|
| |
Ancillary changes: Moved Chars#normalize into a class method; removed
unused UTF_PAT constant.
Signed-off-by: José Valim <jose.valim@gmail.com>
|
|
|
|
| |
Signed-off-by: Jeremy Kemper <jeremy@bitsweat.net>
|
|
|
|
| |
Signed-off-by: Jeremy Kemper <jeremy@bitsweat.net>
|
| |
|
|
|
|
|
|
|
|
| |
a + sign is interpreted by the browser as a space, possibly resulting in a "ArgumentError: illegal character in key"
[#4080 state:committed]
Signed-off-by: Jeremy Kemper <jeremy@bitsweat.net>
|
|
like constantize.
|