Default to sorting user's test cases for now

Goals:

1. Default to :random for newly generated applications
2. Default to :sorted for existing applications with a warning
3. Only show the warning once
4. Only show the warning if the app actually uses AS::TestCase

Fixes #16769

3 files changed, 25 insertions, 4 deletions

diff --git a/guides/source/4_2_release_notes.md b/guides/source/4_2_release_notes.md
index dad973cf5e..cb461feff8 100644
--- a/guides/source/4_2_release_notes.md
+++ b/guides/source/4_2_release_notes.md
@@ -688,13 +688,14 @@ Please refer to the [Changelog][active-support] for detailed changes.
 
 ### Notable changes
 
+*   Introduced new configuration option `active_support.test_order` for
+    specifying the order test cases are executed. This option currently defaults
+    to `:sorted` but will be changed to `:random` in Rails 5.0.
+    ([Commit](TODO: fill me in))
+
 *   The `travel_to` test helper now truncates the `usec` component to 0.
     ([Commit](https://github.com/rails/rails/commit/9f6e82ee4783e491c20f5244a613fdeb4024beb5))
 
-*   `ActiveSupport::TestCase` now randomizes the order that test cases are ran
-    by default.
-    ([Commit](https://github.com/rails/rails/commit/6ffb29d24e05abbd9ffe3ea974140d6c70221807))
-
 *   Introduced `Object#itself` as an identity function.
     (Commit [1](https://github.com/rails/rails/commit/702ad710b57bef45b081ebf42e6fa70820fdd810),
     [2](https://github.com/rails/rails/commit/64d91122222c11ad3918cc8e2e3ebc4b0a03448a))
diff --git a/guides/source/configuring.md b/guides/source/configuring.md
index dbbd0c1aea..667281d1aa 100644
--- a/guides/source/configuring.md
+++ b/guides/source/configuring.md
@@ -471,6 +471,8 @@ There are a few configuration options available in Active Support:
 
 * `config.active_support.bare` enables or disables the loading of `active_support/all` when booting Rails. Defaults to `nil`, which means `active_support/all` is loaded.
 
+* `config.active_support.test_order` sets the order that test cases are executed. Possible values are `:sorted` and `:random`. Currently defaults to `:sorted`. In Rails 5.0, the default will be changed to `:random` instead.
+
 * `config.active_support.escape_html_entities_in_json` enables or disables the escaping of HTML entities in JSON serialization. Defaults to `false`.
 
 * `config.active_support.use_standard_json_time_format` enables or disables serializing dates to ISO 8601 format. Defaults to `true`.
diff --git a/guides/source/upgrading_ruby_on_rails.md b/guides/source/upgrading_ruby_on_rails.md
index 989d8400e5..71358ad3f5 100644
--- a/guides/source/upgrading_ruby_on_rails.md
+++ b/guides/source/upgrading_ruby_on_rails.md
@@ -65,6 +65,24 @@ TODO: mention https://github.com/rails/rails/pull/16526
 
 TODO: mention https://github.com/rails/rails/pull/16537
 
+### Ordering of test cases
+
+In Rails 5.0, test cases will be executed in random order by default. In
+anticipation of this change, Rails 4.2 introduced a new configuration option
+`active_support.test_order` for explicitly specifying the test ordering. This
+allows you to either locking down the current behavior by setting the option to
+`:sorted`, or opt into the future behavior by setting the option to `:random`.
+
+If you do not specify a value for this option, a deprecation warning will be
+emitted. To avoid this, add the following line to your test environment:
+
+```ruby
+# config/environments/test.rb
+Rails.application.configure do
+  config.active_support.test_order = :sorted # or `:random` if you prefer
+end
+```
+
 ### Serialized attributes
 
 When using a custom coder (e.g. `serialize :metadata, JSON`),