Fix line filters running tests from multiple runnables.

`derive_regexp` was written with the assumption that we were run from a
blank slate — that if the filter didn't match we might as well return it
because it was nil.

This isn't the case because minitest calls `run` on every runnable. Which
is any subclass of Minitest::Runnable, such as ActiveSupport::TestCase,
ActionDispatch::IntegrationTest as well as any inheriting from those.

Thus after the first `run` we'd have put in a composite filter in
`options[:filter]` making the next `run` create a linked list when it
failed to match the regexp and put the composite filter as the head.

Every runnable would accumulate more and more of the same filters,
which effectively acted like an expanding whitelist and we ran tests
from other runnables.

Clog the accumulation by returning nil if there's no filter to derive
a regexp from.

Note: we pass a seed in the tests because Minitest shuffles the runnables
to ensure the whitelist is expanded enough that the failure is triggered.

1 files changed, 24 insertions, 0 deletions

diff --git a/railties/test/application/test_runner_test.rb b/railties/test/application/test_runner_test.rb
index a7eb0feb11..33bc998722 100644
--- a/railties/test/application/test_runner_test.rb
+++ b/railties/test/application/test_runner_test.rb
@@ -382,6 +382,30 @@ module ApplicationTests
       end
     end
 
+    def test_line_filters_trigger_only_one_runnable
+      app_file 'test/models/post_test.rb', <<-RUBY
+        require 'test_helper'
+
+        class PostTest < ActiveSupport::TestCase
+          test 'truth' do
+            assert true
+          end
+        end
+
+        class SecondPostTest < ActiveSupport::TestCase
+          test 'truth' do
+            assert false, 'ran second runnable'
+          end
+        end
+      RUBY
+
+      # Pass seed guaranteeing failure.
+      run_test_command('test/models/post_test.rb:4 --seed 30410').tap do |output|
+        assert_no_match 'ran second runnable', output
+        assert_match '1 runs, 1 assertions', output
+      end
+    end
+
     def test_shows_filtered_backtrace_by_default
       create_backtrace_test