Add credentials using a generic EncryptedConfiguration class (#30067)

* WIP: Add credentials using a generic EncryptedConfiguration class

This is sketch code so far.

* Flesh out EncryptedConfiguration and test it

* Better name

* Add command and generator for credentials

* Use the Pathnames

* Extract EncryptedFile from EncryptedConfiguration and add serializers

* Test EncryptedFile

* Extract serializer validation

* Stress the point about losing comments

* Allow encrypted configuration to be read without parsing for display

* Use credentials by default and base them on the master key

* Derive secret_key_base in test/dev, source it from credentials in other envs

And document the usage.

* Document the new credentials setup

* Stop generating the secrets.yml file now that we have credentials

* Document what we should have instead

Still need to make it happen, tho.

* [ci skip] Keep wording to `key base`; prefer defaults.

Usually we say we change defaults, not "spec" out a release.

Can't use backticks in our sdoc generated documentation either.

* Abstract away OpenSSL; prefer MessageEncryptor.

* Spare needless new when raising.

* Encrypted file test shouldn't depend on subclass.

* [ci skip] Some woordings.

* Ditch serializer future coding.

* I said flip it. Flip it good.

* [ci skip] Move require_master_key to the real production.rb.

* Add require_master_key to abort the boot process.

In case the master key is required in a certain environment
we should inspect that the key is there and abort if it isn't.

* Print missing key message and exit immediately.

Spares us a lengthy backtrace and prevents further execution.

I've verified the behavior in a test app, but couldn't figure the
test out as loading the app just exits immediately with:

```
/Users/kasperhansen/Documents/code/rails/activesupport/lib/active_support/testing/isolation.rb:23:in `load': marshal data too short (ArgumentError)
	from /Users/kasperhansen/Documents/code/rails/activesupport/lib/active_support/testing/isolation.rb:23:in `run'
	from /Users/kasperhansen/.rbenv/versions/2.4.1/lib/ruby/gems/2.4.0/gems/minitest-5.10.2/lib/minitest.rb:830:in `run_one_method'
	from /Users/kasperhansen/.rbenv/versions/2.4.1/lib/ruby/gems/2.4.0/gems/minitest-5.10.2/lib/minitest/parallel.rb:32:in `block (2 levels) in start'
```

It's likely we need to capture and prevent the exit somehow.
Kernel.stub(:exit) didn't work. Leaving it for tomorrow.

* Fix require_master_key config test.

Loading the app would trigger the `exit 1` per require_master_key's
semantics, which then aborted the test.

Fork and wait for the child process to finish, then inspect the
exit status.

Also check we aborted because of a missing master key, so something
else didn't just abort the boot.

Much <3 to @tenderlove for the tip.

* Support reading/writing configs via methods.

* Skip needless deep symbolizing.

* Remove save; test config reader elsewhere.

* Move secret_key_base check to when we're reading it.

Otherwise we'll abort too soon since we don't assign the secret_key_base
to secrets anymore.

* Add missing string literal comments; require unneeded yaml require.

* ya ya ya, rubocop.

* Add master_key/credentials after bundle.

Then we can reuse the existing message on `rails new bc4`.

It'll look like:

```
Using web-console 3.5.1 from https://github.com/rails/web-console.git (at master@ce985eb)
Using rails 5.2.0.alpha from source at `/Users/kasperhansen/Documents/code/rails`
Using sass-rails 5.0.6
Bundle complete! 16 Gemfile dependencies, 72 gems now installed.
Use `bundle info [gemname]` to see where a bundled gem is installed.
Adding config/master.key to store the master encryption key: 97070158c44b4675b876373a6bc9d5a0

Save this in a password manager your team can access.

If you lose the key, no one, including you, can access anything encrypted with it.

      create  config/master.key
```

And that'll be executed even if `--skip-bundle` was passed.

* Ensure test app has secret_key_base.

* Assign secret_key_base to app or omit.

* Merge noise

* Split options for dynamic delegation into its own method and use deep symbols to make it work

* Update error to point to credentials instead

* Appease Rubocop

* Validate secret_key_base when reading it.

Instead of relying on the validation in key_generator move that into
secret_key_base itself.

* Fix generator and secrets test.

Manually add config.read_encrypted_secrets since it's not there by default
anymore.

Move mentions of config/secrets.yml to config/credentials.yml.enc.

* Remove files I have no idea how they got here.

* [ci skip] swap secrets for credentials.

* [ci skip] And now, changelogs are coming.

8 files changed, 293 insertions, 22 deletions

diff --git a/railties/lib/rails/application.rb b/railties/lib/rails/application.rb
index 72f8bf0e14..6ce8b0b2d9 100644
--- a/railties/lib/rails/application.rb
+++ b/railties/lib/rails/application.rb
@@ -5,6 +5,7 @@ require "active_support/core_ext/hash/keys"
 require "active_support/core_ext/object/blank"
 require "active_support/key_generator"
 require "active_support/message_verifier"
+require "active_support/encrypted_configuration"
 require_relative "engine"
 require_relative "secrets"
 
@@ -171,12 +172,9 @@ module Rails
       # number of iterations selected based on consultation with the google security
       # team. Details at https://github.com/rails/rails/pull/6952#issuecomment-7661220
       @caching_key_generator ||=
-        if secrets.secret_key_base
-          unless secrets.secret_key_base.kind_of?(String)
-            raise ArgumentError, "`secret_key_base` for #{Rails.env} environment must be a type of String, change this value in `config/secrets.yml`"
-          end
-          key_generator = ActiveSupport::KeyGenerator.new(secrets.secret_key_base, iterations: 1000)
-          ActiveSupport::CachingKeyGenerator.new(key_generator)
+        if secret_key_base
+          ActiveSupport::CachingKeyGenerator.new \
+            ActiveSupport::KeyGenerator.new(secret_key_base, iterations: 1000)
         else
           ActiveSupport::LegacyKeyGenerator.new(secrets.secret_token)
         end
@@ -246,13 +244,11 @@ module Rails
     # will be used by middlewares and engines to configure themselves.
     def env_config
       @app_env_config ||= begin
-        validate_secret_key_config!
-
         super.merge(
           "action_dispatch.parameter_filter" => config.filter_parameters,
           "action_dispatch.redirect_filter" => config.filter_redirect,
           "action_dispatch.secret_token" => secrets.secret_token,
-          "action_dispatch.secret_key_base" => secrets.secret_key_base,
+          "action_dispatch.secret_key_base" => secret_key_base,
           "action_dispatch.show_exceptions" => config.action_dispatch.show_exceptions,
           "action_dispatch.show_detailed_exceptions" => config.consider_all_requests_local,
           "action_dispatch.logger" => Rails.logger,
@@ -406,6 +402,33 @@ module Rails
       @secrets = secrets
     end
 
+    # The secret_key_base is used as the input secret to the application's key generator, which in turn
+    # is used to create all the MessageVerfiers, including the one that signs and encrypts cookies.
+    #
+    # In test and development, this is simply derived as a MD5 hash of the application's name.
+    #
+    # In all other environments, we look for it first in ENV["SECRET_KEY_BASE"],
+    # then credentials[:secret_key_base], and finally secrets.secret_key_base. For most applications,
+    # the correct place to store it is in the encrypted credentials file.
+    def secret_key_base
+      if Rails.env.test? || Rails.env.development?
+        Digest::MD5.hexdigest self.class.name
+      else
+        validate_secret_key_base \
+          ENV["SECRET_KEY_BASE"] || credentials.secret_key_base || secrets.secret_key_base
+      end
+    end
+
+    # Decrypts the credentials hash as kept in `config/credentials.yml.enc`. This file is encrypted with
+    # the Rails master key, which is either taken from ENV["RAILS_MASTER_KEY"] or from loading
+    # `config/master.key`.
+    def credentials
+      @credentials ||= ActiveSupport::EncryptedConfiguration.new \
+        config_path: Rails.root.join("config/credentials.yml.enc"),
+        key_path: Rails.root.join("config/master.key"),
+        env_key: "RAILS_MASTER_KEY"
+    end
+
     def to_app #:nodoc:
       self
     end
@@ -504,14 +527,13 @@ module Rails
       default_stack.build_stack
     end
 
-    def validate_secret_key_config! #:nodoc:
-      if secrets.secret_key_base.blank?
-        ActiveSupport::Deprecation.warn "You didn't set `secret_key_base`. " \
-          "Read the upgrade documentation to learn more about this new config option."
-
-        if secrets.secret_token.blank?
-          raise "Missing `secret_key_base` for '#{Rails.env}' environment, set this value in `config/secrets.yml`"
-        end
+    def validate_secret_key_base(secret_key_base)
+      if secret_key_base.is_a?(String) && secret_key_base.present?
+        secret_key_base
+      elsif secret_key_base
+        raise ArgumentError, "`secret_key_base` for #{Rails.env} environment must be a type of String`"
+      elsif secrets.secret_token.blank?
+        raise ArgumentError, "Missing `secret_key_base` for '#{Rails.env}' environment, set this string with `rails credentials:edit`"
       end
     end
 
diff --git a/railties/lib/rails/commands/credentials/USAGE b/railties/lib/rails/commands/credentials/USAGE
new file mode 100644
index 0000000000..5bd9f940fd
--- /dev/null
+++ b/railties/lib/rails/commands/credentials/USAGE
@@ -0,0 +1,40 @@
+=== Storing Encrypted Credentials in Source Control
+
+The Rails `credentials` commands provide access to encrypted credentials,
+so you can safely store access tokens, database passwords, and the like
+safely inside the app without relying on a mess of ENVs.
+
+This also allows for atomic deploys: no need to coordinate key changes
+to get everything working as the keys are shipped with the code.
+
+=== Setup
+
+Applications after Rails 5.2 automatically have a basic credentials file generated
+that just contains the secret_key_base used by the MessageVerifiers, like the one
+signing and encrypting cookies.
+
+For applications created prior to Rails 5.2, we'll automatically generate a new
+credentials file in `config/credentials.yml.enc` the first time you run `bin/rails credentials:edit`.
+If you didn't have a master key saved in `config/master.key`, that'll be created too.
+
+Don't lose this master key! Put it in a password manager your team can access.
+Should you lose it no one, including you, will be able to access any encrypted
+credentials.
+
+Don't commit the key! Add `config/master.key` to your source control's
+ignore file. If you use Git, Rails handles this for you.
+
+Rails also looks for the master key in `ENV["RAILS_MASTER_KEY"]`, if that's easier to manage.
+
+You could prepend that to your server's start command like this:
+
+   RAILS_MASTER_KEY="very-secret-and-secure" server.start
+
+=== Editing Credentials
+
+This will open a temporary file in `$EDITOR` with the decrypted contents to edit
+the encrypted credentials.
+
+When the temporary file is next saved the contents are encrypted and written to
+`config/credentials.yml.enc` while the file itself is destroyed to prevent credentials
+from leaking.
diff --git a/railties/lib/rails/commands/credentials/credentials_command.rb b/railties/lib/rails/commands/credentials/credentials_command.rb
new file mode 100644
index 0000000000..39a4e3c833
--- /dev/null
+++ b/railties/lib/rails/commands/credentials/credentials_command.rb
@@ -0,0 +1,85 @@
+# frozen_string_literal: true
+
+require "active_support"
+
+module Rails
+  module Command
+    class CredentialsCommand < Rails::Command::Base # :nodoc:
+      no_commands do
+        def help
+          say "Usage:\n  #{self.class.banner}"
+          say ""
+          say self.class.desc
+        end
+      end
+
+      def edit
+        require_application_and_environment!
+
+        ensure_editor_available || (return)
+        ensure_master_key_has_been_added
+        ensure_credentials_have_been_added
+
+        change_credentials_in_system_editor
+
+        say "New credentials encrypted and saved."
+      rescue Interrupt
+        say "Aborted changing credentials: nothing saved."
+      rescue ActiveSupport::EncryptedFile::MissingKeyError => error
+        say error.message
+      end
+
+      def show
+        require_application_and_environment!
+        say Rails.application.credentials.read.presence ||
+          "No credentials have been added yet. Use bin/rails credentials:edit to change that."
+      end
+
+      private
+        def ensure_editor_available
+          if ENV["EDITOR"].to_s.empty?
+            say "No $EDITOR to open credentials in. Assign one like this:"
+            say ""
+            say %(EDITOR="mate --wait" bin/rails credentials:edit)
+            say ""
+            say "For editors that fork and exit immediately, it's important to pass a wait flag,"
+            say "otherwise the credentials will be saved immediately with no chance to edit."
+
+            false
+          else
+            true
+          end
+        end
+
+        def ensure_master_key_has_been_added
+          master_key_generator.add_master_key_file
+          master_key_generator.ignore_master_key_file
+        end
+
+        def ensure_credentials_have_been_added
+          credentials_generator.add_credentials_file_silently
+        end
+
+        def change_credentials_in_system_editor
+          Rails.application.credentials.change do |tmp_path|
+            system("#{ENV["EDITOR"]} #{tmp_path}")
+          end
+        end
+
+
+        def master_key_generator
+          require_relative "../../generators"
+          require_relative "../../generators/rails/master_key/master_key_generator"
+
+          Rails::Generators::MasterKeyGenerator.new
+        end
+
+        def credentials_generator
+          require_relative "../../generators"
+          require_relative "../../generators/rails/credentials/credentials_generator"
+
+          Rails::Generators::CredentialsGenerator.new
+        end
+    end
+  end
+end
diff --git a/railties/lib/rails/generators/rails/app/app_generator.rb b/railties/lib/rails/generators/rails/app/app_generator.rb
index 0f73cc4755..c67baa5e91 100644
--- a/railties/lib/rails/generators/rails/app/app_generator.rb
+++ b/railties/lib/rails/generators/rails/app/app_generator.rb
@@ -111,7 +111,6 @@ module Rails
         template "routes.rb"
         template "application.rb"
         template "environment.rb"
-        template "secrets.yml"
         template "cable.yml" unless options[:skip_action_cable]
         template "puma.rb"   unless options[:skip_puma]
         template "spring.rb" if spring_install?
@@ -159,6 +158,22 @@ module Rails
       end
     end
 
+    def master_key
+      require_relative "../master_key/master_key_generator"
+
+      after_bundle do
+        Rails::Generators::MasterKeyGenerator.new.add_master_key_file
+      end
+    end
+
+    def credentials
+      require_relative "../credentials/credentials_generator"
+
+      after_bundle do
+        Rails::Generators::CredentialsGenerator.new.add_credentials_file_silently
+      end
+    end
+
     def database_yml
       template "config/databases/#{options[:database]}.yml", "config/database.yml"
     end
@@ -289,6 +304,14 @@ module Rails
       end
       remove_task :update_config_files
 
+      def create_master_key
+        build(:master_key)
+      end
+
+      def create_credentials
+        build(:credentials)
+      end
+
       def display_upgrade_guide_info
         say "\nAfter this, check Rails upgrade guide at http://guides.rubyonrails.org/upgrading_ruby_on_rails.html for more details about upgrading your app."
       end
diff --git a/railties/lib/rails/generators/rails/app/templates/config/environments/production.rb.tt b/railties/lib/rails/generators/rails/app/templates/config/environments/production.rb.tt
index f68e13aa8b..2e0b555f6f 100644
--- a/railties/lib/rails/generators/rails/app/templates/config/environments/production.rb.tt
+++ b/railties/lib/rails/generators/rails/app/templates/config/environments/production.rb.tt
@@ -14,10 +14,9 @@ Rails.application.configure do
   config.consider_all_requests_local       = false
   config.action_controller.perform_caching = true
 
-  # Attempt to read encrypted secrets from `config/secrets.yml.enc`.
-  # Requires an encryption key in `ENV["RAILS_MASTER_KEY"]` or
-  # `config/secrets.yml.key`.
-  config.read_encrypted_secrets = true
+  # Ensures that a master key has been made available in either ENV["RAILS_MASTER_KEY"]
+  # or in config/master.key. This key is used to decrypt credentials (and other encrypted files).
+  # config.require_master_key = true
 
   # Disable serving static files from the `/public` folder by default since
   # Apache or NGINX already handles this.
diff --git a/railties/lib/rails/generators/rails/app/templates/gitignore b/railties/lib/rails/generators/rails/app/templates/gitignore
index 83a7b211aa..c37f01a848 100644
--- a/railties/lib/rails/generators/rails/app/templates/gitignore
+++ b/railties/lib/rails/generators/rails/app/templates/gitignore
@@ -7,6 +7,9 @@
 # Ignore bundler config.
 /.bundle
 
+# Ignore master key for decrypting credentials and more.
+/config/master.key
+
 <% if sqlite3? -%>
 # Ignore the default SQLite database.
 /db/*.sqlite3
diff --git a/railties/lib/rails/generators/rails/credentials/credentials_generator.rb b/railties/lib/rails/generators/rails/credentials/credentials_generator.rb
new file mode 100644
index 0000000000..ddcccd5ce5
--- /dev/null
+++ b/railties/lib/rails/generators/rails/credentials/credentials_generator.rb
@@ -0,0 +1,46 @@
+# frozen_string_literal: true
+
+require_relative "../../base"
+require_relative "../master_key/master_key_generator"
+require "active_support/encrypted_configuration"
+
+module Rails
+  module Generators
+    class CredentialsGenerator < Base
+      CONFIG_PATH = "config/credentials.yml.enc"
+      KEY_PATH    = "config/master.key"
+
+      def add_credentials_file
+        unless File.exist?(CONFIG_PATH)
+          template = credentials_template
+
+          say "Adding #{CONFIG_PATH} to store encrypted credentials."
+          say ""
+          say "The following content has been encrypted with the Rails master key:"
+          say ""
+          say template, :on_green
+          say ""
+
+          add_credentials_file_silently(template)
+
+          say "You can edit encrypted credentials with `bin/rails credentials:edit`."
+          say ""
+        end
+      end
+
+      def add_credentials_file_silently(template = nil)
+        unless File.exist?(CONFIG_PATH)
+          setup = { config_path: CONFIG_PATH, key_path: KEY_PATH, env_key: "RAILS_MASTER_KEY" }
+          ActiveSupport::EncryptedConfiguration.new(setup).write(credentials_template)
+        end
+      end
+
+      private
+        def credentials_template
+          "# amazon:\n#  access_key_id: 123\n#  secret_access_key: 345\n\n" +
+          "# Used as the base secret for all MessageVerifiers in Rails, including the one protecting cookies.\n" +
+          "secret_key_base: #{SecureRandom.hex(64)}"
+        end
+    end
+  end
+end
diff --git a/railties/lib/rails/generators/rails/master_key/master_key_generator.rb b/railties/lib/rails/generators/rails/master_key/master_key_generator.rb
new file mode 100644
index 0000000000..36a0b69e76
--- /dev/null
+++ b/railties/lib/rails/generators/rails/master_key/master_key_generator.rb
@@ -0,0 +1,53 @@
+# frozen_string_literal: true
+
+require_relative "../../base"
+require "pathname"
+require "active_support/encrypted_file"
+
+module Rails
+  module Generators
+    class MasterKeyGenerator < Base
+      MASTER_KEY_PATH = Pathname.new("config/master.key")
+
+      def add_master_key_file
+        unless MASTER_KEY_PATH.exist?
+          key = ActiveSupport::EncryptedFile.generate_key
+
+          say "Adding #{MASTER_KEY_PATH} to store the master encryption key: #{key}"
+          say ""
+          say "Save this in a password manager your team can access."
+          say ""
+          say "If you lose the key, no one, including you, can access anything encrypted with it."
+
+          say ""
+          add_master_key_file_silently key
+          say ""
+        end
+      end
+
+      def add_master_key_file_silently(key = nil)
+        create_file MASTER_KEY_PATH, key || ActiveSupport::EncryptedFile.generate_key
+      end
+
+      def ignore_master_key_file
+        if File.exist?(".gitignore")
+          unless File.read(".gitignore").include?(key_ignore)
+            say "Ignoring #{MASTER_KEY_PATH} so it won't end up in Git history:"
+            say ""
+            append_to_file ".gitignore", key_ignore
+            say ""
+          end
+        else
+          say "IMPORTANT: Don't commit #{MASTER_KEY_PATH}. Add this to your ignore file:"
+          say key_ignore, :on_green
+          say ""
+        end
+      end
+
+      private
+        def key_ignore
+          [ "", "# Ignore master key for decrypting credentials and more.", MASTER_KEY_PATH, "" ].join("\n")
+        end
+    end
+  end
+end