Add credentials using a generic EncryptedConfiguration class (#30067)

* WIP: Add credentials using a generic EncryptedConfiguration class

This is sketch code so far.

* Flesh out EncryptedConfiguration and test it

* Better name

* Add command and generator for credentials

* Use the Pathnames

* Extract EncryptedFile from EncryptedConfiguration and add serializers

* Test EncryptedFile

* Extract serializer validation

* Stress the point about losing comments

* Allow encrypted configuration to be read without parsing for display

* Use credentials by default and base them on the master key

* Derive secret_key_base in test/dev, source it from credentials in other envs

And document the usage.

* Document the new credentials setup

* Stop generating the secrets.yml file now that we have credentials

* Document what we should have instead

Still need to make it happen, tho.

* [ci skip] Keep wording to `key base`; prefer defaults.

Usually we say we change defaults, not "spec" out a release.

Can't use backticks in our sdoc generated documentation either.

* Abstract away OpenSSL; prefer MessageEncryptor.

* Spare needless new when raising.

* Encrypted file test shouldn't depend on subclass.

* [ci skip] Some woordings.

* Ditch serializer future coding.

* I said flip it. Flip it good.

* [ci skip] Move require_master_key to the real production.rb.

* Add require_master_key to abort the boot process.

In case the master key is required in a certain environment
we should inspect that the key is there and abort if it isn't.

* Print missing key message and exit immediately.

Spares us a lengthy backtrace and prevents further execution.

I've verified the behavior in a test app, but couldn't figure the
test out as loading the app just exits immediately with:

```
/Users/kasperhansen/Documents/code/rails/activesupport/lib/active_support/testing/isolation.rb:23:in `load': marshal data too short (ArgumentError)
	from /Users/kasperhansen/Documents/code/rails/activesupport/lib/active_support/testing/isolation.rb:23:in `run'
	from /Users/kasperhansen/.rbenv/versions/2.4.1/lib/ruby/gems/2.4.0/gems/minitest-5.10.2/lib/minitest.rb:830:in `run_one_method'
	from /Users/kasperhansen/.rbenv/versions/2.4.1/lib/ruby/gems/2.4.0/gems/minitest-5.10.2/lib/minitest/parallel.rb:32:in `block (2 levels) in start'
```

It's likely we need to capture and prevent the exit somehow.
Kernel.stub(:exit) didn't work. Leaving it for tomorrow.

* Fix require_master_key config test.

Loading the app would trigger the `exit 1` per require_master_key's
semantics, which then aborted the test.

Fork and wait for the child process to finish, then inspect the
exit status.

Also check we aborted because of a missing master key, so something
else didn't just abort the boot.

Much <3 to @tenderlove for the tip.

* Support reading/writing configs via methods.

* Skip needless deep symbolizing.

* Remove save; test config reader elsewhere.

* Move secret_key_base check to when we're reading it.

Otherwise we'll abort too soon since we don't assign the secret_key_base
to secrets anymore.

* Add missing string literal comments; require unneeded yaml require.

* ya ya ya, rubocop.

* Add master_key/credentials after bundle.

Then we can reuse the existing message on `rails new bc4`.

It'll look like:

```
Using web-console 3.5.1 from https://github.com/rails/web-console.git (at master@ce985eb)
Using rails 5.2.0.alpha from source at `/Users/kasperhansen/Documents/code/rails`
Using sass-rails 5.0.6
Bundle complete! 16 Gemfile dependencies, 72 gems now installed.
Use `bundle info [gemname]` to see where a bundled gem is installed.
Adding config/master.key to store the master encryption key: 97070158c44b4675b876373a6bc9d5a0

Save this in a password manager your team can access.

If you lose the key, no one, including you, can access anything encrypted with it.

      create  config/master.key
```

And that'll be executed even if `--skip-bundle` was passed.

* Ensure test app has secret_key_base.

* Assign secret_key_base to app or omit.

* Merge noise

* Split options for dynamic delegation into its own method and use deep symbols to make it work

* Update error to point to credentials instead

* Appease Rubocop

* Validate secret_key_base when reading it.

Instead of relying on the validation in key_generator move that into
secret_key_base itself.

* Fix generator and secrets test.

Manually add config.read_encrypted_secrets since it's not there by default
anymore.

Move mentions of config/secrets.yml to config/credentials.yml.enc.

* Remove files I have no idea how they got here.

* [ci skip] swap secrets for credentials.

* [ci skip] And now, changelogs are coming.

6 files changed, 288 insertions, 0 deletions

diff --git a/activesupport/CHANGELOG.md b/activesupport/CHANGELOG.md
index ac1043df78..f158d5357d 100644
--- a/activesupport/CHANGELOG.md
+++ b/activesupport/CHANGELOG.md
@@ -1,3 +1,22 @@
+*   Add `config/credentials.yml.enc` to store production app secrets.
+
+    Allows saving any authentication credentials for third party services
+    directly in repo encrypted with `config/master.key` or `ENV["RAILS_MASTER_KEY"]`.
+
+    This will eventually replace `Rails.application.secrets` and the encrypted
+    secrets introduced in Rails 5.1.
+
+    *DHH*, *Kasper Timm Hansen*
+
+*   Add `ActiveSupport::EncryptedFile` and `ActiveSupport::EncryptedConfiguration`.
+
+    Allows for stashing encrypted files or configuration directly in repo by
+    encrypting it with a key.
+
+    Backs the new credentials setup above, but can also be used independently.
+
+    *DHH*, *Kasper Timm Hansen*
+
 *   `Module#delegate_missing_to` now raises `DelegationError` if target is nil,
     similar to `Module#delegate`.
 
diff --git a/activesupport/lib/active_support/encrypted_configuration.rb b/activesupport/lib/active_support/encrypted_configuration.rb
new file mode 100644
index 0000000000..b403048627
--- /dev/null
+++ b/activesupport/lib/active_support/encrypted_configuration.rb
@@ -0,0 +1,42 @@
+# frozen_string_literal: true
+
+require "yaml"
+require "active_support/encrypted_file"
+require "active_support/ordered_options"
+require "active_support/core_ext/object/inclusion"
+require "active_support/core_ext/module/delegation"
+
+module ActiveSupport
+  class EncryptedConfiguration < EncryptedFile
+    delegate :[], :fetch, to: :config
+    delegate_missing_to :options
+
+    def initialize(config_path:, key_path:, env_key:)
+      super content_path: config_path, key_path: key_path, env_key: env_key
+    end
+
+    # Allow a config to be started without a file present
+    def read
+      super
+    rescue ActiveSupport::EncryptedFile::MissingContentError
+      ""
+    end
+
+    def config
+      @config ||= deserialize(read).deep_symbolize_keys
+    end
+
+    private
+      def options
+        @options ||= ActiveSupport::InheritableOptions.new(config)
+      end
+
+      def serialize(config)
+        config.present? ? YAML.dump(config) : ""
+      end
+
+      def deserialize(config)
+        config.present? ? YAML.load(config) : {}
+      end
+  end
+end
diff --git a/activesupport/lib/active_support/encrypted_file.rb b/activesupport/lib/active_support/encrypted_file.rb
new file mode 100644
index 0000000000..1c4c1cb457
--- /dev/null
+++ b/activesupport/lib/active_support/encrypted_file.rb
@@ -0,0 +1,101 @@
+# frozen_string_literal: true
+
+require "pathname"
+require "active_support/message_encryptor"
+require "active_support/core_ext/string/strip"
+require "active_support/core_ext/module/delegation"
+
+module ActiveSupport
+  class EncryptedFile
+    class MissingContentError < RuntimeError
+      def initialize(content_path)
+        super "Missing encrypted content file in #{content_path}."
+      end
+    end
+
+    class MissingKeyError < RuntimeError
+      def initialize(key_path:, env_key:)
+        super \
+          "Missing encryption key to decrypt file with. " +
+          "Ask your team for your master key and write it to #{key_path} or put it in the ENV['#{env_key}']."
+      end
+    end
+
+    CIPHER = "aes-128-gcm"
+
+    def self.generate_key
+      SecureRandom.hex(ActiveSupport::MessageEncryptor.key_len(CIPHER))
+    end
+
+
+    attr_reader :content_path, :key_path, :env_key
+
+    def initialize(content_path:, key_path:, env_key:)
+      @content_path, @key_path = Pathname.new(content_path), Pathname.new(key_path)
+      @env_key = env_key
+    end
+
+    def key
+      read_env_key || read_key_file || handle_missing_key
+    end
+
+    def read
+      if content_path.exist?
+        decrypt content_path.binread
+      else
+        raise MissingContentError, content_path
+      end
+    end
+
+    def write(contents)
+      IO.binwrite "#{content_path}.tmp", encrypt(contents)
+      FileUtils.mv "#{content_path}.tmp", content_path
+    end
+
+    def change(&block)
+      writing read, &block
+    end
+
+
+    private
+      def writing(contents)
+        tmp_file = "#{content_path.basename}.#{Process.pid}"
+        tmp_path = Pathname.new File.join(Dir.tmpdir, tmp_file)
+        tmp_path.binwrite contents
+
+        yield tmp_path
+
+        updated_contents = tmp_path.binread
+
+        write(updated_contents) if updated_contents != contents
+      ensure
+        FileUtils.rm(tmp_path) if tmp_path.exist?
+      end
+
+
+      def encrypt(contents)
+        encryptor.encrypt_and_sign contents
+      end
+
+      def decrypt(contents)
+        encryptor.decrypt_and_verify contents
+      end
+
+      def encryptor
+        @encryptor ||= ActiveSupport::MessageEncryptor.new([ key ].pack("H*"), cipher: CIPHER)
+      end
+
+
+      def read_env_key
+        ENV[env_key]
+      end
+
+      def read_key_file
+        key_path.binread.strip if key_path.exist?
+      end
+
+      def handle_missing_key
+        raise MissingKeyError, key_path: key_path, env_key: env_key
+      end
+  end
+end
diff --git a/activesupport/lib/active_support/railtie.rb b/activesupport/lib/active_support/railtie.rb
index 2efe391d01..fe132ca7c6 100644
--- a/activesupport/lib/active_support/railtie.rb
+++ b/activesupport/lib/active_support/railtie.rb
@@ -49,6 +49,17 @@ module ActiveSupport
       Date.beginning_of_week_default = beginning_of_week_default
     end
 
+    initializer "active_support.require_master_key" do |app|
+      if app.config.respond_to?(:require_master_key) && app.config.require_master_key
+        begin
+          app.credentials.key
+        rescue ActiveSupport::EncryptedFile::MissingKeyError => error
+          $stderr.puts error.message
+          exit 1
+        end
+      end
+    end
+
     initializer "active_support.set_configs" do |app|
       app.config.active_support.each do |k, v|
         k = "#{k}="
diff --git a/activesupport/test/encrypted_configuration_test.rb b/activesupport/test/encrypted_configuration_test.rb
new file mode 100644
index 0000000000..53ea9e393f
--- /dev/null
+++ b/activesupport/test/encrypted_configuration_test.rb
@@ -0,0 +1,65 @@
+# frozen_string_literal: true
+
+require "abstract_unit"
+require "active_support/encrypted_configuration"
+
+class EncryptedConfigurationTest < ActiveSupport::TestCase
+  setup do
+    @credentials_config_path = File.join(Dir.tmpdir, "credentials.yml.enc")
+
+    @credentials_key_path = File.join(Dir.tmpdir, "master.key")
+    File.write(@credentials_key_path, ActiveSupport::EncryptedConfiguration.generate_key)
+
+    @credentials = ActiveSupport::EncryptedConfiguration.new \
+      config_path: @credentials_config_path, key_path: @credentials_key_path, env_key: "RAILS_MASTER_KEY"
+  end
+
+  teardown do
+    FileUtils.rm_rf @credentials_config_path
+    FileUtils.rm_rf @credentials_key_path
+  end
+
+  test "reading configuration by env key" do
+    FileUtils.rm_rf @credentials_key_path
+
+    begin
+      ENV["RAILS_MASTER_KEY"] = ActiveSupport::EncryptedConfiguration.generate_key
+      @credentials.write({ something: { good: true, bad: false } }.to_yaml)
+
+      assert @credentials[:something][:good]
+      assert_not @credentials.dig(:something, :bad)
+      assert_nil @credentials.fetch(:nothing, nil)
+    ensure
+      ENV["RAILS_MASTER_KEY"] = nil
+    end
+  end
+
+  test "reading configuration by key file" do
+    @credentials.write({ something: { good: true } }.to_yaml)
+
+    assert @credentials.something[:good]
+  end
+
+  test "change configuration by key file" do
+    @credentials.write({ something: { good: true } }.to_yaml)
+    @credentials.change do |config_file|
+      config = YAML.load(config_file.read)
+      config_file.write config.merge(new: "things").to_yaml
+    end
+
+    assert @credentials.something[:good]
+    assert_equal "things", @credentials[:new]
+  end
+
+  test "raises key error when accessing config via bang method" do
+    assert_raise(KeyError) { @credentials.something! }
+  end
+
+  private
+    def new_credentials_configuration
+      ActiveSupport::EncryptedConfiguration.new \
+        config_path: @credentials_config_path,
+        key_path: @credentials_key_path,
+        env_key: "RAILS_MASTER_KEY"
+    end
+end
diff --git a/activesupport/test/encrypted_file_test.rb b/activesupport/test/encrypted_file_test.rb
new file mode 100644
index 0000000000..7259726d08
--- /dev/null
+++ b/activesupport/test/encrypted_file_test.rb
@@ -0,0 +1,50 @@
+# frozen_string_literal: true
+
+require "abstract_unit"
+require "active_support/encrypted_file"
+
+class EncryptedFileTest < ActiveSupport::TestCase
+  setup do
+    @content = "One little fox jumped over the hedge"
+
+    @content_path = File.join(Dir.tmpdir, "content.txt.enc")
+
+    @key_path = File.join(Dir.tmpdir, "content.txt.key")
+    File.write(@key_path, ActiveSupport::EncryptedFile.generate_key)
+
+    @encrypted_file = ActiveSupport::EncryptedFile.new \
+      content_path: @content_path, key_path: @key_path, env_key: "CONTENT_KEY"
+  end
+
+  teardown do
+    FileUtils.rm_rf @content_path
+    FileUtils.rm_rf @key_path
+  end
+
+  test "reading content by env key" do
+    FileUtils.rm_rf @key_path
+
+    begin
+      ENV["CONTENT_KEY"] = ActiveSupport::EncryptedFile.generate_key
+      @encrypted_file.write @content
+
+      assert_equal @content, @encrypted_file.read
+    ensure
+      ENV["CONTENT_KEY"] = nil
+    end
+  end
+
+  test "reading content by key file" do
+    @encrypted_file.write(@content)
+    assert_equal @content, @encrypted_file.read
+  end
+
+  test "change content by key file" do
+    @encrypted_file.write(@content)
+    @encrypted_file.change do |file|
+      file.write(file.read + " and went by the lake")
+    end
+
+    assert_equal "#{@content} and went by the lake", @encrypted_file.read
+  end
+end