ActiveSupport::Cache refactoring

All Caches

* Add default options to initializer that will be sent to all read, write, fetch, exist?, increment, and decrement
* Add support for the :expires_in option to fetch and write for all caches. Cache entries are stored with the create timestamp and a ttl so that expiration can be handled independently of the implementation.
* Add support for a :namespace option. This can be used to set a global prefix for cache entries.
* Deprecate expand_cache_key on ActiveSupport::Cache and move it to ActionController::Caching and ActionDispatch::Http::Cache since the logic in the method used some Rails specific environment variables and was only used by ActionPack classes. Not very DRY but there didn't seem to be a good shared spot and ActiveSupport really shouldn't be Rails specific.
* Add support for :race_condition_ttl to fetch. This setting can prevent race conditions on fetch calls where several processes try to regenerate a recently expired entry at once.
* Add support for :compress option to fetch and write which will compress any data over a configurable threshold.
* Nil values can now be stored in the cache and are distinct from cache misses for fetch.
* Easier API to create new implementations. Just need to implement the methods read_entry, write_entry, and delete_entry instead of overwriting existing methods.
* Since all cache implementations support storing objects, update the docs to state that ActiveCache::Cache::Store implementations should store objects. Keys, however, must be strings since some implementations require that.
* Increase test coverage.
* Document methods which are provided as convenience but which may not be universally available.

MemoryStore

* MemoryStore can now safely be used as the cache for single server sites.
* Make thread safe so that the default cache implementation used by Rails is thread safe. The overhead is minimal and it is still the fastest store available.
* Provide :size initialization option indicating the maximum size of the cache in memory (defaults to 32Mb).
* Add prune logic that removes the least recently used cache entries to keep the cache size from exceeding the max.
* Deprecated SynchronizedMemoryStore since it isn't needed anymore.

FileStore

* Escape key values so they will work as file names on all file systems, be consistent, and case sensitive
* Use a hash algorithm to segment the cache into sub directories so that a large cache doesn't exceed file system limits.
* FileStore can be slow so implement the LocalCache strategy to cache reads for the duration of a request.
* Add cleanup method to keep the disk from filling up with expired entries.
* Fix increment and decrement to use file system locks so they are consistent between processes.

MemCacheStore

* Support all keys. Previously keys with spaces in them would fail
* Deprecate CompressedMemCacheStore since it isn't needed anymore (use :compress => true)

[#4452 state:committed]

Signed-off-by: Jeremy Kemper <jeremy@bitsweat.net>

1 files changed, 139 insertions, 41 deletions

diff --git a/activesupport/lib/active_support/cache/file_store.rb b/activesupport/lib/active_support/cache/file_store.rb
index 7521efe7c5..fc225e77a2 100644
--- a/activesupport/lib/active_support/cache/file_store.rb
+++ b/activesupport/lib/active_support/cache/file_store.rb
@@ -3,73 +3,171 @@ require 'active_support/core_ext/file/atomic'
 module ActiveSupport
   module Cache
     # A cache store implementation which stores everything on the filesystem.
+    #
+    # FileStore implements the Strategy::LocalCache strategy which implements
+    # an in memory cache inside of a block.
     class FileStore < Store
       attr_reader :cache_path
 
-      def initialize(cache_path)
+      DIR_FORMATTER = "%03X"
+      ESCAPE_FILENAME_CHARS = /[^a-z0-9_.-]/i
+      UNESCAPE_FILENAME_CHARS = /%[0-9A-F]{2}/
+
+      def initialize(cache_path, options = nil)
+        super(options)
         @cache_path = cache_path
+        extend Strategy::LocalCache
       end
 
-      # Reads a value from the cache.
-      #
-      # Possible options:
-      # - +:expires_in+ - the number of seconds that this value may stay in
-      #   the cache.
-      def read(name, options = nil)
-        super do
-          file_name = real_file_path(name)
-          expires = expires_in(options)
-
-          if File.exist?(file_name) && (expires <= 0 || Time.now - File.mtime(file_name) < expires)
-            File.open(file_name, 'rb') { |f| Marshal.load(f) }
-          end
+      def clear(options = nil)
+        root_dirs = Dir.entries(cache_path).reject{|f| ['.', '..'].include?(f)}
+        FileUtils.rm_r(root_dirs.collect{|f| File.join(cache_path, f)})
+      end
+
+      def cleanup(options = nil)
+        options = merged_options(options)
+        each_key(options) do |key|
+          entry = read_entry(key, options)
+          delete_entry(key, options) if entry && entry.expired?
         end
       end
 
-      # Writes a value to the cache.
-      def write(name, value, options = nil)
-        super do
-          ensure_cache_path(File.dirname(real_file_path(name)))
-          File.atomic_write(real_file_path(name), cache_path) { |f| Marshal.dump(value, f) }
-          value
+      def increment(name, amount = 1, options = nil)
+        file_name = key_file_path(namespaced_key(name, options))
+        lock_file(file_name) do
+          options = merged_options(options)
+          if num = read(name, options)
+            num = num.to_i + amount
+            write(name, num, options)
+            num
+          else
+            nil
+          end
         end
-      rescue => e
-        logger.error "Couldn't create cache directory: #{name} (#{e.message})" if logger
       end
 
-      def delete(name, options = nil)
-        super do
-          File.delete(real_file_path(name))
+      def decrement(name, amount = 1, options = nil)
+        file_name = key_file_path(namespaced_key(name, options))
+        lock_file(file_name) do
+          options = merged_options(options)
+          if num = read(name, options)
+            num = num.to_i - amount
+            write(name, num, options)
+            num
+          else
+            nil
+          end
         end
-      rescue SystemCallError => e
-        # If there's no cache, then there's nothing to complain about
       end
 
       def delete_matched(matcher, options = nil)
-        super do
-          search_dir(@cache_path) do |f|
-            if f =~ matcher
-              begin
-                File.delete(f)
-              rescue SystemCallError => e
-                # If there's no cache, then there's nothing to complain about
+        options = merged_options(options)
+        instrument(:delete_matched, matcher.inspect) do
+          matcher = key_matcher(matcher, options)
+          search_dir(cache_path) do |path|
+            key = file_path_key(path)
+            delete_entry(key, options) if key.match(matcher)
+          end
+        end
+      end
+
+      protected
+
+        def read_entry(key, options)
+          file_name = key_file_path(key)
+          if File.exist?(file_name)
+            entry = File.open(file_name) { |f| Marshal.load(f) }
+            if entry && !entry.expired? && !entry.expires_in && !self.options[:expires_in]
+              # Check for deprecated use of +:expires_in+ option from versions < 3.0
+              deprecated_expires_in = options[:expires_in]
+              if deprecated_expires_in
+                ActiveSupport::Deprecation.warn('Setting :expires_in on read has been deprecated in favor of setting it on write.', caller)
+                if entry.created_at + deprecated_expires_in.to_f <= Time.now.to_f
+                  delete_entry(key, options)
+                  entry = nil
+                end
               end
             end
+            entry
           end
+        rescue
+          nil
         end
-      end
 
-      def exist?(name, options = nil)
-        super do
-          File.exist?(real_file_path(name))
+        def write_entry(key, entry, options)
+          file_name = key_file_path(key)
+          ensure_cache_path(File.dirname(file_name))
+          File.atomic_write(file_name, cache_path) {|f| Marshal.dump(entry, f)}
+          true
+        end
+
+        def delete_entry(key, options)
+          file_name = key_file_path(key)
+          if File.exist?(file_name)
+            begin
+              File.delete(file_name)
+              delete_empty_directories(File.dirname(file_name))
+              true
+            rescue => e
+              # Just in case the error was caused by another process deleting the file first.
+              raise e if File.exist?(file_name)
+              false
+            end
+          end
         end
-      end
 
       private
-        def real_file_path(name)
-          '%s/%s.cache' % [@cache_path, name.gsub('?', '.').gsub(':', '.')]
+        # Lock a file for a block so only one process can modify it at a time.
+        def lock_file(file_name, &block) # :nodoc:
+          if File.exist?(file_name)
+            File.open(file_name, 'r') do |f|
+              begin
+                f.flock File::LOCK_EX
+                yield
+              ensure
+                f.flock File::LOCK_UN
+              end
+            end
+          else
+            yield
+          end
+        end
+
+        # Translate a key into a file path.
+        def key_file_path(key)
+          fname = key.to_s.gsub(ESCAPE_FILENAME_CHARS){|match| "%#{match.ord.to_s(16).upcase}"}
+          hash = Zlib.adler32(fname)
+          hash, dir_1 = hash.divmod(0x1000)
+          dir_2 = hash.modulo(0x1000)
+          fname_paths = []
+          # Make sure file name is < 255 characters so it doesn't exceed file system limits.
+          if fname.size <= 255
+            fname_paths << fname
+          else
+            while fname.size <= 255
+              fname_path << fname[0, 255]
+              fname = fname[255, -1]
+            end
+          end
+          File.join(cache_path, DIR_FORMATTER % dir_1, DIR_FORMATTER % dir_2, *fname_paths)
+        end
+
+        # Translate a file path into a key.
+        def file_path_key(path)
+          fname = path[cache_path.size, path.size].split(File::SEPARATOR, 4).last
+          fname.gsub(UNESCAPE_FILENAME_CHARS){|match| $1.ord.to_s(16)}
+        end
+
+        # Delete empty directories in the cache.
+        def delete_empty_directories(dir)
+          return if dir == cache_path
+          if Dir.entries(dir).reject{|f| ['.', '..'].include?(f)}.empty?
+            File.delete(dir) rescue nil
+            delete_empty_directories(File.dirname(dir))
+          end
         end
 
+        # Make sure a file path's directories exist.
         def ensure_cache_path(path)
           FileUtils.makedirs(path) unless File.exist?(path)
         end