Merge pull request #18485 from andreynering/robots.txt

Adding 'Search Engines Indexing' Section [ci skip]

2 files changed, 22 insertions, 0 deletions

diff --git a/guides/CHANGELOG.md b/guides/CHANGELOG.md
index 99d69d5eda..dd5ca4b395 100644
--- a/guides/CHANGELOG.md
+++ b/guides/CHANGELOG.md
@@ -6,4 +6,8 @@
 
     *Andrey Nering*
 
+*   New section in Configuring: Search Engines Indexing
+
+    *Andrey Nering*
+
 Please check [4-2-stable](https://github.com/rails/rails/blob/4-2-stable/guides/CHANGELOG.md) for previous changes.
diff --git a/guides/source/configuring.md b/guides/source/configuring.md
index 9c0f2ddc8a..994cc31cff 100644
--- a/guides/source/configuring.md
+++ b/guides/source/configuring.md
@@ -1049,3 +1049,21 @@ These configuration points are then available through the configuration object:
   Rails.configuration.x.super_debugger              # => true
   Rails.configuration.x.super_debugger.not_set      # => nil
   ```
+
+Search Engines Indexing
+-----------------------
+
+Sometimes, you may want to prevent some pages of your application be visible on search sites like Google,
+Bing, Yahoo or Duck Duck Go. The robots that index these sites will first analyse the
+`https://your-site.com/robots.txt` file to know what pages it is allowed to index.
+
+Rails creates this file for you on `/public` folder. By default, it allows search engines to index all
+pages of your application. If you want to block indexing on all pages of you application, use this:
+
+```
+User-agent: *
+Disallow: /
+```
+
+To block just specific pages, it's necessary to use a more complex syntax. Learn it on the
+[official documentation](http://www.robotstxt.org/robotstxt.html).