Robots.txt for Search Engines in Magento 2.X

Robots.txt is a text file used to inform and instruct search engine robots and other crawlers (such as mobile crawlers and desktop crawlers) about the pages and urls of your website.
This text file is placed in root directory of your websop and bots look for this file when they visit your site to index it.
Using a robots.txt file you can instruct crawlers and bots about the parts of your website that they should crawl and parts that they should ignore. You can give these crawl instructions by allowing or disallowing the behaviour of user agents. This helps search engine bots and crawlers to easily go through the allowed areas of your site and skip disallowed areas.
Think of it as leaving a small note for your family members on your refrigerator door about things that are for dinner and chores that should be performed.
You can read more about robots.txt here.

Alot of webshop and website owners ignore it or do not even know about robots.txt and that is a big SEO mistake. You should do everything possible to make it easier for search engines to crawl and index your website.

Fortunately, you can now easily add Robots.txt from admin in Magento 2.X. This was not possible in Magento 1.X.
This new Magento 2.X feature allows you to add, generate and manage instructions for crawlers and bots that visit your Magento website. All your instructions will be added to a file named 'robots.txt' in your Magento root directory.

Magento 2.X also generates a robot.txt file by default. This default file instructs crawlers and bots to avoid indexing Magento's internal system folders.
It is always recommend that you do not rely solely on this default robots.txt and add your own custom instructions for the bots.

To add custom instructions to your robots.txt file follow these steps: Go to Magento admin backend > Content > Design - Configuration > Click on Edit link of main 'Global' website> Search engine robots

You will see 2 fields here:

1. Default Robots
It will have following 4 options available:

INDEX, FOLLOW: This will instruct the bots to keep indexing your webshop periodically
NOINDEX, FOLLOW: This will tell bots to not to index the site at this moment but to check for changes to this instruction in their future visits.
INDEX, NOFOLLOW: This instruction will tell bots to index your Magento webshop once and then to not come back again for indexing new changes.
NOINDEX, NOFOLLOW: This will tell bots to not index the webshop and to not even come back in future to check for changes.

As expected, it is recommended that you select 'INDEX, FOLLOW' option here.

2.  Edit custom instruction of robots.txt File
In this field, you can add your own instructions that should be added to robots.txt

'Restore to default' button will remove all your custom instructions and populate the field with following default Magento instructions:

  User-agent: *
  Disallow: /index.php/
  Disallow: /*?
  Disallow: /checkout/
  Disallow: /app/
  Disallow: /lib/
  Disallow: /*.php$
  Disallow: /pkginfo/
  Disallow: /report/
  Disallow: /var/
  Disallow: /catalog/
  Disallow: /customer/
  Disallow: /sendfriend/
  Disallow: /review/
  Disallow: /*SID=

There are alot of other things that you can allow or disallow. We recommend adding at-least following instructions:
 

  User-agent: *
  
  # These will disallow bots from indexing Magento system directories
  Disallow: /app/
  Disallow: /bin/
  Disallow: /dev/
  Disallow: /lib/
  Disallow: /phpserver/
  Disallow: /pkginfo/
  Disallow: /report/
  Disallow: /setup/
  Disallow: /update/
  Disallow: /var/
  Disallow: /vendor/
  
  # These will disallow bots from indexing system files present in Magento root
  Disallow: /composer.json
  Disallow: /composer.lock
  Disallow: /CONTRIBUTING.md
  Disallow: /CONTRIBUTOR_LICENSE_AGREEMENT.html
  Disallow: /COPYING.txt
  Disallow: /Gruntfile.js
  Disallow: /LICENSE.txt
  Disallow: /LICENSE_AFL.txt
  Disallow: /nginx.conf.sample
  Disallow: /package.json
  Disallow: /php.ini.sample
  Disallow: /RELEASE_NOTES.txt

  # These will disallow bots from indexing system urls
  Disallow: /index.php/
  Disallow: /catalog/product_compare/
  Disallow: /catalog/category/view/
  Disallow: /catalog/product/view/
  Disallow: /catalogsearch/
  Disallow: /checkout/
  Disallow: /control/
  Disallow: /contacts/
  Disallow: /customer/
  Disallow: /customize/
  Disallow: /newsletter/
  Disallow: /review/
  Disallow: /sendfriend/
  Disallow: /wishlist/

  # These will disallow bots from indexing Magento product list pages that are sorted or filtered.
  Disallow: /*?*product_list_mode=
  Disallow: /*?*product_list_order=
  Disallow: /*?*product_list_limit=
  Disallow: /*?*product_list_dir=

  # You should also disallow bots from indexing urls with session ID mentioned in them
  Disallow: /*?SID=
  Disallow: /*?
  Disallow: /*.php$

Don't forget to mention your XML sitemap url as well at the bottom of your robots.txt file.

Please consult with your Magento developers and SEO experts before allowing or disallowing any parts of your Magento 2.X webshop. It can also negatively harm your SEO scores and ranls if done incorrectly.
You can always write to Hungersoft for a free consulation.