Robots.txt

The robots.txt file instructs search engine crawlers which parts of a website can or cannot be crawled.
Example: Use robots.txt to block admin or duplicate pages.