Robots.txt

A file that tells search engines which pages or folders not to crawl.
Disallow admin sections or duplicate filters using the robots.txt file.