Crawl Delay

A directive in robots.txt that limits how fast bots can crawl a site.
Use Crawl-delay to prevent server overload from bots.