Robots Header

An HTTP header used to control crawling and indexing.
Use X-Robots-Tag headers for PDFs and non-HTML files.