Glossary entry x-robots-tagSchema

X-Robots-Tag

HTTP header equivalent of <meta name=robots>. It controls crawler behavior on non-HTML files (PDFs, images).

Filed under: Schema

Definition

X-Robots-Tag is an HTTP response header that controls crawler indexing rules for any URL, including non-HTML files like PDFs, images, and JSON APIs where meta tags aren't available. Same directives as <meta name='robots'>: noindex, nofollow, noarchive, nosnippet, max-snippet, max-image-preview.

Example

Apache: Header set X-Robots-Tag 'noindex' for *.pdf. Now PDF files at /docs/*.pdf are crawlable but not indexed.

How to optimize

Use X-Robots-Tag for PDFs, JSON files, generated reports. Combine with robots.txt to control crawl behavior on file types. Test via curl -I.

Check a page against a real AI answer

Claude’s live answer to your buyer’s question, its sources, your readiness score and the fixes. Free, no signup, about half a minute.

Run a free audit