Skip to content
VectorCite
Free audit

Glossary entry robots-txt

robots.txt

The /robots.txt file telling crawlers what they can fetch. The 2026 retrieval-vs-training distinction matters most.

Filed under
Frameworks

Definition

robots.txt is a plain-text file at the root of a domain that tells web crawlers which paths they can access. The critical 2026 distinction is between RETRIEVAL bots (OAI-SearchBot, PerplexityBot, Claude-SearchBot) which drive citation, and TRAINING bots (GPTBot, ClaudeBot, Google-Extended) which inform future models. Blocking retrieval bots kills your AEO; blocking training bots is a content-policy decision.

Example

User-agent: OAI-SearchBot Allow: / User-agent: GPTBot Disallow: / (Allows ChatGPT citations; blocks OpenAI training.)

How to optimize

Explicitly Allow OAI-SearchBot, PerplexityBot, Claude-SearchBot, ChatGPT-User. Decide separately whether to Allow GPTBot, ClaudeBot. VectorCite's ai-bot-access signal flags blocked retrieval bots.

Check a page against a real AI answer

Claude’s live answer to your buyer’s question, its sources, your readiness score and the fixes. Free, no signup, about half a minute.

Run a free audit