Skip to content

Glossary entry gptbot

GPTBot

OpenAI's TRAINING-data crawler, distinct from OAI-SearchBot (retrieval).

Filed under
Engines & bots

Definition

GPTBot is the user-agent OpenAI uses to crawl the web for TRAINING data, not for ChatGPT's real-time retrieval. Blocking GPTBot in robots.txt only prevents your content from being used in future OpenAI model training. It does NOT affect ChatGPT's ability to cite your pages via OAI-SearchBot retrieval.

Example

User-agent: GPTBot Disallow: / (Blocks future training use, but ChatGPT can still cite the page through OAI-SearchBot retrieval.)

How to optimize

Decide based on your content policy. Many publishers Disallow GPTBot but Allow OAI-SearchBot: opt out of training, opt in to citation. VectorCite's ai-bot-access signal scores this distinction.

Check a page against a real AI answer

Claude’s live answer to your buyer’s question, its sources, your readiness score and the fixes. Free, no signup, about half a minute.

Run a free audit