Definition
AI agent crawlers are user-agents that IDE coding agents (Cursor, Claude Code, GitHub Copilot, Windsurf) use when fetching documentation or context from the web during a coding session. Distinct from search retrieval bots, these are on-demand fetches triggered by a developer prompt. They preferentially fetch llms.txt + structured docs.
A developer asks Cursor to 'use the Stripe Payments API'. Cursor's agent fetches stripe.com/llms.txt + relevant docs URLs, includes them in the model context, writes code grounded in current API.
How to optimize
Publish llms.txt at site root (use VectorCite's generator at /audit/llms-txt). Mark docs URLs with clean canonical structure. Keep docs versioned and dated.