Home / Updates / Check that your robots.txt lets AI search crawlers in
Check that your robots.txt lets AI search crawlers in
AI assistants read websites with their own search crawlers, and one line in your robots.txt file can quietly keep them out.
Written by John AlewineReviewed by the Kansas City SEO Services team
Published · Updated
When someone asks ChatGPT, Perplexity, or Claude who to call, the assistant can search the web. Each company runs its own search crawler to read sites: OAI-SearchBot for ChatGPT search, PerplexityBot for Perplexity, and Claude-SearchBot for Claude. Your robots.txt file tells them whether they may come in.
How to check
- Open yoursite.com/robots.txt in a browser.
- Look for a User-agent line naming OAI-SearchBot, PerplexityBot, or Claude-SearchBot with Disallow: / under it.
- Look for a blanket block: User-agent: * followed by Disallow: /. That blocks every crawler that follows the file.
Owners rarely write these lines themselves. A website builder setting or an old plugin can add them. If you find a block you did not mean to have, change the setting or ask whoever runs your site.
OpenAI and Perplexity recommend allowing their search crawlers if you want to show up in their search results. Anthropic says blocking Claude-SearchBot may reduce your visibility in Claude search. Allowing them makes your site readable. It does not promise any AI answer or placement.
Want to see where your listing stands first? Start with the free Map Pack Check.
Source: https://developers.openai.com/api/docs/bots
Source: https://docs.perplexity.ai/docs/resources/perplexity-crawlers