Sign in

Robots reader

Read your robots.txt by bot class: search index, fetch for a person, and model training. Catch a noindex that a blocked crawler can never see.

We read /robots.txt on that site. If you give a page, we also check it for a noindex that blocked crawlers can never see.

Bots this reads for

TokenVendorWhat it does
OAI-SearchBotOpenAISearch index
ChatGPT-UserOpenAIFetches for a person
GPTBotOpenAIModel training
Claude-SearchBotAnthropicSearch index
Claude-UserAnthropicFetches for a person
ClaudeBotAnthropicModel training
PerplexityBotPerplexitySearch index
Perplexity-UserPerplexityFetches for a person
GooglebotGoogleSearch index
Google-ExtendedGoogleModel training (a robots.txt token, not a crawler)
BingbotMicrosoftSearch index
Applebot-ExtendedAppleModel training (a robots.txt token, not a crawler)
CCBotCommon CrawlModel training

More free tools

Search setup checkIs your site in Google Search Console and Bing Webmaster Tools? What to do if not.Facts file builderWrite dated, sourced facts about your business, for people and for machines.First-screen previewSee your title and the first 200 characters after the headline.