AI search
Can ChatGPT read your site? AI crawler checker
Find out whether ChatGPT, Claude, Perplexity, Google's AI and Apple can reach your pages. We read your robots.txt rule by rule, then fetch the page as each AI crawler to catch firewall blocks and JavaScript-only content.
Free · No signup · Reads public pages only
How it works
Enter your homepage or any page you want AI answers to find.
We check robots.txt for every known AI crawler, then fetch the page nine times: once as a normal browser and once as each AI bot.
You get a verdict per assistant, the exact rule or response that decided it, and the fix.
Search bots, on-request bots and training bots are different
Each AI company sends more than one crawler, and they do different jobs. Mixing them up is the most common way sites block the wrong thing.
- Search crawlers find pages to show and link in AI answers: OAI-SearchBot for ChatGPT search, Claude-SearchBot, PerplexityBot. Block these and you can't be cited.
- On-request agents open a page because a person asked: ChatGPT-User, Claude-User, Perplexity-User. They're what fetches your page when someone pastes your link into a chat.
- Training crawlers collect pages to train models: GPTBot, ClaudeBot, CCBot (Common Crawl). Blocking them is a choice about training. It doesn't remove you from AI search.
Google and Apple work differently. Google's AI Overviews and AI Mode are part of Google Search and use Googlebot, the same crawler as normal results. Google-Extended is a robots.txt name that never visits your site; it only controls whether Gemini may train on or ground its answers in your content, and Google says it doesn't affect Search. Apple has the same split: Applebot crawls for Siri and Spotlight, and Applebot-Extended only governs training.
Three ways sites hide from AI by accident
robots.txt rules that catch more than intended. A Disallow: / left over from a staging site, a User-agent: * block meant for scrapers, or a list of AI bots copied from a blog post that includes the search crawlers along with the training ones. The checker shows the exact line for each bot.
Firewalls and CDN bot protection. Security tools often block anything that looks automated, and AI crawlers look automated. Cloudflare, for example, has a setting that blocks AI crawlers with one switch, and in 2025 it began turning that on by default for new sites. robots.txt can say "welcome" while the firewall says "403". That's why we fetch the page as each bot and compare it with a normal browser: a bot that gets an error or a "Just a moment..." check page while the browser gets the real page is being turned away.
Content that only exists after JavaScript runs. Googlebot runs JavaScript. Most AI crawlers read the raw HTML and stop there, so a single-page app that ships an empty shell looks empty to them. We count the words in the HTML before any script runs.
How to read the results
Each assistant gets one verdict. "Can read it" means robots.txt allows its crawlers and our live fetch got the page. "Blocked" means a robots.txt rule or your firewall stops it. "Partly blocked" means one of its crawlers gets through and another doesn't, for example PerplexityBot allowed but Perplexity-User turned away. "Opted out" is used for training crawlers and usage controls: it's a choice, not a fault, and it doesn't change whether AI answers can cite you.
Under each crawler you'll see two lines: what robots.txt says (with the line number when a rule decided it) and what happened when we fetched the page under its name. The score at the top only counts the crawlers that affect AI answers, never the training ones.
What this checker can't see
We fetch from our own servers with each crawler's name. The real crawlers come from their companies' IP addresses, and many firewalls check those. So a firewall that allowlists verified bots may let the real GPTBot in after turning us away, or block it while letting our test through. robots.txt verdicts follow the file exactly as your site served it to us; the live checks are a strong hint. For proof, look for the crawlers' visits in your server or CDN logs.
It checks one page. Rules often differ by path, so test your key pages too: pricing, top articles, product pages. It can't see whether an assistant actually cites you; that depends on your content and on what people ask. Our AI visibility checker looks at that side.
Questions people ask
Does blocking GPTBot remove my site from ChatGPT?
No. GPTBot only collects training data. ChatGPT search uses OAI-SearchBot, and OpenAI treats the two separately, so you can opt out of training and still be found and linked in ChatGPT search.
Does blocking Google-Extended keep me out of AI Overviews?
No. AI Overviews and AI Mode are part of Google Search and use Googlebot. Google-Extended only controls Gemini training and grounding. To keep a page's text out of AI Overviews you'd need nosnippet or noindex, which also affect normal search results.
Why does it say my firewall blocks a bot when I've allowed it?
Your firewall may check IP addresses, and our test can't come from the real crawler's addresses. If you've allowed the verified crawler by name in your firewall's bot settings, the real one probably gets through. Your server logs will confirm it.
Do AI crawlers run JavaScript?
Googlebot does. Most AI crawlers don't: they read the HTML your server sends. If your text is added by JavaScript, render it on the server so every crawler sees it.
Do I need an llms.txt file?
It's optional. llms.txt is a proposal for a plain list of your key pages, and no major AI company has said it uses it for search. It's cheap to add, so our llms.txt generator can draft one from your sitemap.
Is this AI crawler checker free?
Yes. No signup, and it only reads pages that are public on the web.
Want it done for you?
Romeo gets you into AI answers, not just past the door
He fixes the robots.txt rules and page settings that hide you from AI search, then rewrites your pages so ChatGPT, Perplexity and Google's AI have a clear answer to quote. Every change comes to you for approval first.
