Check a site
Robots.txt checker and tester
Fetch any site's robots.txt, see every line numbered with its problems marked, and test whether Googlebot, Bingbot, GPTBot or any other crawler may fetch a URL, with the exact line that decides it.
Free · No signup · Reads public pages only
How it works
Enter a website, or a page on it if you want to test that page straight away.
We fetch /robots.txt the way Google does, follow its redirects, and read the rules with Google's own matching logic.
You get the file with line numbers, a list of what's wrong, a verdict for every known crawler, and a tester for any URL and bot.
What robots.txt does, and what it doesn't
robots.txt is a plain text file at the root of a site (example.com/robots.txt) that tells crawlers which paths they may fetch. Each block of rules starts with a User-agent line naming a crawler, followed by Allow and Disallow lines with paths. A crawler is a program that downloads pages: Googlebot for Google Search, Bingbot for Bing, GPTBot for OpenAI's training data, and so on.
It controls crawling, not indexing. A page blocked in robots.txt can still show up in Google if other sites link to it; Google just won't know what's on it. To keep a page out of search results, let Google crawl it and add a noindex tag. If you block it in robots.txt as well, Google never sees the noindex.
It's also a request, not a lock. Well-behaved crawlers follow it. Scrapers that ignore it need a firewall rule instead.
How crawlers decide which rule wins
This is where most robots.txt mistakes come from, so the tester follows Google's documented rules exactly:
- A crawler follows one group only: the one that names it, or the User-agent: * group if none does. A GPTBot group replaces the * rules for GPTBot; the two are never combined.
- Within that group, the longest matching path wins. Disallow: /blog loses to Allow: /blog/public for /blog/public/post.
- When an Allow and a Disallow match with the same length, Allow wins.
- * matches any run of characters, and $ at the end means "the URL ends here". Disallow: /*.pdf$ blocks PDFs but not /file.pdf?download=1.
- Paths are case-sensitive. Disallow: /Private doesn't block /private.
- An empty Disallow: line blocks nothing.
The tester highlights the line that made the decision, and the group the crawler used, so you can see why a URL is blocked instead of guessing.
When the file itself is the problem
Google treats the HTTP answer for robots.txt as part of the rules. A 404 or other 4xx means "no robots.txt", so every crawler may fetch everything. A 5xx error or a 429 means the opposite: Google stops crawling the whole site until it gets a real answer. If the errors last for weeks, it falls back to the last copy it saw.
Other common breakages this checker flags: a single-page app that answers /robots.txt with its HTML page (crawlers find no rules in it), a file over Google's 500 KiB limit (the rest is ignored), rules placed before any User-agent line (nobody follows them), misspelled directives, full URLs where a path belongs, and Sitemap lines with a relative address, which Google ignores.
Google caches robots.txt for up to a day, so a fix can take that long to be noticed.
What this checker can't see
It reads robots.txt for the host you enter. Every subdomain and protocol has its own file: example.com's rules don't cover shop.example.com or the http:// version of the site.
It also can't see firewall rules, login walls or a CDN's bot settings, which can block a crawler that robots.txt allows. The AI crawler checker fetches your page as each AI bot to catch those. For Google, the URL Inspection tool in Search Console shows how Google itself last fetched a page.
Questions people ask
Does robots.txt remove pages from Google?
No. It stops Google from crawling them, but a blocked URL can still be indexed from links, usually without a description. Use a noindex tag on a crawlable page to remove it from results.
Why does the tester say a page is allowed when I disallowed it for all bots?
Most likely the crawler has its own group further down. A crawler that's named in a group ignores the User-agent: * rules entirely, so anything you want to block for everyone has to be repeated in each named group.
Does Google follow Crawl-delay?
No. Google ignores Crawl-delay; Bing and some other crawlers follow it. Google adjusts its crawl rate to how quickly your server responds.
Should I block AI crawlers in robots.txt?
Block training crawlers like GPTBot or CCBot if you don't want your content used to train models. Leave AI search crawlers like OAI-SearchBot and PerplexityBot allowed if you want to be cited in AI answers. They're separate names, so you can choose each one.
Is this robots.txt checker free?
Yes. No signup. It only reads files that are public on the web.
Want it done for you?
Romeo watches robots.txt so a bad rule doesn't sit there for months
He checks your robots.txt every time he crawls your site, flags rules that hide pages you want in Google or AI answers, and sends the fix as a change you approve.
