Generators
Robots.txt generator
Build a robots.txt file with separate choices for search engines, AI search, AI training and AI assistants, then copy it or download it ready to upload.
Free · No signup · Runs in your browser
Your robots.txt
Every crawler allowed · 1 fine
# Every crawler not named in a section below
User-agent: *
Allow: /
Upload it to the root of your site, so it opens at https://yourdomain.com/robots.txt. Each subdomain needs its own file.
What this file does
- Good
Search engines can crawl your site
- Note
No sitemap listed
Adding your sitemap's full address helps every crawler find your pages, not just the ones you've told about it in their webmaster tools. - Note
Robots.txt is a request, not a lock
Well-behaved crawlers follow it; others don't. To keep a page out of search results, use a noindex tag and let crawlers see it. To keep something private, put it behind a password.
Already have a robots.txt? Test it line by line with the robots.txt checker, or see whether AI crawlers can actually reach your pages (firewalls included) with the AI crawler checker.
How it works
Start from a preset, or set each crawler to Allow or Block.
Add any folders to keep crawlers out of, and your sitemap's full address.
Copy the file or download it as robots.txt, and upload it to the root of your site.
What robots.txt does, and what it doesn't
Robots.txt is a plain text file at the root of a site, like example.com/robots.txt. Crawlers read it before they fetch anything else, and it tells each one which paths it may visit. Google, Bing and the well-known AI companies say they follow it.
It controls crawling, not indexing. A page you block can still appear in Google if other sites link to it, usually without a description, because Google never read it. To keep a page out of search results, leave it crawlable and add a noindex tag. To keep something private, put it behind a password. Robots.txt is public, so anyone can read the paths you list in it.
Search bots, AI search bots and AI training bots are different crawlers
Robots.txt used to be about search engines. Now most AI companies run several crawlers with different jobs, and you can treat each one separately:
- Search engines, like Googlebot and Bingbot, put your pages in search results. Google's AI Overviews and AI Mode use the same Googlebot crawl, so blocking Googlebot removes you from those too.
- AI search crawlers, like OAI-SearchBot, Claude-SearchBot and PerplexityBot, fetch pages so an assistant can show and link them in its answers.
- AI training crawlers, like GPTBot, ClaudeBot and CCBot, plus the Google-Extended and Applebot-Extended tokens, decide whether your content is used to train models.
- Assistant fetches, like ChatGPT-User and Claude-User, happen when a person asks an assistant to open a specific page.
That's why "block AI" is rarely one decision. A common choice is the second preset: stay in search results and AI answers, opt out of training. Blocking GPTBot doesn't remove you from ChatGPT search; blocking OAI-SearchBot does.
How the file is put together
Each section starts with one or more User-agent lines naming crawlers, followed by Disallow and Allow rules. A crawler follows the section that names it most specifically and ignores the rest, including the general * section. That's why a blocked bot gets a short section of its own here, and why the generator only writes sections for crawlers that differ from the default: a shorter file is easier to check.
Inside a section, Google and Bing follow the longest matching rule, so Allow: /admin/help/ wins over Disallow: /admin/ for that folder. An asterisk matches any characters and a dollar sign marks the end of an address, so Disallow: /*.pdf$ blocks PDF files. Paths are case-sensitive: /Admin/ and /admin/ are different.
Mistakes worth avoiding
- Leaving Disallow: / on a live site after launch. It's the classic staging leftover, and it slowly removes the whole site from Google.
- Blocking CSS and JavaScript folders. Google renders pages the way a browser does, and blocked files can make pages look broken to it.
- Using robots.txt to hide pages from search. Use noindex instead, and don't block the page, or Google can't see the noindex.
- Writing a relative sitemap address. The Sitemap line needs the full URL.
- Expecting Crawl-delay to slow Google down. Google ignores it; Bing and Yandex follow it.
- Forgetting subdomains. The file covers one host, so blog.example.com needs its own.
What this generator can't tell you
It builds a file from your choices; it doesn't look at your live site. Some firewalls and CDNs block AI crawlers before robots.txt is ever read, and some assistants treat a page a person asked for like a normal browser visit and may not check robots.txt at all. The robots.txt checker tests your live file against any address, and the AI crawler checker fetches your pages as each AI bot to find blocks that robots.txt can't explain.
Questions people ask
Where do I put robots.txt?
At the root of the host, so it opens at https://yourdomain.com/robots.txt. A file in a subfolder is ignored. Many website platforms let you edit robots.txt in their settings instead of uploading a file.
Does blocking GPTBot remove me from ChatGPT?
No. GPTBot collects training data. ChatGPT's search answers rely on OAI-SearchBot, and pages a user asks ChatGPT to open are fetched as ChatGPT-User. Block GPTBot and allow OAI-SearchBot to opt out of training while staying in ChatGPT search.
Will blocking Google-Extended remove me from AI Overviews?
No. Google-Extended controls whether your content is used for Gemini models. Google says it doesn't affect inclusion or ranking in Google Search, and AI Overviews and AI Mode are part of Search, which Googlebot crawls.
How long do changes take to work?
Google generally caches robots.txt for up to a day, so a change can take that long to be picked up. Other crawlers have their own schedules.
Is robots.txt enough to stop AI scrapers?
Only the ones that follow it. Named crawlers from the big AI companies say they respect it; anonymous scrapers don't. For a firm block, use your host's or CDN's bot controls alongside it.
Is this robots.txt generator free?
Yes, with no signup. The file is built in your browser and nothing you choose is sent to us.
Want it done for you?
Romeo keeps your crawl rules and your pages in step
He crawls your site the way search engines do, flags pages that robots.txt, noindex or a firewall hide by mistake, and sends the fix for you to approve.
