How to use the Robots.txt Generator
- Click a preset to start — “Allow everything” for most sites, “WordPress” or “Shopify-style” for those platforms, or “Block everything” for a staging site.
- Add Disallow and Allow rules for each user-agent group. Paths start with
/;*matches any characters and$anchors the end of the URL. - Optionally add “+ Block AI training bots” to opt out of model training while keeping normal search crawlers.
- Enter your sitemap URLs, review the warnings, then download
robots.txtand upload it to your domain root (e.g.https://example.com/robots.txt). Verify it with the Robots.txt Tester.
What is robots.txt?
robots.txt is a plain-text file at the root of a host (https://www.example.com/robots.txt) that tells crawlers which URLs they may request. It follows the Robots Exclusion Protocol, standardised as RFC 9309 in 2022. Each group starts with one or more User-agent lines followed by Allow and Disallow rules; a crawler obeys only the most specific group that matches its name, falling back to User-agent: *.
How rules are matched
- Rules match URL paths by prefix:
Disallow: /adminblocks/admin,/admin/and/administrator. Use/admin/to block only the folder. *matches any sequence of characters and$marks the end:Disallow: /*.pdf$blocks every PDF.- When Allow and Disallow both match, Google applies the longest (most specific) rule; on a tie, Allow wins.
- An empty
Disallow:means “allow everything”.Disallow: /blocks the whole site.
Blocking AI crawlers
AI companies use separate user-agents for model training, search indexing and user-triggered fetches, so you can opt out of training without disappearing from AI search answers:
| User-agent | Operator | Purpose |
|---|---|---|
| GPTBot | OpenAI | Training data |
| OAI-SearchBot / ChatGPT-User | OpenAI | ChatGPT search / user-requested pages |
| ClaudeBot | Anthropic | Training data |
| Claude-SearchBot / Claude-User | Anthropic | Search / user-requested pages |
| Google-Extended | Control token for Gemini training and grounding — does not affect Google Search | |
| Applebot-Extended | Apple | Control token for Apple AI training |
| CCBot | Common Crawl | Open web corpus widely used for training |
| Meta-ExternalAgent, Bytespider | Meta, ByteDance | Training data |
| PerplexityBot / Perplexity-User | Perplexity | Search index / user-requested pages |
Compliance is voluntary: reputable crawlers honour robots.txt, but it is not an access control. Use authentication or firewall rules for anything that must stay private. Check what your live site currently allows with the AI Crawler Checker.
Mistakes to avoid
- Using robots.txt to de-index pages. A blocked URL can still be indexed (without a snippet) if other sites link to it. Use a
noindexmeta tag — and don’t block that page, or Google can’t see the tag. - Blocking CSS and JavaScript. Google renders pages; blocking
/assets/or/wp-includes/can make pages look broken to it. - Relying on Crawl-delay. Bing and Yandex respect it; Googlebot ignores it.
- Shipping the staging file to production. A leftover
Disallow: /is one of the most common causes of sudden traffic drops after a launch.
Tools that pair well with the Robots.txt Generator
- Sitemap Generator — Crawl your website and get a clean sitemap.xml containing only pages that deserve to be indexed — 200 status, no noindex, self-canonical — with every excluded URL and the reason listed.
- llms.txt Generator — Write an llms.txt file that gives AI assistants a curated, Markdown map of your site — project name, summary, key pages and optional extras — following the llmstxt.org proposal.
- Meta Tag Generator — Fill in one form and get a complete, valid <head> — title, description, canonical, robots, Open Graph, X/Twitter cards, theme colour and icons — with live previews and Webflow-specific output.
- SEO Checker & Site Audit — Paste a URL to run up to 70 on-page SEO checks in seconds.
Frequently asked questions
Where do I upload robots.txt?
At the root of each host, e.g.
https://www.example.com/robots.txt. Subdomains need their own file. In Webflow, paste it into Site settings › SEO › robots.txt; on Shopify, edit the robots.txt.liquid theme template.Does blocking GPTBot remove my site from ChatGPT?
Blocking GPTBot stops OpenAI from using your pages for training. ChatGPT search uses OAI-SearchBot and user-triggered fetches use ChatGPT-User, so block those too only if you also want to stay out of ChatGPT answers.
Does Google-Extended affect my Google rankings?
No. Google-Extended is a control token for Gemini training and grounding only; Googlebot still crawls and ranks your pages normally.
Can robots.txt keep a page out of Google?
Not reliably. It stops crawling, not indexing. To remove a page from results, allow crawling and add
<meta name="robots" content="noindex">, or use password protection.How do I test my robots.txt?
Paste it into the Robots.txt Tester to check whether a given URL is allowed for Googlebot, Bingbot or AI crawlers before you publish.