How to use the robots.txt Explainer
- Enter a website and click Fetch robots.txt, or paste the file contents directly.
- Read the explanation for each user-agent group — what’s blocked, what’s allowed as an exception, and what wildcards match.
- Review the issues list: blocked CSS/JS, a site-wide
Disallow: /, unsupportednoindexlines and more. - Test specific URLs for a crawler to see exactly which rule decides.
How crawlers read robots.txt
- Pick a group — a crawler uses the group whose
User-agentmatches its name most specifically, and only that group. Googlebot ignores the*group if aGooglebotgroup exists. - Find matching rules —
DisallowandAllowvalues are path prefixes.*matches any characters,$marks the end of the URL. - Longest rule wins — the most specific (longest) matching rule decides. If an Allow and a Disallow are equally long, Allow wins (Google and RFC 9309).
| Rule | Means |
|---|---|
Disallow: / | Block the whole site |
Disallow: (empty) | Block nothing |
Disallow: /admin/ | Block everything under /admin/ |
Disallow: /admin | Also blocks /admin-login and /administrator |
Disallow: /*.pdf$ | Block URLs ending in .pdf |
Disallow: /*? | Block any URL with a query string |
What robots.txt can’t do
It controls crawling, not indexing. A blocked URL can still appear in Google if other pages link to it — use a noindex meta tag (and allow crawling so Google can see it). Noindex: lines inside robots.txt have been ignored by Google since 2019, and Crawl-delay is ignored by Google (Bing and Yandex respect it).
Build a new file with the robots.txt Generator, test a live site with the robots.txt Tester, check AI bots with the AI Crawler Checker and validate the sitemaps you list with the Sitemap Validator.
Tools that pair well with the robots.txt Explainer
- Meta Tag Checker — See exactly how Google is likely to display your page: title and description measured in pixels, desktop and mobile previews, and every meta tag in the <head>.
- SEO Checker & Site Audit — Paste a URL to run up to 70 on-page SEO checks in seconds.
- IndexNow Submitter — Tell search engines about new and updated pages instantly.
- Heading Checker — Visualise the heading outline of any page.
Frequently asked questions
Does Disallow remove a page from Google?
No. Disallow stops crawling, but the URL can still be indexed from links (shown without a description). To remove a page, allow crawling and add a noindex meta tag or X-Robots-Tag header.
What does the $ sign mean in robots.txt?
It anchors the end of the URL. Disallow: /*.pdf$ blocks /files/report.pdf but not /files/report.pdf?download=1.
If Allow and Disallow both match, which wins?
The longest (most specific) matching rule. When both are the same length, Allow wins for Google and crawlers following RFC 9309.
Should I block CSS and JavaScript?
No. Google renders pages like a browser; blocking CSS or JS files can make pages look broken to Googlebot and hurt rankings, especially for mobile-friendliness.