How to use the Robots.txt Tester
- Enter your site URL. To test a specific page, enter its full URL or type the path (e.g.
/checkout?step=2) in the path box. - Pick a user-agent such as Googlebot, Bingbot or GPTBot — or type any other.
- Click Test to see whether the path is allowed or blocked, which rule matched, and how other crawlers are treated.
- Edit the robots.txt in the editor and click Test edited version to try changes safely, then copy or download the result.
What is a robots.txt tester?
robots.txt is a plain-text file at the root of a host (https://example.com/robots.txt) that tells crawlers which paths they may request. A tester fetches that file, parses it the way search engines do, and answers one question precisely: may this crawler fetch this URL? Since Google retired the robots.txt Tester in Search Console in 2023, a standalone tester is the quickest way to check rules before and after a deployment.
How rules are matched
Matching follows Google’s documented behaviour and the robots exclusion standard (RFC 9309):
- Pick the group. A crawler obeys only the group whose
User-agentmatches it most specifically. IfGooglebothas its own group, theUser-agent: *rules are ignored for it entirely. - Find matching rules. Each
AllowandDisallowpath is matched from the start of the URL path, including the query string.*matches any characters and$anchors the end. - Longest match wins. The rule with the longest path decides. If an Allow and a Disallow are equally long, Allow wins. No matching rule means allowed.
| Rule | Blocks | Does not block |
|---|---|---|
Disallow: /admin | /admin, /admin/users, /administrator | /Admin (paths are case-sensitive) |
Disallow: /*.pdf$ | /files/report.pdf | /report.pdf?v=2 |
Disallow: /*? | Any URL with a query string | /shop/shoes |
Disallow: (empty) | Nothing | Everything is allowed |
Common robots.txt mistakes
- A leftover
Disallow: /from a staging site, which blocks the whole site after launch. - Blocking CSS and JavaScript, which stops Google rendering your pages properly.
- Using robots.txt to remove pages from Google. Disallow stops crawling, not indexing — a blocked URL can still appear in results if other sites link to it. Use a
noindexmeta tag or header and leave the page crawlable. - Forgetting that each host needs its own file:
shop.example.comdoes not use the robots.txt ofwww.example.com.
Need a new file? Build one with the Robots.txt Generator, and see how AI bots are treated with the AI Crawler Checker.
Tools that pair well with the Robots.txt Tester
- Sitemap Validator — Find and validate your XML sitemap — URL count, sitemap index children, lastmod coverage, duplicates and foreign-host URLs — and check that the pages it lists really return 200.
- SEO Checker & Site Audit — Paste a URL to run up to 70 on-page SEO checks in seconds.
- Meta Tag Checker — See exactly how Google is likely to display your page: title and description measured in pixels, desktop and mobile previews, and every meta tag in the <head>.
- Heading Checker — Visualise the heading outline of any page.
Frequently asked questions
Does Disallow remove a page from Google?
No. Disallow only stops crawling. A blocked URL can still be indexed (usually without a snippet) if it is linked from elsewhere. To remove a page, allow crawling and add
noindex.Which wins: Allow or Disallow?
The most specific (longest) matching path wins. If an Allow and a Disallow rule have the same length, Google applies the less restrictive Allow rule.
Is robots.txt case-sensitive?
Paths are:
Disallow: /Admin does not block /admin. Directive names like disallow and user-agent names are not case-sensitive.Where does robots.txt have to be?
At the root of each host and protocol, e.g.
https://www.example.com/robots.txt. Files in subfolders are ignored, and each subdomain needs its own.Does Google support Crawl-delay?
No. Googlebot ignores
Crawl-delay; Bing and some other crawlers respect it. If Googlebot crawls too fast, your server can temporarily return 503 or 429 responses.