devpack
EnglishDeutsch · StartseiteFrançais · AccueilEspañol · InicioPortuguês (Brasil) · Início
Auditoría web

Esta página aún no está traducida al español, así que la mostramos en inglés.

Robots.txt Tester

Check whether a URL is allowed or blocked for Googlebot, Bingbot, GPTBot and other crawlers, see exactly which rule matched, and edit the file to test changes before you deploy them.

Actualizado el

Enter a site URL. Optionally add a path and user-agent to test.

Enter any public URL above and press Enter or Ctrl + Enter.

  • Allowed / Blocked verdict with the exact matching rule
  • Access matrix for Google, Bing and AI crawlers
  • Declared sitemaps and syntax problems
  • Edit the file and re-test before you deploy

Esta herramienta consulta desde nuestro servidor la URL pública que introduces. Los resultados se guardan unos minutos en memoria temporal y nunca en una base de datos.

Cómo usar: Robots.txt Tester

  1. Enter your site URL. To test a specific page, enter its full URL or type the path (e.g. /checkout?step=2) in the path box.
  2. Pick a user-agent such as Googlebot, Bingbot or GPTBot — or type any other.
  3. Click Test to see whether the path is allowed or blocked, which rule matched, and how other crawlers are treated.
  4. Edit the robots.txt in the editor and click Test edited version to try changes safely, then copy or download the result.

What is a robots.txt tester?

robots.txt is a plain-text file at the root of a host (https://example.com/robots.txt) that tells crawlers which paths they may request. A tester fetches that file, parses it the way search engines do, and answers one question precisely: may this crawler fetch this URL? Since Google retired the robots.txt Tester in Search Console in 2023, a standalone tester is the quickest way to check rules before and after a deployment.

How rules are matched

Matching follows Google’s documented behavior and the robots exclusion standard (RFC 9309):

  1. Pick the group. A crawler obeys only the group whose User-agent matches it most specifically. If Googlebot has its own group, the User-agent: * rules are ignored for it entirely.
  2. Find matching rules. Each Allow and Disallow path is matched from the start of the URL path, including the query string. * matches any characters and $ anchors the end.
  3. Longest match wins. The rule with the longest path decides. If an Allow and a Disallow are equally long, Allow wins. No matching rule means allowed.
RuleBlocksDoes not block
Disallow: /admin/admin, /admin/users, /administrator/Admin (paths are case-sensitive)
Disallow: /*.pdf$/files/report.pdf/report.pdf?v=2
Disallow: /*?Any URL with a query string/shop/shoes
Disallow: (empty)NothingEverything is allowed

Common robots.txt mistakes

  • A leftover Disallow: / from a staging site, which blocks the whole site after launch.
  • Blocking CSS and JavaScript, which stops Google rendering your pages properly.
  • Using robots.txt to remove pages from Google. Disallow stops crawling, not indexing — a blocked URL can still appear in results if other sites link to it. Use a noindex meta tag or header and leave the page crawlable.
  • Forgetting that each host needs its own file: shop.example.com does not use the robots.txt of www.example.com.

Need a new file? Build one with the Robots.txt Generator, and see how AI bots are treated with the AI Crawler Checker.

También se busca como: robots.txt checker, robots.txt validator, is url blocked, disallow tester, googlebot blocked, robots txt test, crawl test, user-agent.

Herramientas que combinan con: Robots.txt Tester

  • Sitemap Validator — Find and validate your XML sitemap — URL count, sitemap index children, lastmod coverage, duplicates and foreign-host URLs — and check that the pages it lists really return 200.
  • Website Crawler — Start from your homepage and crawl up to 100 internal pages to uncover site-wide SEO issues — broken pages, duplicate titles, missing descriptions, H1 problems, noindex tags and slow pages — in one live report.
  • Verificador SEO y auditoría del sitio web — Pega una URL para realizar hasta 70 comprobaciones de SEO on-page, además de una auditoría completa del sitio web: velocidad de Lighthouse, encabezados de seguridad y SSL, datos estructurados, enlaces rotos, robots.txt y rastreadores de IA, aspectos básicos de accesibilidad, seguimiento y consentimiento, SPF/DKIM/DMARC y caducidad del dominio — todo ello resumido en una puntuación global del sitio web con soluciones priorizadas.
  • Guía sobre el archivo robots.txt — Las reglas de robots.txt son breves, pero es fácil malinterpretarlas.

Preguntas frecuentes

Does Disallow remove a page from Google?
No. Disallow only stops crawling. A blocked URL can still be indexed (usually without a snippet) if it is linked from elsewhere. To remove a page, allow crawling and add noindex.
Which wins: Allow or Disallow?
The most specific (longest) matching path wins. If an Allow and a Disallow rule have the same length, Google applies the less restrictive Allow rule.
Is robots.txt case-sensitive?
Paths are: Disallow: /Admin does not block /admin. Directive names like disallow and user-agent names are not case-sensitive.
Where does robots.txt have to be?
At the root of each host and protocol, e.g. https://www.example.com/robots.txt. Files in subfolders are ignored, and each subdomain needs its own.
Does Google support Crawl-delay?
No. Googlebot ignores Crawl-delay; Bing and some other crawlers respect it. If Googlebot crawls too fast, your server can temporarily return 503 or 429 responses.

Herramientas relacionadas

localRobots.txt GeneratorBuild a correct robots.txt in seconds: choose a preset, add rules per user-agent, block AI training…apiAI Crawler CheckerSee which AI bots — from OpenAI, Anthropic, Google, Perplexity, Apple, Meta and others — your robots.txt…apiSitemap ValidatorFind and validate your XML sitemap — URL count, sitemap index children, lastmod coverage, duplicates and…apiWebsite CrawlerStart from your homepage and crawl up to 100 internal pages to uncover site-wide SEO issues — broken…apiVerificador SEO y auditoría del sitio webPega una URL para realizar hasta 70 comprobaciones de SEO on-page, además de una auditoría completa del…localGuía sobre el archivo robots.txtLas reglas de robots.txt son breves, pero es fácil malinterpretarlas. Pega un archivo (o descárgalo de una…