How to Fix robots.txt and Sitemap Problems

What robots.txt does, and what it doesn’t

The file lives at https://example.com/robots.txt and tells crawlers which paths to skip. It controls crawling, not indexing. A blocked page can still appear in search if other sites link to it.

To keep a page out of results, use a noindex tag and leave it crawlable, because Google must read the page to see that tag.

Fix rules that block too much

This one line blocks an entire site, and it often survives from a staging setup:

User-agent: *
Disallow: /

A safe starting point for most sites looks like this:

User-agent: *
Disallow: /admin/

Sitemap: https://example.com/sitemap.xml

Don’t block your CSS or JavaScript, because Google needs them to see the page as visitors do. Test a specific address with the robots.txt Tester, or create a clean file with the robots.txt Generator.

Build a clean sitemap

A sitemap lists the pages you want found. It should only contain addresses that:

• return a 200 status,
• are the canonical version of the page,
• aren’t blocked by robots.txt or marked noindex.

One sitemap can hold up to 50,000 addresses or 50 MB, and larger sites split it into several with an index file. Generate one with the Sitemap Generator if your platform doesn’t make one.

Fix common errors

• “Sitemap could not be read”: open the address yourself. It must return XML, not an HTML page or an error.
• “Submitted URL blocked by robots.txt”: either remove the rule or remove that address from the sitemap.
• “Submitted URL not found (404)”: your sitemap is out of date, so rebuild it.
• Wrong dates: only change lastmod when a page really changes. Google ignores dates it can’t trust.

Check your file with the Sitemap Validator.

Tell search engines

Add the Sitemap: line to robots.txt, then submit the address in Google Search Console and Bing Webmaster Tools. After that, check their reports now and then for new errors.