Como usar: Website Crawler
- Enter your homepage (or any start URL) and choose how many pages to crawl — 30 is a quick health check, 100 covers most small sites.
- Click Start crawl. Pages appear live as they are fetched; press Stop at any time to keep the results so far.
- Click an issue chip such as Duplicate title or Missing H1 to see only the affected pages.
- Export the full crawl as CSV to plan fixes or share with your team.
What is a website crawler?
A website crawler (or spider) works like a search engine bot: it loads a start page, collects the internal links, visits those pages, collects their links, and so on. Along the way it records each page’s status code and on-page SEO elements, so problems that repeat across many URLs — duplicate titles, missing descriptions, broken templates — become visible at a glance.
How this crawler works
- It starts at the URL you enter and follows
<a href>links that stay on the same domain (with or withoutwww). - Up to four pages are fetched at a time, so the crawl is gentle on your server.
- Files such as PDFs, images, videos and archives are skipped; only HTML pages are analyzed.
- It reads the raw HTML your server returns. Links that only appear after JavaScript runs are not discovered — if your crawl stops after one page, your navigation is probably rendered client-side.
- It respects robots.txt: it identifies itself as
DevPackBot, follows the rules forDevPackBot(or*if there’s no specific group), lists blocked URLs as “Blocked by robots.txt” instead of fetching them, and honorsCrawl-delay(up to 5 seconds) by fetching one page at a time. If robots.txt returns a server error, it doesn’t crawl at all, as RFC 9309 requires. To see how Googlebot or other bots are treated, use the Robots.txt Tester. - It discovers pages through links only. Pages that appear only in your sitemap aren’t visited — compare with the Sitemap Validator.
Issues the crawler reports
| Issue | Why it matters | Typical fix |
|---|---|---|
| Broken page | Dead ends for users and wasted crawl budget | Restore the page or 301-redirect it, and update links |
| Redirect | Internal links that hop add latency | Link straight to the final URL |
| Missing / duplicate title | Titles are the main ranking and click signal in search results | Write a unique 30–60 character title per page |
| Missing / duplicate description | Google may generate a worse snippet | Write a unique summary of roughly 120–160 characters |
| Missing / multiple H1 | Unclear main topic for users and screen readers | Use one descriptive H1 per page |
| Noindex | The page is excluded from search results | Remove the tag if the page should rank |
| Thin content | Under 300 words rarely answers a query fully | Expand, merge or noindex low-value pages |
| Images without alt | Accessibility and image search | Add descriptive alt text (use alt="" for decorative images) |
| Slow page | Over 1.5 s to download the HTML hurts every later metric | Caching, a CDN, lighter templates |
Word counts include all visible text in the page body, including navigation and footer, so treat “thin content” as a prompt to review rather than a verdict.
Tips for a useful crawl
Start with a small limit to check the crawler can follow your navigation, then run a bigger crawl. Fix issues by template rather than page by page — one wrong title pattern often explains hundreds of duplicates. For an in-depth look at a single URL, use the SEO Checker; to verify every link on one page, use the Broken Link Checker.
Também pesquisado como: seo crawler, site audit, spider, crawl website, technical seo audit, duplicate titles, screaming frog alternative, site crawler.
Ferramentas que combinam com: Website Crawler
- Sitemap Generator — Crawl your website and get a clean sitemap.xml containing only pages that deserve to be indexed — 200 status, no noindex, self-canonical — with every excluded URL and the reason listed.
- Meta Tag Checker — See exactly how Google is likely to display your page: title and description measured in pixels, desktop and mobile previews, and every meta tag in the <head>.
- Heading Checker — Visualize the heading outline of any page.
- Guia explicativo sobre o robots.txt — As regras do robots.txt são curtas, mas fáceis de interpretar mal.
Perguntas frequentes
Does the crawler render JavaScript?
No. It analyzes the HTML your server sends, which is also what search engines see first. Pages and links that only exist after client-side rendering will not be found.
How many pages can I crawl?
Up to 100 pages per crawl. Pick a smaller limit for a quick check; you can stop a crawl at any time and still export what was collected.
Does the crawler respect robots.txt?
Yes. It crawls as DevPackBot and follows the robots.txt rules for DevPackBot (or the * group), skips disallowed URLs and honors Crawl-delay. If you want to block it, add “User-agent: DevPackBot” with “Disallow: /”.
Why did it find fewer pages than my site has?
The crawler only finds pages that are linked from other crawled pages. Orphan pages (only in your sitemap), links added by JavaScript, and links to other subdomains are not followed. Compare against your sitemap with the Sitemap Validator.
Will crawling slow down my website?
It fetches at most four pages at a time (one at a time if your robots.txt sets a Crawl-delay), which is comparable to a single visitor browsing quickly. Each crawl is capped at 100 pages.
Is this a replacement for desktop crawlers like Screaming Frog?
It covers the most common on-page and status checks without installing anything, which is ideal for quick audits. For very large sites, JavaScript rendering or custom extraction, a dedicated desktop crawler is still the better choice.