How to use the Website Crawler
- Enter your homepage (or any start URL) and choose how many pages to crawl — 30 is a quick health check, 500 covers most small and mid-size sites.
- Click Start crawl. Pages appear live as they are fetched; press Stop at any time to keep the results so far.
- Click an issue chip such as Duplicate title or Missing H1 to see only the affected pages.
- Export the full crawl as CSV to plan fixes or share with your team.
What is a website crawler?
A website crawler (or spider) works like a search engine bot: it loads a start page, collects the internal links, visits those pages, collects their links, and so on. Along the way it records each page’s status code and on-page SEO elements, so problems that repeat across many URLs — duplicate titles, missing descriptions, broken templates — become visible at a glance.
How this crawler works
- It starts at the URL you enter and follows
<a href>links that stay on the same domain (with or withoutwww). - Up to four pages are fetched at a time, so the crawl is gentle on your server.
- Files such as PDFs, images, videos and archives are skipped; only HTML pages are analysed.
- It reads the raw HTML your server returns. Links that only appear after JavaScript runs are not discovered — if your crawl stops after one page, your navigation is probably rendered client-side.
- It discovers pages through links only and does not apply robots.txt rules, so you also see pages Googlebot may be blocked from. Check those with the Robots.txt Tester.
Issues the crawler reports
| Issue | Why it matters | Typical fix |
|---|---|---|
| Broken page | Dead ends for users and wasted crawl budget | Restore the page or 301-redirect it, and update links |
| Redirect | Internal links that hop add latency | Link straight to the final URL |
| Missing / duplicate title | Titles are the main ranking and click signal in search results | Write a unique 30–60 character title per page |
| Missing / duplicate description | Google may generate a worse snippet | Write a unique summary of roughly 120–160 characters |
| Missing / multiple H1 | Unclear main topic for users and screen readers | Use one descriptive H1 per page |
| Noindex | The page is excluded from search results | Remove the tag if the page should rank |
| Thin content | Under 300 words rarely answers a query fully | Expand, merge or noindex low-value pages |
| Images without alt | Accessibility and image search | Add descriptive alt text (use alt="" for decorative images) |
| Slow page | Over 1.5 s to download the HTML hurts every later metric | Caching, a CDN, lighter templates |
Word counts include all visible text in the page body, including navigation and footer, so treat “thin content” as a prompt to review rather than a verdict.
Tips for a useful crawl
Start with a small limit to check the crawler can follow your navigation, then run a bigger crawl. Fix issues by template rather than page by page — one wrong title pattern often explains hundreds of duplicates. For an in-depth look at a single URL, use the SEO Checker; to verify every link on one page, use the Broken Link Checker.
Tools that pair well with the Website Crawler
- Meta Tag Checker — See exactly how Google is likely to display your page: title and description measured in pixels, desktop and mobile previews, and every meta tag in the <head>.
- Heading Checker — Visualise the heading outline of any page.
- Keyword Density Checker — Enter a URL and the keyword it should rank for.
- Open Graph Checker — Preview how your link will look when shared on Facebook, LinkedIn, X and Slack.
Frequently asked questions
Does the crawler render JavaScript?
No. It analyses the HTML your server sends, which is also what search engines see first. Pages and links that only exist after client-side rendering will not be found.
How many pages can I crawl?
Up to 500 pages per crawl. Pick a smaller limit for a quick check; you can stop a crawl at any time and still export what was collected.
Why did it find fewer pages than my site has?
The crawler only finds pages that are linked from other crawled pages. Orphan pages (only in your sitemap), links added by JavaScript, and links to other subdomains are not followed. Compare against your sitemap with the Sitemap Validator.
Will crawling slow down my website?
It fetches at most four pages at a time, which is comparable to a single visitor browsing quickly. Each crawl is capped at 500 pages.
Is this a replacement for desktop crawlers like Screaming Frog?
It covers the most common on-page and status checks without installing anything, which is ideal for quick audits. For very large sites, JavaScript rendering or custom extraction, a dedicated desktop crawler is still the better choice.