SEO Site Crawler

Crawl any website to audit SEO, links and metadata

Crawl any public website and discover every page, internal link, title tag, meta description, status code, canonical URL and Open Graph tag. The crawler walks your site like a search engine bot would, giving you a complete picture of how a real crawler sees your structure.

SEO health depends on dozens of small details — broken links, missing titles, duplicate metadata, orphan pages, redirect chains and inconsistent canonicals. Spotting all of these manually is impossible on any site bigger than a few pages. The Site Crawler surfaces them in minutes.

Use it on your own site for routine SEO maintenance, on a competitor for analysis, or on a client site for an audit deliverable. Output is exportable so you can take findings into spreadsheets or reports.

Why use Site Crawler?

Full‑site discovery

Recursively crawls internal links to discover every page on the site.

SEO‑critical metadata

Captures titles, descriptions, canonicals, robots, headings and Open Graph tags.

Status code audit

Spots broken links, redirect chains and unexpected 4xx/5xx responses.

Exportable results

Download crawl results for inclusion in audits and reports.

How to use Site Crawler

1
Enter the start URL
Paste the homepage or any starting page of the site.
2
Set crawl depth
Pick how many link levels deep the crawler should walk.
3
Run the crawl
Watch pages discovered in real time.
4
Review and export
Inspect issues, then export results for further analysis.

Common use cases

SEO audit

Find missing titles, weak metadata, broken links and orphan pages.

Pre‑launch QA

Verify a new site before going live by crawling staging.

Competitor research

Map a competitor's site structure and content focus.

Migration planning

Discover every URL that needs redirecting before a re‑platform.

Features at a glance

Recursive internal link discovery
Title, meta and OG capture
Status code reporting
Configurable crawl depth
Respects robots.txt
Exportable results
Clean tabular UI
Works on any public site

Frequently asked questions

How do I check my website for broken links?

Enter your website URL in our SEO Site Crawler and click Start Crawl. It finds all broken links (404 errors), server errors (5xx), and reports them with the pages they appear on.

Can I find missing meta descriptions with the site crawler?

Yes. Our crawler checks every page for missing or duplicate title tags, meta descriptions, H1 tags, and other on-page SEO elements.

How many pages can the site crawler audit?

Our crawler audits up to several hundred pages per crawl. Large sites can be crawled in sections by specifying a starting URL.

Is the SEO site crawler free to use?

Yes. Our site crawler is completely free with no usage limits, sign-up, or API keys required.

Your privacy matters

Site Crawler runs entirely in your web browser. Your files, text, and data are never uploaded to any server, never stored, and never seen by us or any third party. This client-side architecture means you can safely process confidential business documents, personal records, and proprietary data without privacy concerns.