SEO tools
Website Crawlers tools
Website crawlers request a site page by page the way a search engine would, follow what they find, and report the technical problems that cap everything else in SEO.
What Website Crawlers do
A crawler builds its own map of a site: status codes, redirect chains, canonicals, titles, headings, hreflang, indexability directives, internal link depth and (with JavaScript rendering enabled) what a bot sees after execution rather than in the source. Desktop crawlers run on your machine and your bandwidth, which makes them cheap to run repeatedly and awkward to schedule. Cloud crawlers invert both properties.
The sub-types inside this category
- Desktop crawlers Licensed per user, unlimited crawls, limited by your own hardware and RAM.
- Cloud crawlers Scheduled and shareable, priced per URL or per project.
- Log-aware crawlers Join crawl data to server logs, so requested pages can be compared against crawlable ones.
The Website Crawlers in this directory
Matches product names, former names, categories, use cases and the company behind the tool.
Filter 7 of 7
Entry price is the lowest published figure on a plan you could buy on its own, add-ons and extra seats are excluded, and a tool whose vendor publishes no figure sits under “price not published” rather than being guessed at.
| Tool | What it is for | From | Evidence | Checked |
|---|---|---|---|---|
| Lumar Website Crawlers | Crawl and monitor sites at a scale desktop and mid-market tools cannot reach | Not publicly disclosed | 19 August 2026 | |
| Screaming Frog SEO Spider Website Crawlers | Crawl a site and export every URL with its status code, canonical and indexability | Free up to 500 URLs; licence $279 per user/year | Free plan | 18 August 2026 |
| Sitebulb Website Crawlers | Produce a technical audit a client can read and act on without a specialist translating it | From $18/month (Lite); Pro $42/month | Free trial | 9 August 2026 |
| JetOctopus Log File Analysis | Join crawl data to server logs to answer crawl-budget questions properly | From $159/month for 500,000 crawl pages | Free trial | 19 August 2026 |
| SEO PowerSuite All-in-One SEO Platforms | Track unlimited keywords across unlimited projects for a fixed annual licence | Free edition; Professional $349/year | Free plan | 19 August 2026 |
| Seobility Site Auditing | Audit a small site properly without paying anything | Free for one project; Premium 49.90 €/month | Free plan | 19 August 2026 |
| Sitechecker Site Auditing | Re-crawl after every deploy without spending a quota | From $83/month billed annually; $99/month monthly | Free trial | 19 August 2026 |
Nothing matches that combination. Suggest a tool that is missing.