Sign in Submit site

crawlie

github.com/spronta/crawlie SEO & Web Analytics Open-source project

Crawlie crawls websites for technical SEO and GEO analysis, with an open-source codebase.

Visit crawlie 24 alternatives
github.com/spronta/crawlie
crawlie homepage screenshot

About crawlie

Crawlie is an open-source technical SEO and GEO crawler distributed through a GitHub repository. It crawls a whole site or audits individual pages, reporting broken links, redirects, missing metadata and 40+ SEO and generative-engine checks, each with plain-English guidance on the fix. Results are summarized as three separate scores: Health (technical SEO), GEO (AI-search readiness) and Accessibility (WCAG conformance).

Key features

  • Crawls full sites or audits single pages, respecting robots.txt and seeding from sitemap.xml
  • 40+ SEO and generative-engine checks, each with plain-English guidance on the fix
  • Separate Health, GEO and Accessibility scores reported for each crawl
  • Output as pretty terminal text, JSON, CSV issues list or shareable HTML report
  • MCP server (crawlie-mcp) enabling agent-driven audits over JSON-RPC stdio
  • Flags for page cap, depth, concurrency, URL scoping and CI exit codes
Best for: Developers, SEO practitioners and agent builders who want a locally run, scriptable technical SEO and AI-search audit.
Good for
  • Free to run locally; the project describes its codebase as open-source
  • Reports technical SEO, generative-engine and accessibility findings as separate scores
  • JSON and CSV output plus a non-zero exit option support scripting and CI gating
Watch out for
  • Crawls default to a 500-page cap and 16 parallel requests, adjustable only through flags
  • npm is the delivery channel for the CLI and MCP server; the macOS desktop app is the only direct download
  • The optional hosted Cloud MCP requires an API key and runs crawls on the project's infrastructure rather than locally

crawlie at a glance

Type
Open-source project
Pricing
Not confirmed. Check the site
Open source
Not stated
Language
English
Last checked
Oct 6, 2026 · Online when checked
Listed
Oct 5, 2026
web analyticscrawlertechnical seoaeo

crawlie alternatives

Tools and sites you could use instead, ranked by how closely they match.

All 24 alternatives

Popular in SEO & Web Analytics

Well-known listings from the same topic.

All 211
MS Clarityclarity.microsoft.comGoogle Consolesearch.google.com/search-consoleGoogle Search Consolesearch.google.com/search-console/aboutCloudflare Web Analyticscloudflare.com/web-analyticsBing Webmaster Toolsbing.com/webmasters/aboutBing Webmaster Toolsbing.com/toolbox/webmaster
Related topicsSocial Media Tools

crawlie FAQ

How is Crawlie installed?
The CLI and MCP server install with 'npm i -g crawlie'. A signed macOS .dmg is available from Releases, and building from source requires Rust, plus pnpm and Node for the desktop app.
Can an agent run a crawl without a human in the loop?
Yes. crawlie-mcp ships with the npm package and can be added to MCP-compatible clients such as Claude Desktop, Claude Code, Cursor or Cline.
What is the difference between the local MCP server and Crawlie Cloud MCP?
The local server runs on your machine and speaks JSON-RPC over stdio. The hosted endpoint at https://crawlie.app/mcp uses MCP Streamable HTTP, requires a Crawlie API key, and runs crawls on the project's infrastructure.
Is something wrong or out of date?

Tell us and an editor will check it.

Own this website?

Claim the listing and suggest updates yourself. An editor checks every claim first.

Claim this listing

Listing maintained by Chris McClure and the WebIndexly editors · Updated Oct 11, 2026 · How we rank