Crawler

SpiderSuite is a cross-platform web crawler designed for security professionals.

Awesome Web Scraping collects web scraping libraries, tools, APIs, and manuals across programming languages.

Scrapling provides adaptive web scraping for tasks ranging from single requests to full-scale crawls.

Collect and structure website data with Octoparse, a no-code web scraping tool.

CreatorHub provides a web panel for monitoring, collecting, and transferring content across Douyin, Xiaohongshu, and Kuaishou.

Crawl websites, documentation and local files to find broken links and invalid URLs with this link checker.

High-performance asynchronous Douyin(抖音) TikTok Xiaohongshu(小红书) Kuaishou(快手) Weibo(微博) Instagram YouTube(油管) Twitter(X) Captcha Solver(验证码解决器) Temp Mail(临时邮箱) API(接口).

grab-site crawls websites for archiving, saves output as WARC, and provides a dashboard and dynamic ignore patterns.

Analyze websites with SEO tools that include a site crawler, keyword research and backlink analysis.

Crawlie crawls websites for technical SEO and GEO analysis, with an open-source codebase.

Crawl websites for SEO audits on Windows, macOS and Ubuntu using the Screaming Frog SEO Spider.

Heritrix is the Internet Archive’s open-source, extensible web crawler for archival collection.

Crawly extracts structured content from websites by crawling pages across a site.

Crawl websites for SEO audits with this free, open-source tool, which supports JavaScript rendering and unlimited URLs.

SpiderSuite provides web crawling, interception, and analysis tools for security testing.

Crawl websites and run technical SEO audits with SiteAnalyzer, which supports multiple interface languages.

Brozzler crawls the web through a distributed browser-based crawler project from the Internet Archive.

Crawl websites for SEO audits with desktop or cloud software that checks technical site structure.

Marginalia Search indexes text-oriented websites, including older and less prominent pages from across the web.

Nostr Crawler fetches, backs up, and broadcasts Nostr events, with filters for authors, event kinds, and relays.

Play a randomized dungeon crawler with puzzles that changes each day in Daily Dungeon.