499 Open source projects that are alternatives of or similar to Strong Web Crawler

一个用来快速提取网页内容的信息采集（爬虫）框架， 实现了对网页的动态加载与控制。

新一代爬虫平台，以图形化方式定义爬虫流程，不写代码即可完成爬虫。

The simple, easy to use command line web crawler.

A collection of awesome web scaper, crawler.

A collection of awesome web crawler,spider in different languages

Distributed web crawler admin platform for spiders management regardless of languages and frameworks. 分布式爬虫管理平台，支持任何语言和框架

Android应用市场网络爬虫

A simple but powerful web crawler library for .NET

An interactive Command-Line Interface Build in NodeJS for downloading a single or multiple images to disk from URL

A web crawler. Supercrawler automatically crawls websites. Define custom handlers to parse content. Obeys robots.txt, rate limits and concurrency limits.

Parser and database to index the terpene profile of different strains of Cannabis from online databases

简单易用的Python爬虫框架，QQ交流群：597510560

When you solve the problem of Baekjoon Online Judge, it automatically commits and pushes to the remote repository.

Antch, a fast, powerful and extensible web crawling & scraping framework for Go

Continuous scalable web crawler built on top of Flink and crawler-commons

Universal scrapping tool, which allows you to extract data using multiple environments

Easy way to brute-force web directory.

Rust Web Crawler saving pages on Redis

[WIP] GOPA, a spider written in Golang, for Elasticsearch. DEMO: http://index.elasticsearch.cn

A simple distributed crawler for zhihu && data analysis

📷 Automate full website screenshots and PDF generation with multiple viewport support.

A versatile Ruby web spidering library that can spider a site, multiple domains, certain links or infinitely. Spidr is designed to be fast and easy to use.

Lite version of Crawlab. 轻量版 Crawlab 爬虫管理平台

Cross Platform C# web crawler framework built for speed and flexibility. Please star this project! +1.

以Node.js基于express以及爬虫实现的视频资源后端

Web crawler.

golang light-weight image crawler

多线程知乎用户爬虫，基于python3

京东口罩库存监控爬虫(非selenium)，扫码登录、查价、加购、下单、秒杀

A Laravel wrapper for CrawlerDetect - the web crawler detection library

Gorecon is a All in one Reconnaissance Tool , a.k.a swiss knife for Reconnaissance , A tool that every pentester/bughunter might wanna consider into their arsenal

🍿爬虫代理IP池(proxy pool) python🍟一个还ok的IP代理池

稳健高效的评分制-针对性- IP代理池 + API服务，可以自己插入采集器进行代理IP的爬取，针对你的爬虫的一个或多个目标网站分别生成有效的IP代理数据库，支持MongoDB 4.0 使用 Python3.7（Scored IP proxy pool ,customise proxy data crawler can be added anytime）

Headless Chrome For Java （Java 爬虫）

Capture screenshot of a website and return it as a stream

A Swift Web Crawler 🕷

A Tumblr Backup Application

Apache Nutch is an extensible and scalable web crawler

Get (almost) original messages from google group archives. Your data is yours.

Github 仓库及用户分析爬虫

Easy to use lightweight web crawler（易用的轻量化网络爬虫）

Crawl all unique internal links found on a given website, and extract SEO related information - supports javascript based sites

Async WebDriver implementation for asyncio and asyncio-compatible frameworks

[Crawler/Scraper for Golang]🕷A lightweight distributed friendly Golang crawler framework.一个轻量的分布式友好的 Golang 爬虫框架。

💐Marmot | Web Crawler/HTTP protocol Download Package 🐭

Website style analyzer for designers. This is the service part that does the analyzation etc.

An image crawler written in Python.

基于 Selenium 的知乎关键词爬虫

A Kotlin-based testing/scraping/parsing library providing the ability to analyze and extract data from HTML (server & client-side rendered). It places particular emphasis on ease of use and a high level of readability by providing an intuitive DSL. It aims to be a testing lib, but can also be used to scrape websites in a convenient fashion.

Dynamic file detection tool based on crawler 基于爬虫的动态敏感文件探测工具

Proxy [Finder | Checker | Server]. HTTP(S) & SOCKS 🎭

Opensource Korean chatbot framework

🐝 Web vertical crawler framework for fun

Official Algolia Plugin for Netlify. Index your website to Algolia when deploying your project to Netlify with the Algolia Crawler

链家网和贝壳网房价爬虫，采集北京上海广州深圳等21个中国主要城市的房价数据（小区，二手房，出租房，新房），稳定可靠快速！支持csv,MySQL, MongoDB,Excel, json存储，支持Python2和3，图表展示数据，注释丰富 ，点星支持，仅供学习参考，请勿用于商业用途，后果自负。

Collection of China illegal cases about web crawler 本项目用来整理所有中国大陆爬虫开发者涉诉与违规相关的新闻、资料与法律法规。致力于帮助在中国大陆工作的爬虫行业从业者了解我国相关法律，避免触碰数据合规红线。 [AD]中文知识图谱门户

crawler framework, distributed crawler extractor

Scrapes all photos and videos in a web page / Instagram / Twitter / Tumblr / Reddit / pixiv / TikTok

An API for your Kindle data

A cross platform UI crawler which scans view trees then generate and execute UI test cases.