2392 Open source projects that are alternatives of or similar to Abot

Cross Platform C# Web crawler framework, headless browser, parallel crawler. Please star this project! +1.

Lite version of Crawlab. 轻量版 Crawlab 爬虫管理平台

A simple distributed crawler for zhihu && data analysis

[WIP] GOPA, a spider written in Golang, for Elasticsearch. DEMO: http://index.elasticsearch.cn

Rust Web Crawler saving pages on Redis

新一代爬虫平台，以图形化方式定义爬虫流程，不写代码即可完成爬虫。

The easiest and fastest way to work with sockets in C#

A versatile Ruby web spidering library that can spider a site, multiple domains, certain links or infinitely. Spidr is designed to be fast and easy to use.

Distributed web crawler admin platform for spiders management regardless of languages and frameworks. 分布式爬虫管理平台，支持任何语言和框架

Continuous scalable web crawler built on top of Flink and crawler-commons

简单易用的Python爬虫框架，QQ交流群：597510560

A collection of awesome web crawler,spider in different languages

🐾 Creeper - The Next Generation Crawler Framework (Go)

Scraping scripts for various websites.

[DEPRECATED] Simple, flexible, delightful web crawler/spider package

💐 Full Amazon Automatic Download

A cross-platform C and C++ unit testing framework for the 21st century

Golang爬虫 爬取汽车之家 二手车产品库

自学入门 Python 优质中文资源索引，包含 书籍 / 文档 / 视频，适用于 爬虫 / Web / 数据分析 / 机器学习 方向

Beanbun 是用 PHP 编写的多进程网络爬虫框架，具有良好的开放性、高可扩展性，基于 Workerman。

python crawler spider

AdoNetCore.AseClient - a .NET Core DB Provider for SAP ASE

a puppeteer walker 🕷 🕸

一个基于 .NET Core构建的简单、跨平台、模块化的商城系统

DncZeus 是一个基于ASP.NET Core 3 + Vue.js(iview-admin) 的前后端分离的通用后台权限(页面访问、操作按钮控制)管理系统框架。后端使用.NET Core 3 + Entity Framework Core构建，UI则是目前流行的基于Vue.js的iView(iview-admin)。项目实现了前后端的动态权限管理和控制以及基于JWT的用户令牌认证机制，让前后端的交互更流畅。码云镜像:https://gitee.com/rector/DncZeus 。演示地址(demo):

ng-alain 与 .net core 搭配

A simple but powerful web crawler library for .NET

Python爬虫实战 - 模拟登陆各大网站 包含但不限于：滑块验证、拼多多、美团、百度、bilibili、大众点评、淘宝，如果喜欢请start ❤️

Remote Encrypted Procedure Calling for .Net & .Net Core

zhihu-crawler是一个基于Java的高性能、支持免费http代理池、支持横向扩展、分布式爬虫项目

Dark Web OSINT Tool

Incredibly fast crawler designed for OSINT.

AV 电影管理系统， avmoo , javbus , javlibrary 爬虫，线上 AV 影片图书馆，AV 磁力链接数据库，Japanese Adult Video Library,Adult Video Magnet Links - Japanese Adult Video Database

Gospider - Fast web spider written in Go

Powerful web scraping framework for Crystal

yet another csharp expression parser

Some classic web crawler projects.一些经典的爬虫

Parser and database to index the terpene profile of different strains of Cannabis from online databases

Every programmer needs a rubberduck. COM add-in for the VBA & VB6 IDE (VBE).

Geziyor, a fast web crawling & scraping framework for Go. Supports JS rendering.

GOPA, a spider written in Go.（NOTE: this project moved to https://github.com/infinitbyte/gopa ）

A high performance web crawler in Elixir.

A collection of awesome web scaper, crawler.

🕷 CrawlerDetect is a PHP class for detecting bots/crawlers/spiders via the user agent

蓝天采集器是一款免费的数据采集发布爬虫软件，采用php+mysql开发，可部署在云服务器，几乎能采集所有类型的网页，无缝对接各类CMS建站程序，免登录实时发布数据，全自动无需人工干预！是网页大数据采集软件中完全跨平台的云端爬虫系统

BaiduSpider，一个爬取百度搜索结果的爬虫，目前支持百度网页搜索，百度图片搜索，百度知道搜索，百度视频搜索，百度资讯搜索，百度文库搜索，百度经验搜索和百度百科搜索。

A web crawler (for bug hunting) that gathers more than you can imagine.

Cross platform .Net Core applications with a webview UI

Golang爬虫 爬取豆瓣电影Top250

Expressive is a cross-platform expression parsing and evaluation framework. The cross-platform nature is achieved through compiling for .NET Standard so it will run on practically any platform.

B站用户爬虫 好耶~是爬虫

一些非常有趣的python爬虫例子,对新手比较友好,主要爬取淘宝、天猫、微信、豆瓣、QQ等网站。(Some interesting examples of python crawlers that are friendly to beginners. )

Async Python 3.6+ web scraping micro-framework based on asyncio

爬取北大法宝网http://www.pkulaw.cn/Case/

GenVue is a hostable, web application that lets confidential users upload and share private files. Tech stack: Net Core 2.0 + Vue.js + Vuex + OpenIddict + Vuetifyjs + EF + SQLServer/Postgress

APIs for loginning some websites by using requests.

微博超级话题爬虫，微博词频统计+情感分析+简单分类，新增肺炎超话爬取数据

Digger is a powerful and flexible web crawler implemented by pure golang

[爬虫框架 (golang)] An awesome Go concurrent Crawler(spider) framework. The crawler is flexible and modular. It can be expanded to an Individualized crawler easily or you can use the default crawl components only.

MM131网站图片爬取 🚨