502 Open source projects that are alternatives of or similar to dcard-spider

Light-weight header-only library for parallel function calls and continuations in C++ based on Eric Niebler's talk at CppCon 2019.

一个java版本的分布式的通用爬虫，可以插拔各个组件（提供默认的）

直接通过链家 API 抓取数据的极速爬虫，宇宙最快~~ 🚀

A C++11 thread pool and profiler

dynamic crawler for web vulnerability scanner

一个简单的基于DHT协议的BT磁力链接爬虫

A faster RWLock primitive in Go, 2-3 times faster than RWMutex. A Go implementation of concurrency control algorithm in paper <Left-Right - A Concurrency Control Technique with Wait-Free Population Oblivious Reads>

微信公众号爬虫

A spiders' program for weixin which made by Express & cheerio

LSpider 一个为被动扫描器定制的前端爬虫

犇犇爬虫

基于python语言开发的千万级别搜索引擎

A spider tapestry weaver

Concurrent computing in Julia based on the Actor Model

爬虫集合

This taobao spider has been archived

web spider built by puppeteer, support task-queue and task-scheduling by decorators，support nedb / mongodb, support data visualization; 基于puppeteer的web爬虫框架，提供灵活的任务队列管理调度方案，提供便捷的数据保存方案（nedb/mongodb），提供数据可视化和用户交互的实现方案

📌 强大的动态线程池框架，附带监控报警功能。支持 JDK、Tomcat、Jetty、Undertow 线程池；Dubbo、Dubbox、RabbitMQ、RocketMQ、Hystrix 消费线程池（更多框架线程池还在适配中）。内置两种使用模式：轻量级依赖配置中心以及无中间件依赖版本。

admin ui for scrapy/open source scrapinghub

gathertool是golang脚本化开发库，目的是提高对应场景程序开发的效率；轻量级爬虫库，接口测试&压力测试库，DB操作库等。

京东口罩库存监控爬虫(非selenium)，扫码登录、查价、加购、下单、秒杀

资讯爬虫App

BitTorrent DHT Protocol && DHT Spider.

⚡ Light weight Golang spider framework | 轻量的 Golang 爬虫框架

一个功能全面的微信读书爬虫 wereader

scala结合actor编写的分布式网络爬虫

JavaScript + BeautifulSoup = JSSoup

scrapy spider for scraping imdb {movie_id: [recommended, ...]}

Crawling GitHub Trending Pages every day

零基础学习python爬虫

基于Electron爬虫程序

Lightweight, Generic, Pure C++11 ThreadPool

Dynamic configurable crawl (动态可配置化爬虫)

Google search results crawler, get google search results that you need

🕷️ A simple crawler (spider) writen in php just for fun, with zero dependencies

🔞 JAVClub - 让你的大姐姐不再走丢

🕷️用于爬取B站前top100的小视频

A Penetration Testing Framework, Information gathering tool & Website Vulnerability Scanner

Grapy, a fast high-level web crawling framework for Python 3.3 or later base on asyncio.

招聘网数据爬虫

裁判文书网爬虫

A Laravel wrapper for CrawlerDetect - the web crawler detection library

⭐ 图虫网爬虫

sync playlist between music platform

Jet is a simple OOP, dynamically typed, functional language that runs on the Erlang virtual machine (BEAM). Jet's syntax is Ruby-like syntax.

Web crawler.

爬虫工程师常用的 Chrome 插件 | Chrome extensions used by crawler developer

Bilibili.com视频批量下载工具包

🐴 Distribute load on front-end via parallelism

Distributed Crawler Management Framework Based on Scrapy, Scrapyd, Django and Vue.js

All about Node.js

小说精品屋是一个多平台（web、安卓app、微信小程序）、功能完善的屏幕自适应小说漫画连载系统，包含精品小说专区、轻小说专区和漫画专区。包括小说/漫画分类、小说/漫画搜索、小说/漫画排行、完本小说/漫画、小说/漫画评分、小说/漫画在线阅读、小说/漫画书架、小说/漫画阅读记录、小说下载、小说弹幕、小说/漫画自动采集/更新/纠错、小说内容自动分享到微博、邮件自动推广、链接自动推送到百度搜索引擎等功能。

项目已经移动至：https://github.com/BaiduSpider/BaiduSpider ！！ 一个爬取百度搜索结果的爬虫，目前支持百度网页搜索，百度图片搜索，百度知道搜索，百度视频搜索，百度资讯搜索，百度文库搜索，百度经验搜索和百度百科搜索。

Elegant Scraper and Crawler Framework for Golang

基于nodejs的网络聊天室、爬虫，vue音乐播放器，及php后台开发的管理系统等项目

🕷️ The progressive PHP crawler framework! 优雅的渐进式PHP采集框架。

超高速异步协程Python爬虫

An easy to use C++ Thread Pool

minimal utilities for Scala Futures

文章采集工具 Article collection tool