1017 Open source projects that are alternatives of or similar to Crawlab Lite

Distributed web crawler admin platform for spiders management regardless of languages and frameworks. 分布式爬虫管理平台，支持任何语言和框架

Rust Web Crawler saving pages on Redis

新一代爬虫平台，以图形化方式定义爬虫流程，不写代码即可完成爬虫。

[Crawler/Scraper for Golang]🕷A lightweight distributed friendly Golang crawler framework.一个轻量的分布式友好的 Golang 爬虫框架。

[WIP] GOPA, a spider written in Golang, for Elasticsearch. DEMO: http://index.elasticsearch.cn

Python爬虫实战 - 模拟登陆各大网站 包含但不限于：滑块验证、拼多多、美团、百度、bilibili、大众点评、淘宝，如果喜欢请start ❤️

A collection of awesome web scaper, crawler.

A simple distributed crawler for zhihu && data analysis

Continuous scalable web crawler built on top of Flink and crawler-commons

Cross Platform C# web crawler framework built for speed and flexibility. Please star this project! +1.

ScrapingOutsourcing专注分享爬虫代码 尽量每周更新一个

A Facebook crawler

A collection of awesome web crawler,spider in different languages

💖 High available distributed ip proxy pool, powerd by Scrapy and Redis

简单易用的Python爬虫框架，QQ交流群：597510560

A versatile Ruby web spidering library that can spider a site, multiple domains, certain links or infinitely. Spidr is designed to be fast and easy to use.

💐Marmot | Web Crawler/HTTP protocol Download Package 🐭

A multi-thread crawler framework with many builtin image crawlers provided.

Parser and database to index the terpene profile of different strains of Cannabis from online databases

zhihu-crawler是一个基于Java的高性能、支持免费http代理池、支持横向扩展、分布式爬虫项目

A python web scraper for public email lists.

Unmaintained 🐳 ☕️ 🕷 Scala crawler(spider) framework, inspired by scrapy, created by @gaocegege

APIs for loginning some websites by using requests.

Scrapy as a Web Crawler for Azure Search Samples

[DEPRECATED] Simple, flexible, delightful web crawler/spider package

用爬虫爬取小说网站上所有小说，存储到数据库中，并用爬到的数据构建自己的小说网站

Seeker - another job board aggregator.

Dark Web OSINT Tool

Scraping scripts for various websites.

Life is short, *****.

爬取北大法宝网http://www.pkulaw.cn/Case/

JSpider会每周更新至少一个网站的JS解密方式，欢迎 Star，交流微信：13298307816

基于Scrapy的QQ音乐爬虫(QQ Music Spider)，爬取歌曲信息、歌词、精彩评论等，并且分享了QQ音乐中排名前6400名的内地和港台歌手的49万+的音乐语料

Gospider - Fast web spider written in Go

一些非常有趣的python爬虫例子,对新手比较友好,主要爬取淘宝、天猫、微信、豆瓣、QQ等网站。(Some interesting examples of python crawlers that are friendly to beginners. )

💐 Full Amazon Automatic Download

Incredibly fast crawler designed for OSINT.

AV 电影管理系统， avmoo , javbus , javlibrary 爬虫，线上 AV 影片图书馆，AV 磁力链接数据库，Japanese Adult Video Library,Adult Video Magnet Links - Japanese Adult Video Database

BaiduSpider，一个爬取百度搜索结果的爬虫，目前支持百度网页搜索，百度图片搜索，百度知道搜索，百度视频搜索，百度资讯搜索，百度文库搜索，百度经验搜索和百度百科搜索。

自学入门 Python 优质中文资源索引，包含 书籍 / 文档 / 视频，适用于 爬虫 / Web / 数据分析 / 机器学习 方向

🏀 Python3 网络爬虫实战（部分含详细教程）猫眼 腾讯视频 豆瓣 研招网 微博 笔趣阁小说 百度热点 B站 CSDN 网易云阅读 阿里文学 百度股票 今日头条 微信公众号 网易云音乐 拉勾 有道 unsplash 实习僧 汽车之家 英雄联盟盒子 大众点评 链家 LPL赛程 台风 梦幻西游、阴阳师藏宝阁 天气 牛客网 百度文库 睡前故事 知乎 Wish

Golang爬虫 爬取汽车之家 二手车产品库

Word2vec 千人千面 个性化搜索 + Scrapy2.3.0(爬取数据) + ElasticSearch7.9.1(存储数据并提供对外Restful API) + Django3.1.1 搜索

爬取百度贴吧、TapTap、appstore、微博官方博主上的游戏评论(基于redis_scrapy)，过滤器采用了bloomfilter。

Creating Scrapy scrapers via the Django admin interface

Beanbun 是用 PHP 编写的多进程网络爬虫框架，具有良好的开放性、高可扩展性，基于 Workerman。

lots of spider (很多爬虫）

python crawler spider

AlipaySpider on Scrapy(use chrome driver); 支付宝爬虫(基于Scrapy)

Download images from Google, Bing, Baidu. 谷歌、百度、必应图片下载.

Some classic web crawler projects.一些经典的爬虫

Powerful web scraping framework for Crystal

Some scrapy and web.py exmaples

capture pictures from website like sina, lofter, huaban and so on

Geziyor, a fast web crawling & scraping framework for Go. Supports JS rendering.

Scrapy-based Crawlers for news of Taiwan

Scrapoxy hides your scraper behind a cloud. It starts a pool of proxies to send your requests. Now, you can crawl without thinking about blacklisting!

GOPA, a spider written in Go.（NOTE: this project moved to https://github.com/infinitbyte/gopa ）

Cross Platform C# Web crawler framework, headless browser, parallel crawler. Please star this project! +1.

a puppeteer walker 🕷 🕸