1060 Open source projects that are alternatives of or similar to Awesome Crawler

A versatile Ruby web spidering library that can spider a site, multiple domains, certain links or infinitely. Spidr is designed to be fast and easy to use.

[Crawler/Scraper for Golang]🕷A lightweight distributed friendly Golang crawler framework.一个轻量的分布式友好的 Golang 爬虫框架。

High performance crawler for Nodejs

新一代爬虫平台，以图形化方式定义爬虫流程，不写代码即可完成爬虫。

A simple distributed crawler for zhihu && data analysis

AV 电影管理系统， avmoo , javbus , javlibrary 爬虫，线上 AV 影片图书馆，AV 磁力链接数据库，Japanese Adult Video Library,Adult Video Magnet Links - Japanese Adult Video Database

A collection of awesome web scaper, crawler.

简单易用的Python爬虫框架，QQ交流群：597510560

A web crawler (for bug hunting) that gathers more than you can imagine.

Rust Web Crawler saving pages on Redis

Lite version of Crawlab. 轻量版 Crawlab 爬虫管理平台

Scraping scripts for various websites.

📻 An OLX Scraper using Scrapy + MongoDB. It Scrapes recent ads posted regarding requested product and dumps to NOSQL MONGODB.

Continuous scalable web crawler built on top of Flink and crawler-commons

OSINT Swiss Army Knife

Powerful web scraping framework for Crystal

A Facebook crawler

🕵️‍♂️ LinkedIn profile scraper returning structured profile data in JSON. Works in 2020.

Crawly, a high-level web crawling & scraping framework for Elixir.

[WIP] GOPA, a spider written in Golang, for Elasticsearch. DEMO: http://index.elasticsearch.cn

A high performance web crawler in Elixir.

Geziyor, a fast web crawling & scraping framework for Go. Supports JS rendering.

Distributed web crawler admin platform for spiders management regardless of languages and frameworks. 分布式爬虫管理平台，支持任何语言和框架

Fresh Onions is an open source TOR spider / hidden service onion crawler hosted at zlal32teyptf4tvi.onion

🕷️ The progressive PHP crawler framework! 优雅的渐进式PHP采集框架。

A web crawler for Go

Cross Platform C# web crawler framework built for speed and flexibility. Please star this project! +1.

Elegant Scraper and Crawler Framework for Golang

快速、简洁且强大的PHP爬虫框架

A web spider for shodan.io without using the Developer API.

Get Aliexpress product details in JSON

A web crawler that uses Firefox and js injection to interact with webpages and crawl their content, written in nodejs.

Interactive CLI Web Crawler

An open source webapp for scraping: towards a public service for webscraping

A console application that can scrape the Azur Lane wiki and export the data to Json files

Collection of scrapy spiders which can scrape posts, images, and so on from public Facebook Pages.

Papercut is a scraping/crawling library for Node.js built on top of JSDOM. It provides basic selector features together with features like Page Caching and Geosearch.

A simple and flexible web crawler framework for java.

🎯Python 3 网络爬虫实战、数据分析合集 | 当当 | 网易云音乐 | unsplash | 必胜客 | 猫眼 |

Easy way to brute-force web directory.

🍻 bilibili video (including bangumi) and danmaku downloader | B站视频（含番剧）、弹幕下载器

各种App、小程序、网站的请求签名或加密算法。 现已有：自如、小红书、蛋壳公寓、luckin coffee(瑞幸咖啡)、bangkokair(曼谷航空)

Az összes számlád még egy helyen :)

No description or website provided.

Simple Weibo Scraper

一个轻量级、快速、多线程、多管道、灵活配置的网络爬虫。

🐛一只登录正方教务管理系统，爬取数据的小爬虫

我的爬虫合集

The simple, easy to use command line web crawler.

🍭 epub generator for (light)novels (轻) 小说 epub 生成器，支持站点：轻之国度、轻小说文库

Update Version of weibo_terminator, This is Workflow Version aim at Get Job Done!

Wget-AT is a modern Wget with Lua hooks, Zstandard (+dictionary) WARC compression and URL-agnostic deduplication.

🍰 一个可视化的爬虫平台

A fast tool to fetch URLs from HTML attributes by crawl-in.

A framework for creating semi-automatic web content extractors

LinkedIn Scraper (currently working 2020)

A list of scrapers from around the web.

📰 A responsive interface of Hacker News with summaries and thumbnails.

Python的基础练习代码与各种爬虫代码

一个基于webmagic框架二次开发的java爬虫框架实战，已实现能爬取腾讯，搜狐，今日头条（单独集成功能）等资讯内容，配合elasticsearch框架用法，实现了自动爬虫，已投入线上生产使用。