1160 Open source projects that are alternatives of or similar to Spidr

A collection of awesome web crawler,spider in different languages

Powerful web scraping framework for Crystal

[WIP] GOPA, a spider written in Golang, for Elasticsearch. DEMO: http://index.elasticsearch.cn

📻 An OLX Scraper using Scrapy + MongoDB. It Scrapes recent ads posted regarding requested product and dumps to NOSQL MONGODB.

Lite version of Crawlab. 轻量版 Crawlab 爬虫管理平台

`scrape_linkedin` is a python package that allows you to scrape personal LinkedIn profiles & company pages - turning the data into structured json.

Fresh Onions is an open source TOR spider / hidden service onion crawler hosted at zlal32teyptf4tvi.onion

A web crawler for Go

PHP Scraper - an highly opinionated web-interface for PHP

Scraping scripts for various websites.

Rust Web Crawler saving pages on Redis

A Facebook crawler

A collection of awesome web scaper, crawler.

Geziyor, a fast web crawling & scraping framework for Go. Supports JS rendering.

A simple distributed crawler for zhihu && data analysis

🚀「Douyin_TikTok_Download_API」是一个开箱即用的高性能异步抖音|TikTok数据爬取工具，支持API调用，在线批量解析及下载。

A framework for creating semi-automatic web content extractors

A high performance web crawler in Elixir.

Cross Platform C# web crawler framework built for speed and flexibility. Please star this project! +1.

A web crawler (for bug hunting) that gathers more than you can imagine.

简单易用的Python爬虫框架，QQ交流群：597510560

🕵️‍♂️ LinkedIn profile scraper returning structured profile data in JSON. Works in 2020.

快速、简洁且强大的PHP爬虫框架

[Crawler/Scraper for Golang]🕷A lightweight distributed friendly Golang crawler framework.一个轻量的分布式友好的 Golang 爬虫框架。

🕷️ The progressive PHP crawler framework! 优雅的渐进式PHP采集框架。

新一代爬虫平台，以图形化方式定义爬虫流程，不写代码即可完成爬虫。

AV 电影管理系统， avmoo , javbus , javlibrary 爬虫，线上 AV 影片图书馆，AV 磁力链接数据库，Japanese Adult Video Library,Adult Video Magnet Links - Japanese Adult Video Database

OSINT Swiss Army Knife

Crawly, a high-level web crawling & scraping framework for Elixir.

Web scraper for grabing data from Linkedin profiles or company pages (personal project)

A Smart, Automatic, Fast and Lightweight Web Scraper for Python

Continuous scalable web crawler built on top of Flink and crawler-commons

Distributed web crawler admin platform for spiders management regardless of languages and frameworks. 分布式爬虫管理平台，支持任何语言和框架

Elegant Scraper and Crawler Framework for Golang

Papercut is a scraping/crawling library for Node.js built on top of JSDOM. It provides basic selector features together with features like Page Caching and Geosearch.

High performance crawler for Nodejs

Simple Weibo Scraper

我的爬虫合集

一个超级轻量的百度图片爬虫

🐛一只登录正方教务管理系统，爬取数据的小爬虫

A multi-thread crawler framework with many builtin image crawlers provided.

The Prime Cross Site Request Forgery (CSRF) Audit and Exploitation Toolkit.

The simple, easy to use command line web crawler.

🍭 epub generator for (light)novels (轻) 小说 epub 生成器，支持站点：轻之国度、轻小说文库

Free Web Scraping Tool with Java

Faster requests on Python 3

No description or website provided.

A fast tool to fetch URLs from HTML attributes by crawl-in.

Update Version of weibo_terminator, This is Workflow Version aim at Get Job Done!

带你了解一下Golang的市场行情

💖 High available distributed ip proxy pool, powerd by Scrapy and Redis

An R web crawler and scraper

API of DouYin for Humans used to Crawl Popular Videos and Musics

磁力網站U3C3介紹以及域名更新

一个基于webmagic框架二次开发的java爬虫框架实战，已实现能爬取腾讯，搜狐，今日头条（单独集成功能）等资讯内容，配合elasticsearch框架用法，实现了自动爬虫，已投入线上生产使用。

NetDiscovery 是一款基于 Vert.x、RxJava 2 等框架实现的通用爬虫框架/中间件。

爬蟲極簡教學（fetch, parse, search, multiprocessing, API）- PTT 為例

NBA Stats API via Basketball Reference

golang实现的爬虫框架，使用者只需关心页面规则，提供web管理界面。基于colly开发。

Az összes számlád még egy helyen :)