594 Open source projects that are alternatives of or similar to Github Spider

A framework for creating semi-automatic web content extractors

Dynamic file detection tool based on crawler 基于爬虫的动态敏感文件探测工具

crawler framework, distributed crawler extractor

Crawlera middleware for Scrapy

💐Marmot | Web Crawler/HTTP protocol Download Package 🐭

Scrapy-based Crawlers for news of Taiwan

PTT 網路版爬蟲

Scrapoxy hides your scraper behind a cloud. It starts a pool of proxies to send your requests. Now, you can crawl without thinking about blacklisting!

基于搜狗微信搜索的微信公众号爬虫接口

Scrapy as a Web Crawler for Azure Search Samples

Distributed web crawler admin platform for spiders management regardless of languages and frameworks. 分布式爬虫管理平台，支持任何语言和框架

《数据采集从入门到放弃》源码。内容简介：爬虫介绍、就业情况、爬虫工程师面试题 ；HTTP协议介绍； Requests使用 ；解析器Xpath介绍； MongoDB与MySQL； 多线程爬虫； Scrapy介绍 ；Scrapy-redis介绍； 使用docker部署； 使用nomad管理docker集群； 使用EFK查询docker日志

A Facebook crawler

scrapy专利爬虫（停止维护）

Python爬虫实战 - 模拟登陆各大网站 包含但不限于：滑块验证、拼多多、美团、百度、bilibili、大众点评、淘宝，如果喜欢请start ❤️

HTTP API for Scrapy spiders

A multi-thread crawler framework with many builtin image crawlers provided.

Some scrapy and web.py exmaples

swiss army knife for hackers

💖 High available distributed ip proxy pool, powerd by Scrapy and Redis

Redis-based components for Scrapy.

Life is short, *****.

Simple but useful Python web scraping tutorial code.

DotnetCrawler is a straightforward, lightweight web crawling/scrapying library for Entity Framework Core output based on dotnet core. This library designed like other strong crawler libraries like WebMagic and Scrapy but for enabling extandable your custom requirements. Medium link : https://medium.com/@mehmetozkaya/creating-custom-web-crawler-with-dotnet-core-using-entity-framework-core-ec8d23f0ca7c

爬虫, http代理, 模拟登陆!

ScrapingOutsourcing专注分享爬虫代码 尽量每周更新一个

Lite version of Crawlab. 轻量版 Crawlab 爬虫管理平台

码云仓库链接:AJay13/ECommerceCrawlers Github 仓库链接:DropsDevopsOrg/ECommerceCrawlers 项目展示平台链接:http://wechat.doonsec.com

Parser and database to index the terpene profile of different strains of Cannabis from online databases

基于Scrapy的QQ音乐爬虫(QQ Music Spider)，爬取歌曲信息、歌词、精彩评论等，并且分享了QQ音乐中排名前6400名的内地和港台歌手的49万+的音乐语料

[Crawler/Scraper for Golang]🕷A lightweight distributed friendly Golang crawler framework.一个轻量的分布式友好的 Golang 爬虫框架。

네이버 뉴스 수집을 위한 도구

Polite, slim and concurrent web crawler.

Alternative for VK Audio API

基于 Selenium 的知乎关键词爬虫

🌈Wenshu_Spider-Scrapy框架爬取中国裁判文书网案件数据(2019-1-9最新版)

JS逆向研究

一款分布式爬虫平台，帮助你更好的管理和开发爬虫。 内置一套爬虫定义规则（模版），可使用模版快速定义爬虫，也可当作框架手动开发爬虫。(兴趣使然的项目，用的不爽了就更新)

用 node.js 爬你自己的 leetcode 解题源码

Scrapy Training companion code

Netflix like full-stack application with SPA client and backend implemented in service oriented architecture

直播网站数据采集

🐝 Web vertical crawler framework for fun

疫情数据爬虫，2019新型冠状病毒数据仓库，轨迹数据，同乘数据，报道

DownZemAll! is a download manager for Windows, MacOS and Linux

Cross Platform C# web crawler framework built for speed and flexibility. Please star this project! +1.

scrapes medias, likes, followers, tags and all metadata. Inspired by instagram-php-scraper,bot

An easy to use, powerful crawler implemented in PHP. Can execute Javascript.

This is a sina weibo spider built by scrapy [微博爬虫/持续维护]

🥄 A package for building specific Proxy Pool for different Sites.

根据关键词抓取微博数据，再生成词云

Free proxy server, continuously crawling and providing proxies, based on Tornado and Scrapy. 免费代理服务器，基于Tornado和Scrapy，在本地搭建属于自己的代理池

🕵️‍♂️ LinkedIn profile scraper returning structured profile data in JSON. Works in 2020.

基于Spark的电影推荐系统，包含爬虫项目、web网站、后台管理系统以及spark推荐系统

Easy to use lightweight web crawler（易用的轻量化网络爬虫）

Web app for Scrapyd cluster management, Scrapy log analysis & visualization, Auto packaging, Timer tasks, Monitor & Alert, and Mobile UI. DEMO 👉

链家网和贝壳网房价爬虫，采集北京上海广州深圳等21个中国主要城市的房价数据（小区，二手房，出租房，新房），稳定可靠快速！支持csv,MySQL, MongoDB,Excel, json存储，支持Python2和3，图表展示数据，注释丰富 ，点星支持，仅供学习参考，请勿用于商业用途，后果自负。

Crawler for LinkedIn full profiles 2019

Dynamic meta tags in your AngularJS single page application