721 Open source projects that are alternatives of or similar to Hive

🎊 Design and implement of lightweight crawler framework.

A django admin site for scrapy

Collection of scrapy spiders which can scrape posts, images, and so on from public Facebook Pages.

Iota is a web scraper which can find all of the images and links/suburls on a webpage

A Facebook crawler

A web search engine built with Python which uses TF-IDF and PageRank to sort search results.

capture pictures from website like sina, lofter, huaban and so on

一个基于Scrapy的数据采集爬虫代码库

AlipaySpider on Scrapy(use chrome driver); 支付宝爬虫(基于Scrapy)

A framework for creating semi-automatic web content extractors

admin ui for scrapy/open source scrapinghub

爬取百度贴吧、TapTap、appstore、微博官方博主上的游戏评论(基于redis_scrapy)，过滤器采用了bloomfilter。

招聘网数据爬虫

NScrapy is a .net core corss platform Distributed Spider Framework which provide an easy way to write your own Spider

日常爬虫

A series of distributed components for Scrapy. Including RabbitMQ-based components, Kafka-based components, and RedisBloom-based components for Scrapy.

python爬虫小项目【持续更新】【笔趣阁小说下载、Tweet数据抓取、天气查询、网易云音乐逆向、天天基金网查询、微博数据抓取（生成cookie）、有道翻译逆向、企查查免登陆爬虫、大众点评svg加密破解、B站用户爬虫、拉钩免登录爬虫、自如租房字体加密、知乎问答

A tutorial-based introduction to web scraping with Python.

免费 IP 代理池。Scrapy 爬虫框架插件

Uno Crawler por https://douban.com

百度贴吧爬虫(基于scrapy和mysql)

🎊 Design and implement of lightweight crawler framework.

A set of spiders and scrapers to extract location information from places that post their location on the internet.

Seeker - another job board aggregator.

A python web scraper for public email lists.

Distributed Crawler Management Framework Based on Scrapy, Scrapyd, Django and Vue.js

Creating Scrapy scrapers via the Django admin interface

基于python语言开发的千万级别搜索引擎

Dynamic configurable crawl (动态可配置化爬虫)

JavaScript + BeautifulSoup = JSSoup

百度贴吧生日爬虫，可抓取贴吧内吧友生日，并且在对应日期自动发送祝福

PTT 表特版爬蟲圖片下載器

163music spider by scrapy.

[Crawler/Scraper for Golang]🕷A lightweight distributed friendly Golang crawler framework.一个轻量的分布式友好的 Golang 爬虫框架。

An open source webapp for scraping: towards a public service for webscraping

千万级图片爬虫、视频爬虫 [开源版本] Image Spider

收集各种免费的 Python 爬虫项目

10 photo website spiders, 10 个国外图库的 scrapy 爬虫代码

V2EX爬虫

一些爬虫开发的技巧和案例

Download images from Google, Bing, Baidu. 谷歌、百度、必应图片下载.

Web app for Scrapyd cluster management, Scrapy log analysis & visualization, Auto packaging, Timer tasks, Monitor & Alert, and Mobile UI. DEMO 👉

Generating spiders dynamically to crawl and check those free proxy ip on the internet with scrapy.

PTT 圖片下載器 (抓取整個看板的圖片，並用文章標題作為資料夾的名稱 ) (使用Scrapy)

🔧 🔩 🔨 收集整理了爬虫相关的工具、模拟登陆技术、代理IP、scrapy模板代码等内容。

今日头条科技新闻接口爬虫

Linkedin Scraper using Selenium Web Driver, Chromium headless, Docker and Scrapy

Simple yet powerful automation stuffs.

💖 High available distributed ip proxy pool, powerd by Scrapy and Redis

基于Scrapy框架的豆瓣电影爬虫

Word2vec 千人千面 个性化搜索 + Scrapy2.3.0(爬取数据) + ElasticSearch7.9.1(存储数据并提供对外Restful API) + Django3.1.1 搜索

A multi-thread crawler framework with many builtin image crawlers provided.

JSpider会每周更新至少一个网站的JS解密方式，欢迎 Star，交流微信：13298307816

豆瓣电影top250、斗鱼爬取json数据以及爬取美女图片、淘宝、有缘、CrawlSpider爬取红娘网相亲人的部分基本信息以及红娘网分布式爬取和存储redis、爬虫小demo、Selenium、爬取多点、django开发接口、爬取有缘网信息、模拟知乎登录、模拟github登录、模拟图虫网登录、爬取多点商城整站数据、爬取微信公众号历史文章、爬取微信群或者微信好友分享的文章、itchat监听指定微信公众号分享的文章

ScrapingOutsourcing专注分享爬虫代码 尽量每周更新一个

💐Marmot | Web Crawler/HTTP protocol Download Package 🐭

🕷️some python spiders with BeautifulSoup or scarpy

Simple but useful Python web scraping tutorial code.

Distributed web crawler admin platform for spiders management regardless of languages and frameworks. 分布式爬虫管理平台，支持任何语言和框架

🏀 Python3 网络爬虫实战（部分含详细教程）猫眼 腾讯视频 豆瓣 研招网 微博 笔趣阁小说 百度热点 B站 CSDN 网易云阅读 阿里文学 百度股票 今日头条 微信公众号 网易云音乐 拉勾 有道 unsplash 实习僧 汽车之家 英雄联盟盒子 大众点评 链家 LPL赛程 台风 梦幻西游、阴阳师藏宝阁 天气 牛客网 百度文库 睡前故事 知乎 Wish