291 Open source projects that are alternatives of or similar to Pixiv Crawler

A Facebook crawler

A Pixiv client for UWP users

A tool for pixiv.net. 人人可用的P站爬虫

Simple but useful Python web scraping tutorial code.

Build and deploy a machine learning app from scratch 🚀

Seeker - another job board aggregator.

A web extension for Pixiv

Scrapy as a Web Crawler for Azure Search Samples

Docs and files for ScrapydWeb, Scrapyd, Scrapy, and other projects

Python + Scrapy + MongoDB . 5 million data per day !!!💥 The world's largest website.

🖼 Download illusts from pixiv.net P站插画批量下载器

Pixiv Batch Downloader，功能丰富的 Pixiv 图片批量下载器

Pixiv第三方Android客户端

python爬虫

Analysis of the characteristics of different countries

💖 High available distributed ip proxy pool, powerd by Scrapy and Redis

Life is short, *****.

An unofficial Pixiv app client for Android and iOS, built with React Native.

Muzei Live Wallpaper's Source for Pixiv

豆瓣电影/豆瓣读书 Scarpy 爬虫

两只蠢萌京东的分布式爬虫.

🚀电商网站爬虫合集，淘宝京东亚马逊等

A scrapy spider that scraps times and prices from Voyages Sncf. It uses scrapyrt to provide an API interface.

Pixiv 辅助工具

HTTP API for Scrapy spiders

Python入门网络爬虫之精华版

🎊 Design and implement of lightweight crawler framework.

A Scrapy Spider for downloading PDF files from a webpage.

P站画师个人作品批量下载工具，UserScript + Aria2。可高度自定义重命名，发送到本地或远程(如路由器)下载。

Simple Web UI for Scrapy spider management via Scrapyd

基于搜狗微信搜索的微信公众号爬虫接口

[OUTDATED] scrapy spiders to crawl the financial text data 📚 📜 pertinent to train word vectors 🚀

Scrapy middleware to handle javascript pages using selenium

慕课网python分布式爬虫源码-长期更新维护

Redis-based components for Scrapy.

🤖 基于 saucenao / ascii2d / whatanime 的搜图机器人

Random User-Agent middleware based on fake-useragent

JSpider会每周更新至少一个网站的JS解密方式，欢迎 Star，交流微信：13298307816

use multiple proxies with Scrapy

Word2vec 千人千面 个性化搜索 + Scrapy2.3.0(爬取数据) + ElasticSearch7.9.1(存储数据并提供对外Restful API) + Django3.1.1 搜索

A framework for creating semi-automatic web content extractors

Creating Scrapy scrapers via the Django admin interface

💕 pixiv client for Desktop like TweetDeck 💕

Possibly the best practice of Scrapy 🕷 and renting a house 🏡

基于 scrapy-redis 的通用分布式爬虫框架

Get anime style wallpapers from daily Pixiv high ranking illustrations

The Zipru scraper developed in the Advanced Web Scraping Tutorial.

TweetScraper is a simple crawler/spider for Twitter Search without using API

一个支持免代理直连+多种额外特性功能优化的第三方 Pixiv android 客户端 | A third-party Pixiv Android client with modern design and many other enhancements

A python3 crawler for crawling Pixiv ranking top and any illustrator all artworks

A curated list of awesome packages, articles, and other cool resources from the Scrapy community.

Faster requests on Python 3

swiss army knife for hackers

This Scrapy project uses Redis and Kafka to create a distributed on demand scraping cluster.

A multi-thread crawler framework with many builtin image crawlers provided.

A Strong, Fast and Flexible Pixiv Client based on .NET Core and WPF

Distributed web crawler admin platform for spiders management regardless of languages and frameworks. 分布式爬虫管理平台，支持任何语言和框架

爬取百度贴吧、TapTap、appstore、微博官方博主上的游戏评论(基于redis_scrapy)，过滤器采用了bloomfilter。

A python web scraper for public email lists.

豆瓣电影top250、斗鱼爬取json数据以及爬取美女图片、淘宝、有缘、CrawlSpider爬取红娘网相亲人的部分基本信息以及红娘网分布式爬取和存储redis、爬虫小demo、Selenium、爬取多点、django开发接口、爬取有缘网信息、模拟知乎登录、模拟github登录、模拟图虫网登录、爬取多点商城整站数据、爬取微信公众号历史文章、爬取微信群或者微信好友分享的文章、itchat监听指定微信公众号分享的文章