398 Open source projects that are alternatives of or similar to Pyspider

Async Python 3.6+ web scraping micro-framework based on asyncio

👩 美女写真套图爬虫（一）

稳健高效的评分制-针对性- IP代理池 + API服务，可以自己插入采集器进行代理IP的爬取，针对你的爬虫的一个或多个目标网站分别生成有效的IP代理数据库，支持MongoDB 4.0 使用 Python3.7（Scored IP proxy pool ,customise proxy data crawler can be added anytime）

各种App、小程序、网站的请求签名或加密算法。 现已有：自如、小红书、蛋壳公寓、luckin coffee(瑞幸咖啡)、bangkokair(曼谷航空)

🎵 将可下载的网易云音乐的缓存文件转换为 MP3 文件

Dynamic meta tags in your AngularJS single page application

A search engine that doesn't track you.

GOPA, a spider written in Go.（NOTE: this project moved to https://github.com/infinitbyte/gopa ）

小说下载|小说爬取|起点|笔趣阁|导出Markdown|导出txt|转换epub|广告过滤|自动校对

直接通过链家 API 抓取数据的极速爬虫，宇宙最快~~ 🚀

A non API python program to crawl public photos, posts or followers

Amazon商品引流的 python 爬虫

Crawler (Bot) searching for credential leaks on different paste sites.

📢 Ptt 文章通知機器人！Notify Ptt Article in Realtime

Download videos from pornhub.

a scaleable and efficient crawelr with docker cluster , crawl million pages in 2 hours with a single machine

今日头条机器人，支持用户登陆、关注、取消关注、获取关注粉丝、发文、发悟空问答、点赞、评论、采集各种类型新闻讯息等，使用今日头条网页版API实现

Get (almost) original messages from google group archives. Your data is yours.

知乎模拟登录，支持提取验证码和保存 Cookies

Secret and/ credential patterns used for gf.

🌭💦 91porn爬虫在线API接口（永久有效） 及 在线web预览

CoCrawler is a versatile web crawler built using modern tools and concurrency.

The DomCrawler component eases DOM navigation for HTML and XML documents.

利用文本挖掘技术进行新闻热点关注问题分析

Intelligent proxy pool for Humans™ (Maintainer needed)

京东口罩库存监控爬虫(非selenium)，扫码登录、查价、加购、下单、秒杀

Every web site provides APIs.

爬虫代理IP池服务，可供其他爬虫程序通过restapi获取

An extremely fast web scraper that parses megabytes of invalid HTML in a blink of an eye. PHP5.3+, no dependencies.

一些爬虫的代码

Simple yet powerful automation stuffs.

Geziyor, a fast web crawling & scraping framework for Go. Supports JS rendering.

一个灵活、友好的爬虫框架

Easy to use lightweight web crawler（易用的轻量化网络爬虫）

爬蟲極簡教學（fetch, parse, search, multiprocessing, API）- PTT 為例

Scrapy-based Crawlers for news of Taiwan

DotnetSpider, a .NET standard web crawling library. It is lightweight, efficient and fast high-level web crawling & scraping framework

Touhou Project random music video generator/player, crawling image and video from websites to generate MV.

[WIP] GOPA, a spider written in Golang, for Elasticsearch. DEMO: http://index.elasticsearch.cn

Verify that a request is from Google crawlers using Google's DNS verification steps

An R web crawler and scraper

本仓库收集整理爬虫相关资源，开发语言以Java为主

line-bot-tutorial use python flask

Lightweight Ruby web crawler/scraper with an elegant DSL which extracts structured data from pages.

Update Version of weibo_terminator, This is Workflow Version aim at Get Job Done!

Go process used to crawl websites

The simple, easy to use command line web crawler.

a puppeteer walker 🕷 🕸

A fast tool to fetch URLs from HTML attributes by crawl-in.

💐Marmot | Web Crawler/HTTP protocol Download Package 🐭

Python: An all-in-one Web Crawler, Web Parser and Web Scrapping library!

Be nice on the web

免登录下载微博图片 爬虫 Download Weibo Images without Logging-in

A Kotlin-based testing/scraping/parsing library providing the ability to analyze and extract data from HTML (server & client-side rendered). It places particular emphasis on ease of use and a high level of readability by providing an intuitive DSL. It aims to be a testing lib, but can also be used to scrape websites in a convenient fashion.

Proxy [Finder | Checker | Server]. HTTP(S) & SOCKS 🎭

A Python module to scrape several search engines (like Google, Yandex, Bing, Duckduckgo, ...). Including asynchronous networking support.

Crawl some picture for fun

一些非常有趣的python爬虫例子,对新手比较友好,主要爬取淘宝、天猫、微信、豆瓣、QQ等网站。(Some interesting examples of python crawlers that are friendly to beginners. )

Crawls a website, gets PageSpeed Insights data for each page, and exports an HTML report.