1170 Open source projects that are alternatives of or similar to Geziyor

Home Assistant custom component for scraping (html, xml or json) multiple values (from a single HTTP request) with a separate sensor/attribute for each value. Support for (login) form-submit functionality.

ProxyCrawl Python library for scraping and crawling

robots.txt file parsing and checking for R

Command line program to download documents from web portals.

A series of distributed components for Scrapy. Including RabbitMQ-based components, Kafka-based components, and RedisBloom-based components for Scrapy.

Retrieve real (with Javascript executed) HTML code from an URL, ultra fast and supports multiple parallel loading of webs

Distributed web crawler admin platform for spiders management regardless of languages and frameworks. 分布式爬虫管理平台，支持任何语言和框架

Continuous scalable web crawler built on top of Flink and crawler-commons

🕸 List of mini projects that involve web scraping 🕸

A Scraper, Downloader, & Recorder for static open directories.

A Scraper made 100% in Python using BeautifulSoup and Tor. It can be used to scrape both normal and onion links. Happy Scraping :)

🚀「Douyin_TikTok_Download_API」是一个开箱即用的高性能异步抖音|TikTok数据爬取工具，支持API调用，在线批量解析及下载。

🍻 bilibili video (including bangumi) and danmaku downloader | B站视频（含番剧）、弹幕下载器

💐 Full Amazon Automatic Download

Simple Weibo Scraper

A collection of tools to (semi-)automatically collect and analyze data from online discussions on Facebook groups and pages.

🍭 epub generator for (light)novels (轻) 小说 epub 生成器，支持站点：轻之国度、轻小说文库

🐛一只登录正方教务管理系统，爬取数据的小爬虫

An R web crawler and scraper

磁力網站U3C3介紹以及域名更新

Crawl BookCorpus

LinkedIn Scraper (currently working 2020)

Python的基础练习代码与各种爬虫代码

Rust Web Crawler saving pages on Redis

一个灵活、友好的爬虫框架

Every web site provides APIs.

golang实现的爬虫框架，使用者只需关心页面规则，提供web管理界面。基于colly开发。

🌭💦 91porn爬虫在线无限制API接口（永久有效，口令每日更新） 及 在线web预览

图片爬取下载工具，极速爬取下载 站酷https://www.zcool.com.cn/, CNU 视觉 http://www.cnu.cc/ 设计师/用户 上传的 图片/照片/插画。

A framework for creating semi-automatic web content extractors

各种App、小程序、网站的请求签名或加密算法。 现已有：自如、小红书、蛋壳公寓、luckin coffee(瑞幸咖啡)、bangkokair(曼谷航空)

Some classic web crawler projects.一些经典的爬虫

新一代爬虫平台，以图形化方式定义爬虫流程，不写代码即可完成爬虫。

💖 High available distributed ip proxy pool, powerd by Scrapy and Redis

The Prime Cross Site Request Forgery (CSRF) Audit and Exploitation Toolkit.

Html网页正文提取

a reliable high-level web crawling & scraping framework for Node.js.

NetDiscovery 是一款基于 Vert.x、RxJava 2 等框架实现的通用爬虫框架/中间件。

Extract structured data from web sites. Web sites scraping.

Simple but useful Python web scraping tutorial code.

API of DouYin for Humans used to Crawl Popular Videos and Musics

Golang pkg to quickly return a preview of a webpage (title/description/images)

带你了解一下Golang的市场行情

Crawler for Nintendo Switch eShop

A distributed web crawler framework.（分布式爬虫框架XXL-CRAWLER）

小说下载|小说爬取|起点|笔趣阁|导出Markdown|导出txt|转换epub|广告过滤|自动校对

A multi-thread crawler framework with many builtin image crawlers provided.

HTTP API for Scrapy spiders

🐾 Creeper - The Next Generation Crawler Framework (Go)

✂️ High performance, multi-threaded image scraper

The archivist's web crawler: WARC output, dashboard for all crawls, dynamic ignore patterns

Python爬虫，京东自动登录，在线抢购商品

Gospider - Fast web spider written in Go

[DEPRECATED] Simple, flexible, delightful web crawler/spider package

一个超级轻量的百度图片爬虫

zhihu-crawler是一个基于Java的高性能、支持免费http代理池、支持横向扩展、分布式爬虫项目

Dark Web OSINT Tool

Beanbun 是用 PHP 编写的多进程网络爬虫框架，具有良好的开放性、高可扩展性，基于 Workerman。

Fetches PubMed article IDs (PMIDs) from email inbox, then crawls PubMed, Google Scholar and Sci-Hub for respective PDF files.

Search for and retrieve US Patent and Trademark Office Patent Data