1158 Open source projects that are alternatives of or similar to Baiduspider

APIs for loginning some websites by using requests.

爬蟲極簡教學（fetch, parse, search, multiprocessing, API）- PTT 為例

一个超级轻量的百度图片爬虫

🕷 CrawlerDetect is a PHP class for detecting bots/crawlers/spiders via the user agent

OSINT Swiss Army Knife

Crawly, a high-level web crawling & scraping framework for Elixir.

A distributed web crawler framework.（分布式爬虫框架XXL-CRAWLER）

GOPA, a spider written in Go.（NOTE: this project moved to https://github.com/infinitbyte/gopa ）

NetDiscovery 是一款基于 Vert.x、RxJava 2 等框架实现的通用爬虫框架/中间件。

Free Web Scraping Tool with Java

Dark Web OSINT Tool

新一代爬虫平台，以图形化方式定义爬虫流程，不写代码即可完成爬虫。

各种App、小程序、网站的请求签名或加密算法。 现已有：自如、小红书、蛋壳公寓、luckin coffee(瑞幸咖啡)、bangkokair(曼谷航空)

A web crawler (for bug hunting) that gathers more than you can imagine.

Open-source Enterprise Grade Search Engine Software

A Facebook crawler

爬取百度百科中文页面，抽取三元组信息，构建中文知识图谱

蓝天采集器是一款免费的数据采集发布爬虫软件，采用php+mysql开发，可部署在云服务器，几乎能采集所有类型的网页，无缝对接各类CMS建站程序，免登录实时发布数据，全自动无需人工干预！是网页大数据采集软件中完全跨平台的云端爬虫系统

A collection of awesome web crawler,spider in different languages

🐾 Creeper - The Next Generation Crawler Framework (Go)

A high performance web crawler in Elixir.

zhihu-crawler是一个基于Java的高性能、支持免费http代理池、支持横向扩展、分布式爬虫项目

[DEPRECATED] Simple, flexible, delightful web crawler/spider package

Rust Web Crawler saving pages on Redis

Distributed web crawler admin platform for spiders management regardless of languages and frameworks. 分布式爬虫管理平台，支持任何语言和框架

Beanbun 是用 PHP 编写的多进程网络爬虫框架，具有良好的开放性、高可扩展性，基于 Workerman。

A search engine that doesn't track you.

a reliable high-level web crawling & scraping framework for Node.js.

🍻 bilibili video (including bangumi) and danmaku downloader | B站视频（含番剧）、弹幕下载器

小说下载|小说爬取|起点|笔趣阁|导出Markdown|导出txt|转换epub|广告过滤|自动校对

Async Python 3.6+ web scraping micro-framework based on asyncio

Html网页正文提取

Python的基础练习代码与各种爬虫代码

Fresh Onions is an open source TOR spider / hidden service onion crawler hosted at zlal32teyptf4tvi.onion

The Prime Cross Site Request Forgery (CSRF) Audit and Exploitation Toolkit.

带你了解一下Golang的市场行情

抖音 SDK，数据采集，爬虫抓取不是梦

💖 High available distributed ip proxy pool, powerd by Scrapy and Redis

A search application to explore, discover and share online files

Fess is very powerful and easily deployable Enterprise Search Server.

API of DouYin for Humans used to Crawl Popular Videos and Musics

快速、简洁且强大的PHP爬虫框架

The archivist's web crawler: WARC output, dashboard for all crawls, dynamic ignore patterns

A versatile Ruby web spidering library that can spider a site, multiple domains, certain links or infinitely. Spidr is designed to be fast and easy to use.

Gospider - Fast web spider written in Go

A multi-thread crawler framework with many builtin image crawlers provided.

Scrapy as a Web Crawler for Azure Search Samples

Scraping scripts for various websites.

💐 Full Amazon Automatic Download

简单搜索，一个简单的前端界面。用惯了各种导航首页，满屏幕尽是各种不厌其烦的广告和资讯；尝试自己写个自己的主页。

Golang爬虫 爬取汽车之家 二手车产品库

自学入门 Python 优质中文资源索引，包含 书籍 / 文档 / 视频，适用于 爬虫 / Web / 数据分析 / 机器学习 方向

Incredibly fast crawler designed for OSINT.

python crawler spider

Powerful web scraping framework for Crystal

Download images from Google, Bing, Baidu. 谷歌、百度、必应图片下载.

🌭💦 91porn爬虫在线无限制API接口（永久有效，口令每日更新） 及 在线web预览

今日头条机器人，支持用户登陆、关注、取消关注、获取关注粉丝、发文、发悟空问答、点赞、评论、采集各种类型新闻讯息等，使用今日头条网页版API实现

a puppeteer walker 🕷 🕸

AV 电影管理系统， avmoo , javbus , javlibrary 爬虫，线上 AV 影片图书馆，AV 磁力链接数据库，Japanese Adult Video Library,Adult Video Magnet Links - Japanese Adult Video Database