1792 Open source projects that are alternatives of or similar to Nodespider

A Facebook crawler

Python爬虫实战 - 模拟登陆各大网站 包含但不限于：滑块验证、拼多多、美团、百度、bilibili、大众点评、淘宝，如果喜欢请start ❤️

稳健高效的评分制-针对性- IP代理池 + API服务，可以自己插入采集器进行代理IP的爬取，针对你的爬虫的一个或多个目标网站分别生成有效的IP代理数据库，支持MongoDB 4.0 使用 Python3.7（Scored IP proxy pool ,customise proxy data crawler can be added anytime）

The archivist's web crawler: WARC output, dashboard for all crawls, dynamic ignore patterns

知乎模拟登录，支持提取验证码和保存 Cookies

[爬虫框架 (golang)] An awesome Go concurrent Crawler(spider) framework. The crawler is flexible and modular. It can be expanded to an Individualized crawler easily or you can use the default crawl components only.

Python爬虫代理IP池(proxy pool)

🍿爬虫代理IP池(proxy pool) python🍟一个还ok的IP代理池

geetest，滑动验证码

🌭💦 91porn爬虫在线无限制API接口（永久有效，口令每日更新） 及 在线web预览

今日头条机器人，支持用户登陆、关注、取消关注、获取关注粉丝、发文、发悟空问答、点赞、评论、采集各种类型新闻讯息等，使用今日头条网页版API实现

golang实现的爬虫框架，使用者只需关心页面规则，提供web管理界面。基于colly开发。

Streamlined Future<Value, Error> implementation

Utilities that make array iteration easy when using async/await or Promises

Every web site provides APIs.

新一代爬虫平台，以图形化方式定义爬虫流程，不写代码即可完成爬虫。

a reliable high-level web crawling & scraping framework for Node.js.

🍻 bilibili video (including bangumi) and danmaku downloader | B站视频（含番剧）、弹幕下载器

Python的基础练习代码与各种爬虫代码

快速、简洁且强大的PHP爬虫框架

OSINT Swiss Army Knife

React custom hooks for async functions with abortability and composability

Transpile ES2017 async/await to vanilla ES6 Promise chains: a Babel plugin

Distributed crawler powered by Headless Chrome

A distributed web crawler framework.（分布式爬虫框架XXL-CRAWLER）

API of DouYin for Humans used to Crawl Popular Videos and Musics

爬蟲極簡教學（fetch, parse, search, multiprocessing, API）- PTT 為例

📰 A responsive interface of Hacker News with summaries and thumbnails.

微信公众号爬虫，公众号历史文章，文章评论，文章阅读及在看数据，可视化web页面，可部署于Windows服务器。基于Python3之flask/mysql/redis/mitmproxy/pywin32等实现，高效微信爬虫，微信公众号爬虫，历史文章，文章评论，数据更新。

[WIP] GOPA, a spider written in Golang, for Elasticsearch. DEMO: http://index.elasticsearch.cn

Promise packages, patterns, chat, and tutorials

🦅 Aigle is an ideal promise library which is faster and more efficient than other libraries. It is a production-ready library that implements the Promise A+ standard.

Sophisticated and functionally-minded async with advanced features: coroutines, promises, ES2015 iterables, fantasy-land

A multi-thread crawler framework with many builtin image crawlers provided.

Abstraction for managing asynchronous code in JS

小说下载|小说爬取|起点|笔趣阁|导出Markdown|导出txt|转换epub|广告过滤|自动校对

各种App、小程序、网站的请求签名或加密算法。 现已有：自如、小红书、蛋壳公寓、luckin coffee(瑞幸咖啡)、bangkokair(曼谷航空)

Fresh Onions is an open source TOR spider / hidden service onion crawler hosted at zlal32teyptf4tvi.onion

Crawly, a high-level web crawling & scraping framework for Elixir.

Html网页正文提取

A collection of awesome web crawler,spider in different languages

磁力網站U3C3介紹以及域名更新

Composable async primitives with cancelation, control over scheduling, and coroutines. Superior replacement for JS Promises.

Free Web Scraping Tool with Java

Map over promises concurrently

A versatile Ruby web spidering library that can spider a site, multiple domains, certain links or infinitely. Spidr is designed to be fast and easy to use.

NetDiscovery 是一款基于 Vert.x、RxJava 2 等框架实现的通用爬虫框架/中间件。

The Prime Cross Site Request Forgery (CSRF) Audit and Exploitation Toolkit.

INFO-SPIDER 是一个集众多数据源于一身的爬虫工具箱🧰，旨在安全快捷的帮助用户拿回自己的数据，工具代码开源，流程透明。支持数据源包括GitHub、QQ邮箱、网易邮箱、阿里邮箱、新浪邮箱、Hotmail邮箱、Outlook邮箱、京东、淘宝、支付宝、中国移动、中国联通、中国电信、知乎、哔哩哔哩、网易云音乐、QQ好友、QQ群、生成朋友圈相册、浏览器浏览历史、12306、博客园、CSDN博客、开源中国博客、简书。

一个超级轻量的百度图片爬虫

带你了解一下Golang的市场行情

A high performance web crawler in Elixir.

🐾 Creeper - The Next Generation Crawler Framework (Go)

Gospider - Fast web spider written in Go

PROJECT MOVED: https://github.com/kdex/ws-promise

✨强大、优雅的微信小程序异步库🚀

Dark Web OSINT Tool

zhihu-crawler是一个基于Java的高性能、支持免费http代理池、支持横向扩展、分布式爬虫项目

⏳ Improve your loading state control with pretty simple methods and helpers.

🐛一只登录正方教务管理系统，爬取数据的小爬虫