682 Open source projects that are alternatives of or similar to Fast Lianjia Crawler

蓝天采集器是一款免费的数据采集发布爬虫软件，采用php+mysql开发，可部署在云服务器，几乎能采集所有类型的网页，无缝对接各类CMS建站程序，免登录实时发布数据，全自动无需人工干预！是网页大数据采集软件中完全跨平台的云端爬虫系统

High performance crawler for Nodejs

Continuous scalable web crawler built on top of Flink and crawler-commons

🐛一只登录正方教务管理系统，爬取数据的小爬虫

Golang爬虫 爬取豆瓣电影Top250

疫情数据爬虫，2019新型冠状病毒数据仓库，轨迹数据，同乘数据，报道

📰 A responsive interface of Hacker News with summaries and thumbnails.

dynamic crawler for web vulnerability scanner

[WIP] GOPA, a spider written in Golang, for Elasticsearch. DEMO: http://index.elasticsearch.cn

🥄 A package for building specific Proxy Pool for different Sites.

Every web site provides APIs.

🕵️‍♂️ LinkedIn profile scraper returning structured profile data in JSON. Works in 2020.

🕷 CrawlerDetect is a PHP class for detecting bots/crawlers/spiders via the user agent

a reliable high-level web crawling & scraping framework for Node.js.

小说下载|小说爬取|起点|笔趣阁|导出Markdown|导出txt|转换epub|广告过滤|自动校对

Web crawling framework based on asyncio.

Fresh Onions is an open source TOR spider / hidden service onion crawler hosted at zlal32teyptf4tvi.onion

🍻 bilibili video (including bangumi) and danmaku downloader | B站视频（含番剧）、弹幕下载器

各种App、小程序、网站的请求签名或加密算法。 现已有：自如、小红书、蛋壳公寓、luckin coffee(瑞幸咖啡)、bangkokair(曼谷航空)

Html网页正文提取

快速、简洁且强大的PHP爬虫框架

带你了解一下Golang的市场行情

💖 High available distributed ip proxy pool, powerd by Scrapy and Redis

A Facebook crawler

A collection of awesome web crawler,spider in different languages

JS逆向研究

NetDiscovery 是一款基于 Vert.x、RxJava 2 等框架实现的通用爬虫框架/中间件。

Free Web Scraping Tool with Java

A web crawler (for bug hunting) that gathers more than you can imagine.

Cross Platform C# web crawler framework built for speed and flexibility. Please star this project! +1.

🐾 Creeper - The Next Generation Crawler Framework (Go)

Gospider - Fast web spider written in Go

The archivist's web crawler: WARC output, dashboard for all crawls, dynamic ignore patterns

Scraping scripts for various websites.

zhihu-crawler是一个基于Java的高性能、支持免费http代理池、支持横向扩展、分布式爬虫项目

Python爬虫实战 - 模拟登陆各大网站 包含但不限于：滑块验证、拼多多、美团、百度、bilibili、大众点评、淘宝，如果喜欢请start ❤️

A versatile Ruby web spidering library that can spider a site, multiple domains, certain links or infinitely. Spidr is designed to be fast and easy to use.

自学入门 Python 优质中文资源索引，包含 书籍 / 文档 / 视频，适用于 爬虫 / Web / 数据分析 / 机器学习 方向

Incredibly fast crawler designed for OSINT.

Beanbun 是用 PHP 编写的多进程网络爬虫框架，具有良好的开放性、高可扩展性，基于 Workerman。

AV 电影管理系统， avmoo , javbus , javlibrary 爬虫，线上 AV 影片图书馆，AV 磁力链接数据库，Japanese Adult Video Library,Adult Video Magnet Links - Japanese Adult Video Database

a puppeteer walker 🕷 🕸

Some classic web crawler projects.一些经典的爬虫

A Laravel wrapper for CrawlerDetect - the web crawler detection library

A multi-thread crawler framework with many builtin image crawlers provided.

一个超级轻量的百度图片爬虫

抖音 SDK，数据采集，爬虫抓取不是梦

Digger is a powerful and flexible web crawler implemented by pure golang

中国大陆大学列表爬虫

web spider built by puppeteer, support task-queue and task-scheduling by decorators，support nedb / mongodb, support data visualization; 基于puppeteer的web爬虫框架，提供灵活的任务队列管理调度方案，提供便捷的数据保存方案（nedb/mongodb），提供数据可视化和用户交互的实现方案

Official Algolia Plugin for Netlify. Index your website to Algolia when deploying your project to Netlify with the Algolia Crawler

Collection of China illegal cases about web crawler 本项目用来整理所有中国大陆爬虫开发者涉诉与违规相关的新闻、资料与法律法规。致力于帮助在中国大陆工作的爬虫行业从业者了解我国相关法律，避免触碰数据合规红线。 [AD]中文知识图谱门户

Web Scraping Framework

Proxy [Finder | Checker | Server]. HTTP(S) & SOCKS 🎭

小说精品屋是一个多平台（web、安卓app、微信小程序）、功能完善的屏幕自适应小说漫画连载系统，包含精品小说专区、轻小说专区和漫画专区。包括小说/漫画分类、小说/漫画搜索、小说/漫画排行、完本小说/漫画、小说/漫画评分、小说/漫画在线阅读、小说/漫画书架、小说/漫画阅读记录、小说下载、小说弹幕、小说/漫画自动采集/更新/纠错、小说内容自动分享到微博、邮件自动推广、链接自动推送到百度搜索引擎等功能。

A cross platform UI crawler which scans view trees then generate and execute UI test cases.

Crawl instagram photos, posts and videos for download.

Scrapes all photos and videos in a web page / Instagram / Twitter / Tumblr / Reddit / pixiv / TikTok

네이버 뉴스 수집을 위한 도구

用 node.js 爬你自己的 leetcode 解题源码