881 Open source projects that are alternatives of or similar to Newcrawler

[WIP] GOPA, a spider written in Golang, for Elasticsearch. DEMO: http://index.elasticsearch.cn

🕵️‍♂️ LinkedIn profile scraper returning structured profile data in JSON. Works in 2020.

Crawly, a high-level web crawling & scraping framework for Elixir.

自学入门 Python 优质中文资源索引，包含 书籍 / 文档 / 视频，适用于 爬虫 / Web / 数据分析 / 机器学习 方向

Geziyor, a fast web crawling & scraping framework for Go. Supports JS rendering.

Elegant Scraper and Crawler Framework for Golang

带你了解一下Golang的市场行情

Wget-AT is a modern Wget with Lua hooks, Zstandard (+dictionary) WARC compression and URL-agnostic deduplication.

Python的基础练习代码与各种爬虫代码

Distributed crawler powered by Headless Chrome

High performance crawler for Nodejs

🐛一只登录正方教务管理系统，爬取数据的小爬虫

API of DouYin for Humans used to Crawl Popular Videos and Musics

Crawl all unique internal links found on a given website, and extract SEO related information - supports javascript based sites

A Laravel wrapper for CrawlerDetect - the web crawler detection library

爬虫工程师常用的 Chrome 插件 | Chrome extensions used by crawler developer

web spider built by puppeteer, support task-queue and task-scheduling by decorators，support nedb / mongodb, support data visualization; 基于puppeteer的web爬虫框架，提供灵活的任务队列管理调度方案，提供便捷的数据保存方案（nedb/mongodb），提供数据可视化和用户交互的实现方案

A simple and flexible web crawler framework for java.

Continuous scalable web crawler built on top of Flink and crawler-commons

No description or website provided.

A series of distributed components for Scrapy. Including RabbitMQ-based components, Kafka-based components, and RedisBloom-based components for Scrapy.

golang实现的爬虫框架，使用者只需关心页面规则，提供web管理界面。基于colly开发。

一个灵活、友好的爬虫框架

Every web site provides APIs.

A collection of awesome web crawler,spider in different languages

A Smart, Automatic, Fast and Lightweight Web Scraper for Python

今日头条机器人，支持用户登陆、关注、取消关注、获取关注粉丝、发文、发悟空问答、点赞、评论、采集各种类型新闻讯息等，使用今日头条网页版API实现

OSINT Swiss Army Knife

A framework for creating semi-automatic web content extractors

dynamic crawler for web vulnerability scanner

A distributed web crawler framework.（分布式爬虫框架XXL-CRAWLER）

京东口罩库存监控爬虫(非selenium)，扫码登录、查价、加购、下单、秒杀

Google search results crawler, get google search results that you need

直接通过链家 API 抓取数据的极速爬虫，宇宙最快~~ 🚀

💖 High available distributed ip proxy pool, powerd by Scrapy and Redis

Web crawler.

Papercut is a scraping/crawling library for Node.js built on top of JSDOM. It provides basic selector features together with features like Page Caching and Geosearch.

Html网页正文提取

Collection of scrapy spiders which can scrape posts, images, and so on from public Facebook Pages.

一个轻量级、快速、多线程、多管道、灵活配置的网络爬虫。

🍰 一个可视化的爬虫平台

A fast tool to fetch URLs from HTML attributes by crawl-in.

Universal scrapping tool, which allows you to extract data using multiple environments

爬蟲極簡教學（fetch, parse, search, multiprocessing, API）- PTT 為例

Crawlera middleware for Scrapy

微信公众号爬虫，公众号历史文章，文章评论，文章阅读及在看数据，可视化web页面，可部署于Windows服务器。基于Python3之flask/mysql/redis/mitmproxy/pywin32等实现，高效微信爬虫，微信公众号爬虫，历史文章，文章评论，数据更新。

📰 A responsive interface of Hacker News with summaries and thumbnails.

🌭💦 91porn爬虫在线无限制API接口（永久有效，口令每日更新） 及 在线web预览

知乎模拟登录，支持提取验证码和保存 Cookies

A Facebook crawler

小说下载|小说爬取|起点|笔趣阁|导出Markdown|导出txt|转换epub|广告过滤|自动校对

a reliable high-level web crawling & scraping framework for Node.js.

新一代爬虫平台，以图形化方式定义爬虫流程，不写代码即可完成爬虫。

🍻 bilibili video (including bangumi) and danmaku downloader | B站视频（含番剧）、弹幕下载器

Fresh Onions is an open source TOR spider / hidden service onion crawler hosted at zlal32teyptf4tvi.onion

各种App、小程序、网站的请求签名或加密算法。 现已有：自如、小红书、蛋壳公寓、luckin coffee(瑞幸咖啡)、bangkokair(曼谷航空)

NetDiscovery 是一款基于 Vert.x、RxJava 2 等框架实现的通用爬虫框架/中间件。

JavaScript + BeautifulSoup = JSSoup

Declarative web scraping

磁力網站U3C3介紹以及域名更新