1003 Open source projects that are alternatives of or similar to Mailinglistscraper

golang实现的爬虫框架，使用者只需关心页面规则，提供web管理界面。基于colly开发。

基于Scrapy的爬虫demo

基于 scrapy-redis 的通用分布式爬虫框架

API for scrapping common meme sites

AngularJS SPA interface for awesome lists. Awesome lists parsed using python.

爬蟲極簡教學（fetch, parse, search, multiprocessing, API）- PTT 為例

拟物校园，一个开源的高校教务移动化解决方案。

Declarative web scraping

📰 A responsive interface of Hacker News with summaries and thumbnails.

Use Google Flights API and scrape Expedia to find the cheapest/shortest flights!

NetDiscovery 是一款基于 Vert.x、RxJava 2 等框架实现的通用爬虫框架/中间件。

扒网站工具，看好哪个网站，指定好URL，自动扒下来做成模版。所见网站，皆可为我所用！

Some useful scripts simplifying bureaucracy

All-in-one Python (And now Go!) module to help solve captchas with Capmonster, 2captcha and Anticaptcha API's!

[WIP] GOPA, a spider written in Golang, for Elasticsearch. DEMO: http://index.elasticsearch.cn

A framework for creating semi-automatic web content extractors

A Simple, LightWeight, Statically-Typed Python3 API wrapper for GogoAnime.

Docs and files for ScrapydWeb, Scrapyd, Scrapy, and other projects

fetch data from Facebook, Instagram and LinkedIn

jobSpider是一只scrapy爬虫，用于爬取职位信息

Let's you to access your FB account from the command line and returns various things number of unread notifications, messages or friend requests you have.

Scrape all the media from an OnlyFans account - Updated regularly

抓取网易云音乐热门评论

QQ空间爬虫，可导出并显示日志、相册、留言板、说说、照片、视频等数据。

Torrent Api ✨

QQ空间导出助手，用于备份QQ空间的说说、日志、私密日记、相册、视频、留言板、QQ好友、收藏夹、分享、最近访客为文件，便于迁移与保存

Proxy pool. Finds and checks proxies with rest api for querying results. Can find over 25k proxies in under 5 minutes.

一款信息泄漏利用工具，适用于.git/.svn源代码泄漏和.DS_Store泄漏

Checks USD/PYG exchange rate from several sites, with a calculator, RESTful API and a twitter bot

Scraping Ebay's products using Scrapy Web Crawling Framework

Extract structured data from web sites. Web sites scraping.

Android Web Scraper is a simple library for android web automation. You can perform web task in background to fetch website data programmatically.

🍭 epub generator for (light)novels (轻) 小说 epub 生成器，支持站点：轻之国度、轻小说文库

Bangumi小工具

📷 Instagram private API in Java

Schedules tweets using TweetDeck

✨ 使用Node监视支付宝订单，即时通知服务器以实现免签约支付接口

Make a ZIM file from any Web site and surf offline!

Twitter Troll & Fake News Hunter - Crawls news websites and twitter to identify fake news

LinkedIn Scraper (currently working 2020)

Blog do Brasil.IO

A fast tool to fetch URLs from HTML attributes by crawl-in.

zhihu-crawler是一个基于Java的高性能、支持免费http代理池、支持横向扩展、分布式爬虫项目

A distributed web crawler framework.（分布式爬虫框架XXL-CRAWLER）

A scraper to grab and publish TTC subway arrival times.

🕸 Generates and delivers RSS feeds via HTTP. Create your own feeds or get started quickly with the included configs.

Command line program to download documents from web portals.

Crawl BookCorpus

CHeSF is the Chrome Headless Scraping Framework, a very very alpha code to scrape javascript intensive web pages

Stream anime from kaa.si and sync with anilist

scraper for Rensselaer Polytechnic Institute (RPI)'s Covid Dashboard

🍻 bilibili video (including bangumi) and danmaku downloader | B站视频（含番剧）、弹幕下载器

PTT 網路版爬蟲

A collection of pipelines for Scrapy

Automated Script to scrape flight prices from any website into a csv format

An unofficial DuckDuckGo search API.

哔哩哔哩的API调用模块

基于搜狗微信搜索的微信公众号爬虫接口

各种App、小程序、网站的请求签名或加密算法。 现已有：自如、小红书、蛋壳公寓、luckin coffee(瑞幸咖啡)、bangkokair(曼谷航空)

A Scraper, Downloader, & Recorder for static open directories.