1003 Open source projects that are alternatives of or similar to Mailinglistscraper

Creating Scrapy scrapers via the Django admin interface

robots.txt file parsing and checking for R

[Crawler/Scraper for Golang]🕷A lightweight distributed friendly Golang crawler framework.一个轻量的分布式友好的 Golang 爬虫框架。

Collection of scrapy spiders which can scrape posts, images, and so on from public Facebook Pages.

An open source webapp for scraping: towards a public service for webscraping

A Facebook crawler

📻 An OLX Scraper using Scrapy + MongoDB. It Scrapes recent ads posted regarding requested product and dumps to NOSQL MONGODB.

A web spider for shodan.io without using the Developer API.

A web crawler that uses Firefox and js injection to interact with webpages and crawl their content, written in nodejs.

A high performance web crawler in Elixir.

PHP Metacritic API - Mirrored by my GitLab

A django admin site for scrapy

Seeker - another job board aggregator.

Wget-AT is a modern Wget with Lua hooks, Zstandard (+dictionary) WARC compression and URL-agnostic deduplication.

🎊 Design and implement of lightweight crawler framework.

A series of distributed components for Scrapy. Including RabbitMQ-based components, Kafka-based components, and RedisBloom-based components for Scrapy.

10 photo website spiders, 10 个国外图库的 scrapy 爬虫代码

Word2vec 千人千面 个性化搜索 + Scrapy2.3.0(爬取数据) + ElasticSearch7.9.1(存储数据并提供对外Restful API) + Django3.1.1 搜索

API for fetching data from news websites.

Web scraper project to collect the useful Hong Kong weather data from HKO website

python爬虫小项目【持续更新】【笔趣阁小说下载、Tweet数据抓取、天气查询、网易云音乐逆向、天天基金网查询、微博数据抓取（生成cookie）、有道翻译逆向、企查查免登陆爬虫、大众点评svg加密破解、B站用户爬虫、拉钩免登录爬虫、自如租房字体加密、知乎问答

ARGUS is an easy-to-use web scraping tool. The program is based on the Scrapy Python framework and is able to crawl a broad range of different websites. On the websites, ARGUS is able to perform tasks like scraping texts or collecting hyperlinks between websites. See: https://link.springer.com/article/10.1007/s11192-020-03726-9

PTT 圖片下載器 (抓取整個看板的圖片，並用文章標題作為資料夾的名稱 ) (使用Scrapy)

Generating spiders dynamically to crawl and check those free proxy ip on the internet with scrapy.

An R web crawler and scraper

Uno Crawler por https://douban.com

一个基于webmagic框架二次开发的java爬虫框架实战，已实现能爬取腾讯，搜狐，今日头条（单独集成功能）等资讯内容，配合elasticsearch框架用法，实现了自动爬虫，已投入线上生产使用。

Linkedin Scraper using Selenium Web Driver, Chromium headless, Docker and Scrapy

A versatile Ruby web spidering library that can spider a site, multiple domains, certain links or infinitely. Spidr is designed to be fast and easy to use.

163music spider by scrapy.

免费 IP 代理池。Scrapy 爬虫框架插件

A web search engine built with Python which uses TF-IDF and PageRank to sort search results.

Get Aliexpress product details in JSON

HTTP API for Scrapy spiders

NScrapy is a .net core corss platform Distributed Spider Framework which provide an easy way to write your own Spider

A multi-thread crawler framework with many builtin image crawlers provided.

收集各种免费的 Python 爬虫项目

一个基于Scrapy的数据采集爬虫代码库

News extraction and scraping. Article Parsing

豆瓣电影top250、斗鱼爬取json数据以及爬取美女图片、淘宝、有缘、CrawlSpider爬取红娘网相亲人的部分基本信息以及红娘网分布式爬取和存储redis、爬虫小demo、Selenium、爬取多点、django开发接口、爬取有缘网信息、模拟知乎登录、模拟github登录、模拟图虫网登录、爬取多点商城整站数据、爬取微信公众号历史文章、爬取微信群或者微信好友分享的文章、itchat监听指定微信公众号分享的文章

V2EX爬虫

Fresh Onions is an open source TOR spider / hidden service onion crawler hosted at zlal32teyptf4tvi.onion

今日头条科技新闻接口爬虫

基于Scrapy框架的豆瓣电影爬虫

Scrape Instagram content and stories anonymously, using a new technique based on the har file (No Token + No public API).

High performance crawler for Nodejs

🔧 🔩 🔨 收集整理了爬虫相关的工具、模拟登陆技术、代理IP、scrapy模板代码等内容。

百度贴吧爬虫(基于scrapy和mysql)

A set of spiders and scrapers to extract location information from places that post their location on the internet.

Download all the ebooks with indexed csv of "allitebooks.com"

快速、简洁且强大的PHP爬虫框架

Command line tool to download and extract data from HTML/XML pages or JSON-APIs, using CSS, XPath 3.0, XQuery 3.0, JSONiq or pattern matching. It can also create new or transformed XML/HTML/JSON documents.

A Smart, Automatic, Fast and Lightweight Web Scraper for Python

OSINT Swiss Army Knife

The Zipru scraper developed in the Advanced Web Scraping Tutorial.

Crawly, a high-level web crawling & scraping framework for Elixir.

日常爬虫

🚀「Douyin_TikTok_Download_API」是一个开箱即用的高性能异步抖音|TikTok数据爬取工具，支持API调用，在线批量解析及下载。

bingip2hosts is a Bing.com web scraper that discovers websites by IP address

🎊 Design and implement of lightweight crawler framework.