Top 615 crawler open source projects

Amazon S3 bucket finder and crawler.

虎扑步行街爬虫

百度莱茨狗爬虫。

🤖 robots.txt as a service. Crawls robots.txt files, downloads and parses them to check rules through an API

英雄联盟胜负预测

Distributed crawling/scraping, Kafka And Redis based components for Scrapy

Backup your friends' Instagram Stories forever and get to keep them even after 24 hours.

🕷️ The pipeline for the OSCAR corpus

Python3 DHT 磁力种子爬虫 种子解析 种子搜索 演示地址

该项目是一个使用celery作为主体框架的爬虫应用，能够灵活的添加爬虫任务，并且同时运行多站点的爬虫工作，所有组件都能够原生支持规模并发和分布式，加上celery原生的分布式调用，实现大规模并发。

Explore a website recursively and download all the wanted documents (PDF, ODT…)

Powerful mutable web directory fuzzer to bruteforce existing and/or hidden files or directories.

Parallel Searching and Crawling Data from PTT 🚀

This is a Tiktok Crawler App.

🤖 Python bot that crawls your website looking for dead stuff

A spider on Dcard. Strong and speedy.

Crawling zhihu, jobbole, lagou by Scrapy, and using Elasticsearch+Django to build a Search Engine website --- README_zh.md (including: implementation roadmap, distributed-crawler and coping with anti-crawling strategies).

新浪博客文章/wenku8轻小说文库爬虫，可抓取图片保存，一键制作电子书。kindle读书党的神器！

Python & command-line tool to gather text on the Web: web crawling/scraping, extraction of text, metadata, comments

💫 Spider is a PHP library with easily module integration for crawling website that allows you to scrape informations.

Crawler dos dados metereológicos de estações convencionais do INMET (BDMEP)

[Updated] A simple python crawler for my tutorial blog at http://www.jianshu.com/p/8fb5bc33c78e

Fetch Iranian calendar events (Jalali, Hijri and Gregorian) from time.ir website

🕷️ A simple crawler (spider) writen in php just for fun, with zero dependencies

Lightweight library for scalable crawlers in Go.

nodejs 爬虫框架. crawler framework for nodejs

(2020年最新)斗鱼弹幕抓取及可视化管理平台第二版，提供弹幕抓取、弹幕实时发送速度可视化、抓取记录查询、弹幕下载、自定义关键词统计、铁粉统计、高光时刻自动捕获、高频弹幕词云等功能，起飞~~~

Crawling the data from lazada, websosanh, compare.vn, cdiscount and cungmua with flexible configs

使用asyncio和aiohttp开发的轻量级异步协程web爬虫框架

Grapy, a fast high-level web crawling framework for Python 3.3 or later base on asyncio.

爬虫工程师常用的 Chrome 插件 | Chrome extensions used by crawler developer

Google Play Unofficial Python 3 API Library

获取滚动新闻

🎵 缓存文件转换为 MP3 文件

PHP library to find podcasts

Iota is a web scraper which can find all of the images and links/suburls on a webpage

YT metadata exporter

sponge is a website crawler and links downloader command-line tool

Python Web Crawler with Selenium and PhantomJS

Collect email addresses by crawling search engine results.

Crawlzone is a fast asynchronous internet crawling framework for PHP.

Get notified of new listings on popular German real estate portals.

Dynamic configurable crawl (动态可配置化爬虫)

NASTY Advanced Search Tweet Yielder

Nodejs crawler for cnbeta.com

Python Crawler

Crawling and scraping the Web for fun and profit

Apify actor to scrape Youtube search results. You can set the maximum videos to scrape per page as well as the date from which to start scraping.

Price tracker of Amazon

crawler dev tools using electron webview

托福考位爬虫 NEEA TOEFL Testseat Crawler

Diskover Community Edition - Open source file indexer, file search engine and data management and analytics powered by Elasticsearch

TseTmc Crawler

📥 Downloader for lezhin comics

Python爬虫，京东自动登录，在线抢购商品

crawl QR-codes from search engines and look for bitcoin private keys

This is a Puppeteer+AngleSharp crawler console app samples, used C# 7.1 coding and dotnet core build.

Scrape Learning (ctrip)

SimFin's open source PDF crawler

Crawl all your citations from Google Scholar