Top 80 crawling open source projects

Fetch, store and access user agent strings for different browsers

ProxyCrawl Python library for scraping and crawling

An open source web crawling platform

🕷 Automatically detect changes made to the official Telegram sites, clients and servers.

Easily crawl news portals or blog sites using Storm Crawler.

A Scrapy extension to log items coverage when the spider shuts down

[Deprecated - Maintenance mode - use APIs directly please!] The official Diffbot client library

Dataset search engine, discovering data from a variety of sources, profiling it, and allowing advanced queries on the index

A dataset for textual analysis on arguably the best written comedy television show ever.

b̶̡̪̬͒l̸̰̗̝̀ỏ̷̡̩g̴͇̑g̶̲̱̽͐i̵̹͗n̶̤̥͂̅̆g̴̮̾̅͜ ̷̧͎͆i̷̛͒͜͠n̸̥̺͒ ̶͚͚͊̿͜t̸̺͙̭̆̊̈́ḧ̶̟́̐e̸̱͔̟̓̓͝ ̶̨͔̾͛̑d̵̥̣̏ȧ̷̼̊r̷̰̝̥̅̌͝k̵̟̥̞̉̍͛

👨‍👩‍👦 Social account detection and extraction in Python, e.g. for crawling/scraping.

Build a small, 3 domain internet using Github pages and Wikipedia and construct a crawler to crawl, render, and index.

github repo for MyAnimeList analysis. Also links to the MAL dataset.

A test suite of common scraper detection techniques. See how detectable your scraper stack is.

The complete web scraping toolkit for PHP.

Tutorial for web scraping / crawling with Node.js.

PHP library to find podcasts

Puppeteer as a service hosted on Saasify.

SimFin's open source PDF crawler

项目已经移动至：https://github.com/BaiduSpider/BaiduSpider ！！ 一个爬取百度搜索结果的爬虫，目前支持百度网页搜索，百度图片搜索，百度知道搜索，百度视频搜索，百度资讯搜索，百度文库搜索，百度经验搜索和百度百科搜索。