Top 9 webcrawling open source projects

Heritrix is the Internet Archive's open-source, extensible, web-scale, archival-quality web crawler project.

API definition, resources and reference implementation of URL Frontiers

Project on building a web crawler to collect the fundamentals of the stock and review their performance in one go

ARGUS is an easy-to-use web scraping tool. The program is based on the Scrapy Python framework and is able to crawl a broad range of different websites. On the websites, ARGUS is able to perform tasks like scraping texts or collecting hyperlinks between websites. See: https://link.springer.com/article/10.1007/s11192-020-03726-9

News extraction and scraping. Article Parsing

This program provides efficient web scraping services for Tor and non-Tor sites. The program has both a CLI and REST API.

An open source web crawling platform

Web Scraping Framework

Raspagem de dados para iniciante usando Scrapy e outras libs básicas