378 Open source projects that are alternatives of or similar to crawling-framework

Crawl sites for RSS, Atom, and JSON feeds.

An open source web crawling platform

[Unmaintained] A simple and clean video/music/image downloader 👾

A Scrapy extension to log items coverage when the spider shuts down

Distributed crawler powered by Headless Chrome

🌌 High productivity semi-automatic crawler generator 🛠️🧰

Tutorial for web scraping / crawling with Node.js.

Wget-AT is a modern Wget with Lua hooks, Zstandard (+dictionary) WARC compression and URL-agnostic deduplication.

Apify SDK — The scalable web scraping and crawling library for JavaScript/Node.js. Enables development of data extraction and web automation jobs (not only) with headless Chrome and Puppeteer.

Extract structured data from web sites. Web sites scraping.

Web crawling and scraping framework for Golang

Antch, a fast, powerful and extensible web crawling & scraping framework for Go

ARGUS is an easy-to-use web scraping tool. The program is based on the Scrapy Python framework and is able to crawl a broad range of different websites. On the websites, ARGUS is able to perform tasks like scraping texts or collecting hyperlinks between websites. See: https://link.springer.com/article/10.1007/s11192-020-03726-9

[Deprecated - Maintenance mode - use APIs directly please!] The official Diffbot client library

DotnetCrawler is a straightforward, lightweight web crawling/scrapying library for Entity Framework Core output based on dotnet core. This library designed like other strong crawler libraries like WebMagic and Scrapy but for enabling extandable your custom requirements. Medium link : https://medium.com/@mehmetozkaya/creating-custom-web-crawler-with-dotnet-core-using-entity-framework-core-ec8d23f0ca7c

Scrapy Extension for monitoring spiders execution.

一个灵活、友好的爬虫框架

🕵️‍♂️ LinkedIn profile scraper returning structured profile data in JSON. Works in 2020.

Grawler is a tool written in PHP which comes with a web interface that automates the task of using google dorks, scrapes the results, and stores them in a file.

Screen scraping and web crawling framework

No description or website provided.

Crawly, a high-level web crawling & scraping framework for Elixir.

[WIP] GOPA, a spider written in Golang, for Elasticsearch. DEMO: http://index.elasticsearch.cn

Scrapy, a fast high-level web crawling & scraping framework for Python.

A curated list of awesome puppeteer resources.

A series of distributed components for Scrapy. Including RabbitMQ-based components, Kafka-based components, and RedisBloom-based components for Scrapy.

Distributed crawling framework for documents and structured data.

A test suite of common scraper detection techniques. See how detectable your scraper stack is.

Declarative web scraping

ProxyCrawl Python library for scraping and crawling

Simple but useful Python web scraping tutorial code.

Elegant Scraper and Crawler Framework for Golang

👨‍👩‍👦 Social account detection and extraction in Python, e.g. for crawling/scraping.

scrapers for building your own image databases

A website dedicated to making recipes from any website easy to read.

Build a small, 3 domain internet using Github pages and Wikipedia and construct a crawler to crawl, render, and index.

Overwatch API library for player details and career stats

github repo for MyAnimeList analysis. Also links to the MAL dataset.

📊 Python tool to scrape real-time information about ETFs from the web and mixing them together by proportionally distributing their assets allocation

Instagram Photo Downloader

NBA Daily Fantasy Lineup Optimizer for FanDuel Using Python

Projeto de monitoramento de publicações oficiais relacionadas a COVID-19 no Brasil.

大数据生态圈学习

With Selenium headless browsing and CAPTCHA solving

**[ARCHIVED]** website changes tracker 🔍

Battle-tested Apache Storm Multi-Lang implementation for Python

The 4CAT Capture and Analysis Toolkit provides modular data capture & analysis for a variety of social media platforms.

爬虫工程师常用的 Chrome 插件 | Chrome extensions used by crawler developer

High quality web component for modal dialogs. Part of the Vaadin platform.

Dataset search engine, discovering data from a variety of sources, profiling it, and allowing advanced queries on the index

The complete web scraping toolkit for PHP.

🤖 A Versatile Telegram Bot

👔 Dimmed Color Theme for Sublime Text 2/3

PHP library to find podcasts

Crawlzone is a fast asynchronous internet crawling framework for PHP.

Download images from reddit

The documentation for the Vaadin framework version 10+

Storm Kafka 流数据 处理系统

Web Component for creating flexible responsive layouts and building nice looking dashboards.

Pillaging the seven seas for torrents, pieces of eight and other bounty.