Top 392 scraper open source projects

❄️ Extract links, ids, and names from a youtube playlist

Home Assistant custom component for scraping (html, xml or json) multiple values (from a single HTTP request) with a separate sensor/attribute for each value. Support for (login) form-submit functionality.

Tools for downloading agendas, minutes and other documents produced by local government

By serving course 📚 data that is more "edible" 🍰 for developers, we hope CourseCake offers a smooth approach to build useful tools for students.

Extract rich metadata from URLs

Download public videos on TikTok using Python with Selenium

robots.txt file parsing and checking for R

StackExchange websites to ZIM scraper

Scrape all the media from an OnlyFans account - Updated regularly

Download subreddit comments

Instagram get images 🌄 (hashtags, account, locations) with puppeteer

💾 YouTube video metadata archiver written in Golang

Retrieve real (with Javascript executed) HTML code from an URL, ultra fast and supports multiple parallel loading of webs

在线爬虫，online web scraper

Python API to extract data from rateyourmusic.com.

NodeJS application for scraping recent top posts from Instagram by hashtag without API access.

Simple Flickr Image Scraper

👾 Fast and simple video download library and CLI tool written in Go

Extract all employees from LinkedIn. Especially useful for companies with thousands of employees.

...Just a place for me to share my various YT-DLP & related bash scripts.

Automation for updating an index of live George Floyd protest streams

An easiest crawling and scraping module for NestJS

A scraper for videos that are trending on YouTube (https://www.youtube.com/feed/trending)

Automated Tinder bot and scraper using selenium in python.

Fast concurrent image scraper

A extensible app for scraping property listings

Scrape everything you can from MyAnimeList.net

A Python class that scrapes ESRI Rest Endpoints and exports data to a geodatabase

Scripts to maintain German law git repository

🚀「Douyin_TikTok_Download_API」是一个开箱即用的高性能异步抖音|TikTok数据爬取工具，支持API调用，在线批量解析及下载。

DorkScout - Golang tool to automate google dork scan against the entiere internet or specific targets

NTU CEIBA 資料下載工具

A LinkedIn Scraper to scrape up to 10k LinkedIn profiles from company profile links and save their e-mail addresses if available!

Continuously search imageboards threads for images/webms and download them

A JS package for scraping recipes from the web.

web scraping extension

Easily allows for scraping a VSCO

图片爬取下载工具，极速爬取下载 站酷https://www.zcool.com.cn/, CNU 视觉 http://www.cnu.cc/ 设计师/用户 上传的 图片/照片/插画。

Converts website to json using jQuery selectors

Bulk Downloader for Reddit

Aviation charts

Scrape boards and threads from 4chan (8kun WIP). Downloads images, videos and HTML if desired.

Grabs all of the audio files from all of the Blinkist books

Extract emails from a given website

[Deprecated - Maintenance mode - use APIs directly please!] The official Diffbot client library

A web crawler for Go

Essential PHP library that scrapes HTTP(S) and UDP trackers for torrent information.

Unofficial client to access your Itaú bank data

Analyse all kinds of data for a TV series

Twitter High level scraper for humans.

Proof of concept for a security issue (in my opinion) that I found in accounts.google.com

Scraping Visually Similar Images from Pinterest

A scraper for obtaining information on the workings of the Belgian Federal Parliament.

scraper for facebook, gab, google and tiktok

Web scraping using python, requests and selenium

Parallel Searching and Crawling Data from PTT 🚀

🌎 🏨 Scrape Booking.com 🏨 🌎

Simple, easy-to-use scraper to scrape data from WordPress JSON API

A collection of random scripts I coded up

Simply scrape / download all the media from an fansly account