Top 392 scraper open source projects

Node.js scraper to get data from Google Play

A web crawler (for bug hunting) that gathers more than you can imagine.

🌐 Web scraping made easy with the visual 🗺 mind map editor to JSON

A Devtools driver for web automation and scraping

Automate the creation of Python Selenium Scripts by drawing coloured boxes on webpage elements

The most integrated web scraper package for Laravel.

Scrapoxy hides your scraper behind a cloud. It starts a pool of proxies to send your requests. Now, you can crawl without thinking about blacklisting!

This is a list of repositories and libraries that allow for scripted downloading of online content.

PhantomJS/Node.js web scraper for AWS Lambda

Python Package for scraping NHL Play-by-Play and Shift data

Google Maps reviews scraping

Python Library to download images and metadata from popular search engines.

An Instagram OSINT tool to collect all the geotagged locations available on an Instagram profile in order to plot them on a map, and dump them in a JSON.

Geziyor, a fast web crawling & scraping framework for Go. Supports JS rendering.

http interceptor to hoomanize cloudflare requests

The main functionality is to extract all the emails from one or several URLs - La funcionalidad principal es extraer todos los correos electrónicos de una o varias Url

Lightweight Ruby web crawler/scraper with an elegant DSL which extracts structured data from pages.

Social media research and promotion, semi-autonomous CLI bot

kikoeru 后端，不再维护，请到https://github.com/umonaca/kikoeru-express 获取更新

Library for scraping free proxies lists

🚀 A fast and lightweight utility and Python library for downloading posts, stories, and highlights from Instagram.

An API to easily get data from the University of Pittsburgh

Python3 API wrapper and web scraper for https://coinmarketcap.com

Golang pkg to quickly return a preview of a webpage (title/description/images)

Python爬虫，京东自动登录，在线抢购商品

Kotlin/Java library and cli tool for scraping posts and media from various sources with neither authorization nor full page rendering (Facebook, Instagram, Twitter, Youtube, Tiktok, Telegram, Twitch, Reddit, 9GAG, Pinterest, Flickr, Tumblr, IFunny, VK, Pikabu)

Web scraper that can create an offline readable version of a website

🎶 Unofficial python API for pitchfork.com reviews.

Live-scraping pastebin to fight boredom.

Distributed Scraper

🐙 Get Daily 400-500 Real Followers 👽 [BadRobo] is Best Instagram Bot Available Now with All Features!. Our BOT did not violate any of Instagram's rules, so you don't have to worry about getting ACTION BLOCK!

Indonesia Index News Crawler, including 10 online media

Tangerine Bank scraper

Intuitive timetable builder for the Australian National University.

An Unofficial Pitchfork Music API client for Node.js

An Instagram Scrapper

Tổng hợp script crawl dữ liệu từ các mạng xã hội & website tiếng Việt

Webscraper

Kodi Bae Repository - Kodi is a registered trademark of the XBMC Foundation. We are not connected to or in any other way affiliated with Kodi - DMCA: [email protected]

Creating Scrapy scrapers via the Django admin interface

Tool to fetch Instagram's public content.

Shopify Scraper (not monitor)

📙 中华新华字典数据库。包括歇后语，成语，词语，汉字。

AV 电影管理系统， avmoo , javbus , javlibrary 爬虫，线上 AV 影片图书馆，AV 磁力链接数据库，Japanese Adult Video Library,Adult Video Magnet Links - Japanese Adult Video Database

Google Search SERP Scraper

Messenger Bot that scrapes for COVID-19 data and periodically updates subscribers via Facebook Messages. Created using Python/Flask, MYSQL, HTML, Heroku

Web scraper that makes it easier to find real estate in Slovenia.

Search for and retrieve US Patent and Trademark Office Patent Data

Anitop is an unofficial simple API from https://anitrendz.net/ site

A Node.js wrapper for scraping allrecipes.com

Create agents that monitor and act on your behalf. Your agents are standing by!

Download website to local directory (including all css, images, js, etc.)

Fetches PubMed article IDs (PMIDs) from email inbox, then crawls PubMed, Google Scholar and Sci-Hub for respective PDF files.

online port scan scraper

Emby/Jellyfin 的一个日本电影刮削器插件，可以从某些网站抓取影片信息。

A scrapy spider that scraps times and prices from Voyages Sncf. It uses scrapyrt to provide an API interface.

A data scraper for retrieving old tweets in Twitter using Python3.

Scraping scripts for various websites.

Scrapping tool for GISAID data regarding SARS-CoV-2