Top 135 web-scraping open source projects

An unofficial API for Quora.

`scrape_linkedin` is a python package that allows you to scrape personal LinkedIn profiles & company pages - turning the data into structured json.

A command-line utility and Scrapy middleware for scraping time series data from Archive.org's Wayback Machine.

Công cụ quét và phân tích từ khoá các trang báo mạng Việt Nam

Scrape, standardize and share public meetings from local government websites

Selenium-python but lighter: Helium is the best Python library for web automation.

Python scripts for building 'Short Jokes' dataset, featured on Kaggle

Guide, reference and cheatsheet on web scraping using rvest, httr and Rselenium.

Tutorial: Web scraping in Python with Beautiful Soup

Machine Learning Model for Sport Predictions (Football, Basketball, Baseball, Hockey, Soccer & Tennis)

Twitter Intelligence OSINT project performs tracking and analysis of the Twitter

Web Scraping Framework

Web scrapping and related analytics using Python tools

This repository provides everything you need to get started with Python for (social science) research.

Scrapy Training companion code

Netflix like full-stack application with SPA client and backend implemented in service oriented architecture

Detailed web scraping tutorials for dummies with financial data crawlers on Reddit WallStreetBets, CME (both options and futures), US Treasury, CFTC, LME, SHFE and news data crawlers on BBC, Wall Street Journal, Al Jazeera, Reuters, Financial Times, Bloomberg, CNN, Fortune, The Economist

A Chrome extension for writing custom web scraping programs and web automation programs. Just demonstrate how to collect the first row of data, then let the extension write the program for collecting all rows.

Scrapy crawler to collect data on the back catalog of songs listed for sale.

PHP Scraper - an highly opinionated web-interface for PHP

Simple Query Scraping with CSS and Go Reflection (MOVED to Gitlab)

Zillow Scraper for Python using Selenium

MetaData html scraper and parser for Node.js (supports Promises and callback style)

Apify actor that opens a web page in headless Chrome and analyzes the HTML and JavaScript objects, looks for schema.org microdata and JSON-LD metadata, analyzes AJAX requests, etc.

Learn Python for the next 30 (or so) Days.

⚡️ Ayakashi.io - The next generation web scraping framework

General Assembly's 2015 Data Science course in Washington, DC

Android app for saving webpages for offline reading.

Set up free and scalable Scrapyd cluster for distributed web-crawling with just a few clicks. DEMO 👉

A Devtools driver for web automation and scraping

Turn large Web sites into tables and charts using simple SQLs.

Automate the creation of Python Selenium Scripts by drawing coloured boxes on webpage elements

💦 Tools to Work with the 'Splash' JavaScript Rendering Service in R

Python Package for scraping NHL Play-by-Play and Shift data

Node.js package to bypass CloudFlare's anti-bot JavaScript challenges

A library that enables programmatic interaction with daft.ie. Daft.ie has nationwide coverage and contains about 80% of the total available properties in Ireland.

Simple web scraping for R

PHP Library for detecting CMS

Extract clean(er), readable text from web pages via Mercury Web Parser.

Receive an email or Telegram message as soon as Migros Sanalmarket is available for delivery in your neighborhood.

Powerful web scraping framework for Crystal

Go cascadia package command line CSS selector

Headless 'Chrome' Orchestration in R

Fetch user's data across social media

Download/access photos, videos, stories, story highlights, postlives, following and followers of Instagram

Web Scraping Craigslist's Engineering Jobs in NY with Scrapy

A Router WiFi key recovery/cracking tool with a twist.

Apify actor that crawls Google Search result pages (SERPs) and extracts a list of organic results, ads, related queries and more. It supports selection of custom country, language and location.

📈 Regression and Classification with UC Davis student quiz data and exam data

Snoop — инструмент разведки на основе открытых данных (OSINT world)

Node.js framework for modular web scraping and data extraction

Scraping publicly-accessible Letterboxd data and creating a movie recommendation model with it that can generate recommendations when provided with a Letterboxd username

Collection of scripts corresponding to LucidProgramming YouTube tutorials

A versatile Ruby web spidering library that can spider a site, multiple domains, certain links or infinitely. Spidr is designed to be fast and easy to use.

Faster requests on Python 3

Nextjs server to query websites with GraphQL

The Python Code Tutorials

Random User-Agent middleware based on fake-useragent

A JavaScript library for generating random user agents with data that's updated daily.

UI.Vision: Open-Source RPA Software (formerly Kantu) - Modern Robotic Process Automation with Selenium IDE++