Top 135 web-scraping open source projects

A framework for creating semi-automatic web content extractors

List of libraries, tools and APIs for web scraping and data processing.

Python binding to Modest engine (fast HTML5 parser with CSS selectors).

A Smart, Automatic, Fast and Lightweight Web Scraper for Python

ACHE is a web crawler for domain-specific search.

NBA Stats API via Basketball Reference

[WIP] GOPA, a spider written in Golang, for Elasticsearch. DEMO: http://index.elasticsearch.cn

Apify SDK — The scalable web scraping and crawling library for JavaScript/Node.js. Enables development of data extraction and web automation jobs (not only) with headless Chrome and Puppeteer.

PHP Curl Class makes it easy to send HTTP requests and integrate with web APIs

API to get user details for competitive coding platforms - Codeforces, Codechef, SPOJ, Interviewbit

[Python] Scraps comics and manga from various websites and creates cbz files from them

Project on building a web crawler to collect the fundamentals of the stock and review their performance in one go

Explaining textual analysis tools in Python. Including Preprocessing, Skip Gram (word2vec), and Topic Modelling.

📓 Minicurso de raspagem de dados web com Python ministrado na Semana de Tecnologia da FATEC Jundiaí

Computational Thinking for Social Scientists book project

Content based movie recommendation system with sentiment analysis

📖 Using deep learning and scraping to analyze/summarize articles! Just drop in any URL!

Papercut is a scraping/crawling library for Node.js built on top of JSDOM. It provides basic selector features together with features like Page Caching and Geosearch.

A web scraping tool to systematically extract the text of scientific papers and corresponding metadata from university accessible journals.

Audio Book scrapper

A Docker tutorial using a link extraction application example

🚆This web scraper builds a dataset for São Paulo subway operation status

🇺🇸 Is the US flag at half-staff?

Web application hosted on Heroku cloud platform based on web scraping in python using lxml library (XML Path Language).

Materials to reproduce our findings in our stories, "Amazon Puts Its Own 'Brands' First Above Better-Rated Products" and "When Amazon Takes the Buy Box, it Doesn’t Give it up"

Simple search for organisations participating/participated in the GSoC

House of Apify Scrapers. Generic scraping actors with a simple UI to handle complex web crawling and scraping use cases.

Build a Flask web application to help users retrieve key restaurant information and feature-based reviews (generated by applying market-basket model – Apriori algorithm and NLP on user reviews).

Heroshi – open source web crawler.

Scape top GitHub repositories and users based on keywords

Tableau scraper python library. R and Python scripts to scrape data from Tableau viz

No description or website provided.

Text Mining for Indonesian Online News Articles About Corona

Scraping Ebay's products using Scrapy Web Crawling Framework

WhatsApps related deaths News Articles along with other articles across India during that period

Scrapy project for scraping data from IMDB with Movie Dataset including 58,623 movies' data.

📻 An OLX Scraper using Scrapy + MongoDB. It Scrapes recent ads posted regarding requested product and dumps to NOSQL MONGODB.

Simple scripts that I'm using to automate the boring things.

This repository contains very useful restful API's and functionalities in node-js containing many important tutorial code for mastering node-js, all tutorials have been published on medium.com, tutorials link is given below

Compensation analysis on the posts scraped from leetcode.com/discuss/compensation. At present, the reports have been generated only for Indian cities.

(DISCONTINUED) Save WhatsApp Web Sessions as files and open them everywhere!

𝐫⟋ Get Reddit data

A Node.js library to easily manage and rotate a pool of web browsers, using any of the popular browser automation libraries like Puppeteer, Playwright, or SecretAgent.

A Dragnet that also extract author, headline, date, keywords from context

AI based web-wrapper for web-content-extraction

Web scraper for grabing data from Linkedin profiles or company pages (personal project)

🕸🧰☕️Tools to Scrape Dynamic Web Content via the 'HtmlUnit' Java Library

web scraping tool for grailed.com

Simplify your ETL processes with these hands-on data sanitation tips, tricks, and best practices

Searching torrents on popular trackers - CLI, readline, GUI, web client. Tutorial and binaries (issue tracker on https://gitlab.com/vindarel/cl-torrents/)

Python API to extract data from rateyourmusic.com.

A library to read a YML file with Xpath or CSS Selectors and extract data from HTML pages using them

The API for Furaffinity you wish existed

covers python basic to advance topics, practice questions, logical problems in python, web development using html, css, bootstrap, jquery, DOM, Django 🚀🚀. 💥 🌈

🚀「Douyin_TikTok_Download_API」是一个开箱即用的高性能异步抖音|TikTok数据爬取工具，支持API调用，在线批量解析及下载。

🕸→ℹ️ Reap Information from Websites

Web Scraping Framework

Bulk Downloader for Reddit

Stock Market Predictor with LSTM network. Web scraping and analyzing tools (ohlc, mean)

You can use this act to monitor any page's content and get a notification when content changes.