Top 229 scraping open source projects

Nim library for querying HTML using CSS-selectors (like JavaScripts document.querySelector)

Contains all the public APIs listed in Phantombuster's API store. Pull requests welcome!

Torrengo is a CLI (command line) program written in Go which concurrently searches torrents from various sources.

OCaml functional web scraping library

自学入门 Python 优质中文资源索引，包含 书籍 / 文档 / 视频，适用于 爬虫 / Web / 数据分析 / 机器学习 方向

Code for the collection and analysis of the MTNT dataset

artoo.js - the client-side scraping companion.

Creating Scrapy scrapers via the Django admin interface

Tracking PG&E outages

Public, free to use, repository with diggers configs for scraping / extracting data from various e-commerce websites and online stores

Search for and retrieve US Patent and Trademark Office Patent Data

This Scrapy project uses Redis and Kafka to create a distributed on demand scraping cluster.

Scrape the Instagram frontend. Inspired from twitter-scraper by @kennethreitz.

HTML scraping for Objective-C.

[Unmaintained] A simple and clean video/music/image downloader 👾

✂️ High performance, multi-threaded image scraper

Parsel lets you extract data from XML/HTML documents using XPath or CSS selectors

Free Web Scraping Tool with Java

Simple but useful Python web scraping tutorial code.

Tabula is a tool for liberating data tables trapped inside PDF files

Distributed crawler powered by Headless Chrome

🥫 The simple, fast, and modern web scraping library

Tools for various online judges. Downloading sample cases, generating additional test cases, testing your code, and submitting it.

Analyze facebook copy of your data with ruby language. Download zip file from facebook and get info about friends ranking by message, vocabulary, contacts, friends added statistics and more

Scrape Facebook public pages without an API key

Web scraping library made by the Phantombuster team. Modern, simple & works on all websites. (Deprecated)

Topic wise PDFs of Geeks for Geeks articles. (Last updated in October 2018)

Declarative web scraping

A framework for creating semi-automatic web content extractors

Extract structured data from web sites. Web sites scraping.

Crawly, a high-level web crawling & scraping framework for Elixir.

Jekyll-based static site for The Programming Historian

Lookyloo is a web interface that allows users to capture a website page and then display a tree of domains that call each other.

Custom Selenium Chromedriver | Zero-Config | Passes ALL bot mitigation systems (like Distil / Imperva/ Datadadome / CloudFlare IUAM)

Collection of useful data science topics along with code and articles

COVID-19 Coronavirus data scraped from government and curated data sources.

Build and deploy a machine learning app from scratch 🚀

Comic-dl is a command line tool to download manga and comics from various comic and manga sites. Supported sites : readcomiconline.to, mangafox.me, comic naver and many more.

A Python Tool For google Hacking

Social media scraping / data collection library for Facebook, Twitter, Reddit, YouTube, Pinterest, and Tumblr APIs

A Smart, Automatic, Fast and Lightweight Web Scraper for Python

🧶 Extract data from any website without code, just clicks.

📇 Extract social media profiles and more with regular expressions

Scrapy Extension for monitoring spiders execution.

Linkedin Scraper using Selenium Web Driver, Chromium headless, Docker and Scrapy

Scrape any website, article or RSS/Atom Feed with ease!

Generate Free Edu Mail(s) within minutes

一个灵活、友好的爬虫框架

🧹 Python package for text cleaning

Crawlera middleware for Scrapy

Functional HTML scraping and rewriting with CSS in OCaml

[WIP] GOPA, a spider written in Golang, for Elasticsearch. DEMO: http://index.elasticsearch.cn

Apify SDK — The scalable web scraping and crawling library for JavaScript/Node.js. Enables development of data extraction and web automation jobs (not only) with headless Chrome and Puppeteer.

Mechanize is a ruby library that makes automated web interaction easy.

Schedules tweets using TweetDeck

📷 An app to scrap instagram posts and analyze data.

A collection of tools to (semi-)automatically collect and analyze data from online discussions on Facebook groups and pages.

The Scripting Engine that Combines Speed, Safety, and Simplicity

ARGUS is an easy-to-use web scraping tool. The program is based on the Scrapy Python framework and is able to crawl a broad range of different websites. On the websites, ARGUS is able to perform tasks like scraping texts or collecting hyperlinks between websites. See: https://link.springer.com/article/10.1007/s11192-020-03726-9

No description or website provided.