278 Open source projects that are alternatives of or similar to dust

Scrape and take screenshots of dynamic and static webpages

Monitor instagram user account and automatically post new images to discord channel via a webhook. Working 2022!

🌍 Dataset that shows the Internet affordability by country (a shocking reality!)

This script is used to scrap images from the Internet to classify 5 common noodle "mee" dishes in Singapore. Wanton Mee, Bak Chor Mee, Lor Mee, Prawn Mee and Mee Siam.

Retrieve real (with Javascript executed) HTML code from an URL, ultra fast and supports multiple parallel loading of webs

Crawl sites for RSS, Atom, and JSON feeds.

Simple and friendly Bot for Instagram, using Selenium and Scrapy with Python.

Kuwala is the no-code data platform for BI analysts and engineers enabling you to build powerful analytics workflows. We are set out to bring state-of-the-art data engineering tools you love, such as Airbyte, dbt, or Great Expectations together in one intuitive interface built with React Flow. In addition we provide third-party data into data sc…

Browser automation API for repetitive web-based tasks, with a friendly user interface. You can use it to scrape content or do many other things like capture a screenshot, generate pdf, extract content or execute custom Puppeteer, Playwright functions.

CHeSF is the Chrome Headless Scraping Framework, a very very alpha code to scrape javascript intensive web pages

C++ HTTP requests for humans

🔎Search LinkedIn profile by email address📧

A lightweight, simple plugin that allows you to intercept request and response objects and modify them if desired.

Spin up Tor containers and then proxy HTTP requests via these Tor instances

📉 Uptime and error monitoring CLI

[partially working] Scrape and add anime episode stream URLs to uGet (Linux) or IDM (Windows) ~ Python3

Transform unstructured document collections to structured Linked Data

Relay lets you write HTTP requests as easy to read, structured YAML and dispatch them easily using a CLI. Similar to tools like Postman

A POSIX shell script to parse HTML

Home Assistant custom component for scraping (html, xml or json) multiple values (from a single HTTP request) with a separate sensor/attribute for each value. Support for (login) form-submit functionality.

Your good mornings ☀️

Kotlin Wrapper for Jsoup

HTML parsing and processing tool for PowerShell.

🕵‍♂ Bot detection tests for Puppeteer. Hide and seek!

The package provides a very thin wrapper (no external dependencies) for http.Client allowing the use of layers (middleware).

Wget-AT is a modern Wget with Lua hooks, Zstandard (+dictionary) WARC compression and URL-agnostic deduplication.

A library to read a YML file with Xpath or CSS Selectors and extract data from HTML pages using them

Library to download subtitles from subscene.com

Open Graph Protocol Parser for Node.js

🕸 List of mini projects that involve web scraping 🕸

No description or website provided.

Generate HAR entries for requests made with node-fetch

A Node.js library to easily manage and rotate a pool of web browsers, using any of the popular browser automation libraries like Puppeteer, Playwright, or SecretAgent.

A GUI for Instaloader to scrape users and hashtags with on Instagram

Mastering the art of scrapping 🎓

All-in-one Python (And now Go!) module to help solve captchas with Capmonster, 2captcha and Anticaptcha API's!

A PHP Scraping Class

Easily download the main content of a web page in html, markdown, and/or epub format from command line.

Scrapping Facebook with JavaScript.

Proxy pool. Finds and checks proxies with rest api for querying results. Can find over 25k proxies in under 5 minutes.

ProxyCrawl Python library for scraping and crawling

Collection of scrapy spiders which can scrape posts, images, and so on from public Facebook Pages.

Command-line utility to convert HTML tables into CSV files

A series of distributed components for Scrapy. Including RabbitMQ-based components, Kafka-based components, and RedisBloom-based components for Scrapy.

Mock and stub HTTP requests. Test your apps with fake data and files responses.

Parse a raw HTTP request and generate request code in different languages

An open source web crawling platform

Command line program to download documents from web portals.

Succint and readable integration tests over RESTful APIs

Twitter bot powering @arichduvet

extract data from html table

Web crawling and scraping framework for Golang

youtube & tiktok analysis + youchoose recommendation custmizer. backend, extensions, and tooling

Core Node.js HTTP client

A scan of all .gov.uk sites for the most common security headers or lack of

Screen scraping and web crawling framework

The best way to do HTTP requests in NativeScript, a drop-in replacement for the core HTTP with important improvements and additions like proper connection pooling, form data support and certificate pinning

Download images from Google Image Search

Scape top GitHub repositories and users based on keywords

Rubium is a lightweight alternative to Selenium/Capybara/Watir if you need to perform some operations (like web scraping) using Headless Chromium and Ruby