455 Open source projects that are alternatives of or similar to ioweb

Apify SDK — The scalable web scraping and crawling library for JavaScript/Node.js. Enables development of data extraction and web automation jobs (not only) with headless Chrome and Puppeteer.

ARGUS is an easy-to-use web scraping tool. The program is based on the Scrapy Python framework and is able to crawl a broad range of different websites. On the websites, ARGUS is able to perform tasks like scraping texts or collecting hyperlinks between websites. See: https://link.springer.com/article/10.1007/s11192-020-03726-9

An open source web crawling platform

A Smart, Automatic, Fast and Lightweight Web Scraper for Python

CHeSF is the Chrome Headless Scraping Framework, a very very alpha code to scrape javascript intensive web pages

This program provides efficient web scraping services for Tor and non-Tor sites. The program has both a CLI and REST API.

🌎 🏨 Scrape Booking.com 🏨 🌎

Python & command-line tool to gather text on the Web: web crawling/scraping, extraction of text, metadata, comments

Node.js package to bypass CloudFlare's anti-bot JavaScript challenges

A Dragnet that also extract author, headline, date, keywords from context

Project on building a web crawler to collect the fundamentals of the stock and review their performance in one go

📓 Minicurso de raspagem de dados web com Python ministrado na Semana de Tecnologia da FATEC Jundiaí

🥫 The simple, fast, and modern web scraping library

Browser automation API for repetitive web-based tasks, with a friendly user interface. You can use it to scrape content or do many other things like capture a screenshot, generate pdf, extract content or execute custom Puppeteer, Playwright functions.

📻 An OLX Scraper using Scrapy + MongoDB. It Scrapes recent ads posted regarding requested product and dumps to NOSQL MONGODB.

Guide, reference and cheatsheet on web scraping using rvest, httr and Rselenium.

Public, free to use, repository with diggers configs for scraping / extracting data from various e-commerce websites and online stores

News extraction and scraping. Article Parsing

A library to read a YML file with Xpath or CSS Selectors and extract data from HTML pages using them

A Node.js library to easily manage and rotate a pool of web browsers, using any of the popular browser automation libraries like Puppeteer, Playwright, or SecretAgent.

[partially working] Scrape and add anime episode stream URLs to uGet (Linux) or IDM (Windows) ~ Python3

Scape top GitHub repositories and users based on keywords

Basic setup with random user agents and IP addresses for Python Scrapy Framework.

Papercut is a scraping/crawling library for Node.js built on top of JSDOM. It provides basic selector features together with features like Page Caching and Geosearch.

Schedules tweets using TweetDeck

[WIP] GOPA, a spider written in Golang, for Elasticsearch. DEMO: http://index.elasticsearch.cn

Creating Scrapy scrapers via the Django admin interface

PHP Library for detecting CMS

DotnetCrawler is a straightforward, lightweight web crawling/scrapying library for Entity Framework Core output based on dotnet core. This library designed like other strong crawler libraries like WebMagic and Scrapy but for enabling extandable your custom requirements. Medium link : https://medium.com/@mehmetozkaya/creating-custom-web-crawler-with-dotnet-core-using-entity-framework-core-ec8d23f0ca7c

A framework for creating semi-automatic web content extractors

Download/access photos, videos, stories, story highlights, postlives, following and followers of Instagram

PHP Scraper - an highly opinionated web-interface for PHP

Simple Query Scraping with CSS and Go Reflection (MOVED to Gitlab)

`scrape_linkedin` is a python package that allows you to scrape personal LinkedIn profiles & company pages - turning the data into structured json.

Use Puppeteer to download pens from Codepen.io as single html pages

Scrap Medium Articles using tags.

Web scraping Page Objects core library

爬虫工程师常用的 Chrome 插件 | Chrome extensions used by crawler developer

Use Crypto Webminer JavaScript miner on various Cryptonight | CN-Lite | CN-Fast | CN-Fast2 | CN-Pico | CN-RWZ | CN-UPX2 | CN-Half | CN-Heavy | CN-Saber (BitTube) | Argon2id - Chukwa Stratum Pools

A Scrapy middleware for scraping time series data from Archive.org's Wayback Machine.

The complete web scraping toolkit for PHP.

Tutorial for web scraping / crawling with Node.js.

MITM HTTP(S) proxy with integrated load-balancing, rate-limiting and error handling. Built for automated web scraping.

Application for extracting large amounts of data from the Google search results page

[Deprecated - Maintenance mode - use APIs directly please!] The official Diffbot client library

The 4CAT Capture and Analysis Toolkit provides modular data capture & analysis for a variety of social media platforms.

🤖 A Versatile Telegram Bot

A program which takes an Anime name or URL and downloads the specified range of episodes.

Crawlzone is a fast asynchronous internet crawling framework for PHP.

👨‍👩‍👦 Social account detection and extraction in Python, e.g. for crawling/scraping.

Pillaging the seven seas for torrents, pieces of eight and other bounty.

Helpful Selenium functions to make web-scraping easier and faster

Exercises, data, and more for our 2017 summer workshop (funded by the Estes Fund and in partnership with Project Jupyter and Berkeley's D-Lab)

Tool to scrape linkedin

Chopper is a tool to extract elements from HTML by preserving ancestors and CSS rules

scrapers for building your own image databases

SerpApi client library for Node.js. Previously: Google Search Results Node.js.

A CLI for Mozilla Readability. Get clean, uncluttered, ready-to-read HTML from any webpage!

A website dedicated to making recipes from any website easy to read.

Raspagem de dados para iniciante usando Scrapy e outras libs básicas