670 Open source projects that are alternatives of or similar to pomp

Crawl sites for RSS, Atom, and JSON feeds.

Simple but useful Python web scraping tutorial code.

Declarative web scraping

Apify SDK — The scalable web scraping and crawling library for JavaScript/Node.js. Enables development of data extraction and web automation jobs (not only) with headless Chrome and Puppeteer.

一个灵活、友好的爬虫框架

Extract structured data from web sites. Web sites scraping.

No description or website provided.

A test suite of common scraper detection techniques. See how detectable your scraper stack is.

Easily crawl news portals or blog sites using Storm Crawler.

DotnetCrawler is a straightforward, lightweight web crawling/scrapying library for Entity Framework Core output based on dotnet core. This library designed like other strong crawler libraries like WebMagic and Scrapy but for enabling extandable your custom requirements. Medium link : https://medium.com/@mehmetozkaya/creating-custom-web-crawler-with-dotnet-core-using-entity-framework-core-ec8d23f0ca7c

Antch, a fast, powerful and extensible web crawling & scraping framework for Go

Elegant Scraper and Crawler Framework for Golang

A curated list of awesome puppeteer resources.

Linkedin Learning videos downloader

Grawler is a tool written in PHP which comes with a web interface that automates the task of using google dorks, scrapes the results, and stores them in a file.

[WIP] GOPA, a spider written in Golang, for Elasticsearch. DEMO: http://index.elasticsearch.cn

A series of distributed components for Scrapy. Including RabbitMQ-based components, Kafka-based components, and RedisBloom-based components for Scrapy.

Crawly, a high-level web crawling & scraping framework for Elixir.

ARGUS is an easy-to-use web scraping tool. The program is based on the Scrapy Python framework and is able to crawl a broad range of different websites. On the websites, ARGUS is able to perform tasks like scraping texts or collecting hyperlinks between websites. See: https://link.springer.com/article/10.1007/s11192-020-03726-9

Tutorial for web scraping / crawling with Node.js.

Distributed crawler powered by Headless Chrome

[Unmaintained] A simple and clean video/music/image downloader 👾

🕵️‍♂️ LinkedIn profile scraper returning structured profile data in JSON. Works in 2020.

Scrapy, a fast high-level web crawling & scraping framework for Python.

[Deprecated - Maintenance mode - use APIs directly please!] The official Diffbot client library

A Scrapy extension to log items coverage when the spider shuts down

An open source web crawling platform

Wget-AT is a modern Wget with Lua hooks, Zstandard (+dictionary) WARC compression and URL-agnostic deduplication.

👨‍👩‍👦 Social account detection and extraction in Python, e.g. for crawling/scraping.

Scrapy Extension for monitoring spiders execution.

Distributed crawling framework for documents and structured data.

ProxyCrawl Python library for scraping and crawling

Web crawling and scraping framework for Golang

A powerful, cool and well-made userbot for your Telegram profile, with promising extension capabilities.

A fully async ORM for Python 3.5+

MQTT version 5.0 client and broker using asyncio

A D port of Python's asyncio library

A Prometheus Python client library for asyncio-based applications

📉 Uptime and error monitoring CLI

A work-in-progress Discord API wrapper written in Python.

The ultrarapid and multifunctional wrapper over QIWI and YooMoney

Easily download the main content of a web page in html, markdown, and/or epub format from command line.

IETF PPSP RFC7574 in Python/asyncio

🐍 i2plib: A modern asynchronous library for building I2P applications

A POSIX shell script to parse HTML

Kuwala is the no-code data platform for BI analysts and engineers enabling you to build powerful analytics workflows. We are set out to bring state-of-the-art data engineering tools you love, such as Airbyte, dbt, or Great Expectations together in one intuitive interface built with React Flow. In addition we provide third-party data into data sc…

An async python wrapper to interact with the Steam API and its CMs

All-in-one Python (And now Go!) module to help solve captchas with Capmonster, 2captcha and Anticaptcha API's!

Mentally computable verification codes for email aliases implemented as a postfix tcp table or milter; uses asyncio.

Fetching Financial Data （US/China）

Transform unstructured document collections to structured Linked Data

📦 A simple API to track package info from 17track.com

An async/await wrapper around aria2-json-rpc with websocket support

Proxy pool. Finds and checks proxies with rest api for querying results. Can find over 25k proxies in under 5 minutes.

A web crawler that uses Firefox and js injection to interact with webpages and crawl their content, written in nodejs.

와이빅타 엔지니어링팀의 자료를 정리해두는 곳입니다.

🌍 Dataset that shows the Internet affordability by country (a shocking reality!)

Decode and verify Amazon Cognito JWT tokens

A package to mix django and asyncio in one application

Python 3 asyncio module for interfacing with Insteon Powerline modems