22 Open source projects that are alternatives of or similar to warc

Parse And Create Web ARChive (WARC) files with node.js

Query 'GreyNoise Intelligence 'API' in R

Read Web ARChive (WARC) files in PHP.

🗃 Work With Filesystem Object Extended Attributes — https://hrbrmstr.github.io/xattrs/index.html

👀 Analyze Websites and Resources They Request

Wget-AT is a modern Wget with Lua hooks, Zstandard (+dictionary) WARC compression and URL-agnostic deduplication.

🐋 One-Click User Instigated Preservation

🕸🧰☕️Tools to Scrape Dynamic Web Content via the 'HtmlUnit' Java Library

A small tool which uses the CommonCrawl URL Index to download documents with certain file types or mime-types. This is used for mass-testing of frameworks like Apache POI and Apache Tika

🍯 Analyze and Visualize Data from Modern Honey Network Servers with R

🕸→ℹ️ Reap Information from Websites

Tools to work with the Google DNS over HTTPS API in R

A robust web archive analytics toolkit

➰ ➡️ ➖ Translate cURL command lines into parameters for use with httr or actual httr calls (R)

📄◻️ Create, Maniuplate and Extract Data from PDF Files (R Apache PDFBox wrapper)

⏪ Tools to Work with the Various Internet Archive Wayback Machine APIs

Heritrix is the Internet Archive's open-source, extensible, web-scale, archival-quality web crawler project.

🗃 Open source self-hosted web archiving. Takes URLs/browser history/bookmarks/Pocket/Pinboard/etc., saves HTML, JS, PDFs, media, and more...

⚙️ A Rust library for reading and writing WARC files

🌑 R package to work with the Shodan API

🔨 Tools to Work with the 'webhose.io' 'API' in R

📔 Extract plain or structured text from HTML content in R