35 Open source projects that are alternatives of or similar to CommonCrawlDocumentDownload

Khudro is a very light weight web-server built with C.

A simple Mime type getter

🕷️ The pipeline for the OSCAR corpus

A robust web archive analytics toolkit

PHP mime type and extension mapping library: built with jshttp/mime-db, compatible with Symfony and Laravel

Word analysis, by domain, on the Common Crawl data set for the purpose of finding industry trends

MIME Sniffing Standard

Send email in Python conveniently for gmail using yagmail

Main Package for IO, loading all different kind of files

a simple, customisable theme for your apache directory listing

🗂 Swift MIME type checking based on magic bytes

The Hoa\Mime library.

Fast, dependency-free, small Go package to infer the binary file type based on the magic numbers signature

A different take on the UNIX tool cat

Map filenames to MIME types

SixArm.com » Ruby » MagicNumberType infers a data type from the data's leading bytes

The ultimate javascript content-type utility.

Shared MIME-info database in D programming language

Media Type Database

A fast golang library for MIME type and file extension detection, based on magic numbers

Ruby MIME type registry library

🔥🔥🔥Android文件、图片选择器，可按文件夹查找，文件类型查找，支持自定义相机

Simple interface to libmagic for Ruby Programming Language

.NET wrapper for libmagic

Erlang MIME types library

Share data beyond the browser boundaries. Enable users to transfer data to external apps, and open your webapp to receive external data.

Simple dictionary provides a few methods to lookup mime type/extension, generated From Apache's mime.types.

Heritrix is the Internet Archive's open-source, extensible, web-scale, archival-quality web crawler project.

🗃 Open source self-hosted web archiving. Takes URLs/browser history/bookmarks/Pocket/Pinboard/etc., saves HTML, JS, PDFs, media, and more...

⚙️ A Rust library for reading and writing WARC files

Parse And Create Web ARChive (WARC) files with node.js

📇 Tools to Work with the Web Archive Ecosystem in R

Read Web ARChive (WARC) files in PHP.

Wget-AT is a modern Wget with Lua hooks, Zstandard (+dictionary) WARC compression and URL-agnostic deduplication.

🐋 One-Click User Instigated Preservation