Top 41 deduplication open source projects

A list of free data matching and record linkage software.

Locality Sensitive Hashing using MinHash in Python/Cython to detect near duplicate text documents

Fast, secure, efficient backup program

A pair of kernel modules which provide pools of deduplicated and/or compressed block storage.

Find duplicate files

Quickly detect already witnessed data.

Userspace tools for managing VDO volumes.

Spark RDD with Lucene's query and entity linkage capabilities

Make it easier to compare and cross-reference the names of companies and people by applying strong normalisation.

Record Linkage ToolKit (Find and link entities)

Extremely fast tool to remove duplicates and other lint from your filesystem

FastCDC implementation in Rust

📊 python package for performing deduplication using flexible text matching and cleaning in pandas dataframe

Simple, configuration-driven backup software for servers and workstations

A powerful duplicate file finder and an enhanced fork of 'fdupes'.

Data deduplication engine, supporting optional compression and public key encryption.

Straightforward fuzzy matching, information retrieval and NLP building blocks for JavaScript.

A toolkit for record linkage and duplicate detection in Python

Cross-platform backup tool for Windows, macOS & Linux with fast, incremental backups, client-side end-to-end encryption, compression and data deduplication. CLI and GUI included.

Prometheus Alertmanager

A C library for parsing/normalizing street addresses around the world. Powered by statistical NLP and open geo data.

Dedupe/batch geocode addresses and venues around the world with libpostal

Accelerating the deduplication and collapsing process for reads with Unique Molecular Identifiers (UMI). Heavily optimized for scalability and orders of magnitude faster than a previous tool.

RocketMQ消息幂等去重消费者，支持使用MySQL或者Redis做幂等表，开箱即用

Resources for tackling record linkage / deduplication / data matching problems

Generate duplex/single consensus reads to reduce sequencing noises and remove duplications

PyTorch library for transforming entities like companies, products, etc. into vectors to support scalable Record Linkage / Entity Resolution using Approximate Nearest Neighbors.

Implementation of Fellegi-Sunter's canonical model of record linkage in Apache Spark, including EM algorithm to estimate parameters

Fast block-level out-of-band BTRFS deduplication tool.

A library for secure, deduplicated, transactional, and verifiable data storage

Tool for managing data-deduplication within extant compressed archive files, along with a relatively performant BK tree implementation for fuzzy image searching.

Yet Another Dupes Finder

Cargo with less noise: warnings are skipped until errors are fixed, Neovim integration, etc.

Scalable identity resolution, entity resolution, data mastering and deduplication using ML

Deduplicating archiver with encryption and paranoid-level tests. Swiss army knife for the serious backup and disaster recovery manager. Ransomware neutralizer. Win/Linux/Unix

📧 CLI to deduplicate mails from mail boxes.

Web Scraping, Document Deduplication & GPT-2 Fine-tuning with a newly created scam dataset.

Fast multi-threaded content-dependent chunking deduplication for Buffers in C++ with a reference implementation in Javascript. Ships with extensive tests, a fuzz test and a benchmark.

A backup program that does deduplication, compression, encryption

Deduplicating filesystem via Python3, FUSE and SQLite

Framework and command-line tools for integrating FollowTheMoney data streams from multiple sources