53 Open source projects that are alternatives of or similar to yadf

Scalable identity resolution, entity resolution, data mastering and deduplication using ML

📧 CLI to deduplicate mails from mail boxes.

Fast, secure, efficient backup program

Fast block-level out-of-band BTRFS deduplication tool.

A fast file deduplicator

Cargo with less noise: warnings are skipped until errors are fixed, Neovim integration, etc.

A collection of publicly available bug reports

Deduplicating archiver with encryption and paranoid-level tests. Swiss army knife for the serious backup and disaster recovery manager. Ransomware neutralizer. Win/Linux/Unix

Utilities for de-duping Django model instances

Web Scraping, Document Deduplication & GPT-2 Fine-tuning with a newly created scam dataset.

Fast multi-threaded content-dependent chunking deduplication for Buffers in C++ with a reference implementation in Javascript. Ships with extensive tests, a fuzz test and a benchmark.

Nextcloud Media Duplicate Collector application

A backup program that does deduplication, compression, encryption

Deduplicating filesystem via Python3, FUSE and SQLite

Near Duplicate Video Detection (Perceptual Video Hashing) - Get a 64-bit comparable hash-value for any video.

Framework and command-line tools for integrating FollowTheMoney data streams from multiple sources

Duplicate code finder for Elixir

Remove Duplicate Messages

🆔 A python library for accurate and scalable fuzzy matching, record deduplication and entity-resolution.

A list of free data matching and record linkage software.

Locality Sensitive Hashing using MinHash in Python/Cython to detect near duplicate text documents

A pair of kernel modules which provide pools of deduplicated and/or compressed block storage.

Find duplicate files

Quickly detect already witnessed data.

Userspace tools for managing VDO volumes.

Spark RDD with Lucene's query and entity linkage capabilities

Make it easier to compare and cross-reference the names of companies and people by applying strong normalisation.

Record Linkage ToolKit (Find and link entities)

Extremely fast tool to remove duplicates and other lint from your filesystem

FastCDC implementation in Rust

📊 python package for performing deduplication using flexible text matching and cleaning in pandas dataframe

Simple, configuration-driven backup software for servers and workstations

A powerful duplicate file finder and an enhanced fork of 'fdupes'.

Data deduplication engine, supporting optional compression and public key encryption.

Straightforward fuzzy matching, information retrieval and NLP building blocks for JavaScript.

A toolkit for record linkage and duplicate detection in Python

Cross-platform backup tool for Windows, macOS & Linux with fast, incremental backups, client-side end-to-end encryption, compression and data deduplication. CLI and GUI included.

Prometheus Alertmanager

A C library for parsing/normalizing street addresses around the world. Powered by statistical NLP and open geo data.

Dedupe/batch geocode addresses and venues around the world with libpostal

Accelerating the deduplication and collapsing process for reads with Unique Molecular Identifiers (UMI). Heavily optimized for scalability and orders of magnitude faster than a previous tool.

RocketMQ消息幂等去重消费者，支持使用MySQL或者Redis做幂等表，开箱即用

Resources for tackling record linkage / deduplication / data matching problems

Generate duplex/single consensus reads to reduce sequencing noises and remove duplications

PyTorch library for transforming entities like companies, products, etc. into vectors to support scalable Record Linkage / Entity Resolution using Approximate Nearest Neighbors.

Implementation of Fellegi-Sunter's canonical model of record linkage in Apache Spark, including EM algorithm to estimate parameters

A library for secure, deduplicated, transactional, and verifiable data storage

Tool for managing data-deduplication within extant compressed archive files, along with a relatively performant BK tree implementation for fuzzy image searching.

OpenStaticAnalyzer is a source code analyzer tool, which can perform deep static analysis of the source code of complex systems.

Advanced similarity and duplicate source code proof of concept for our research efforts.

Detecting near-duplicate videos by aggregating features from intermediate CNN layers

Welcome to Snowman App – a Data Matching Benchmark Platform.

Wanna get DRY? Static analysis tool for detecting repeat code.