DejavuQuickly detect already witnessed data.
Stars: ✭ 151 (-93.67%)
VdoUserspace tools for managing VDO volumes.
Stars: ✭ 138 (-94.21%)
Spark LucenerddSpark RDD with Lucene's query and entity linkage capabilities
Stars: ✭ 114 (-95.22%)
FingerprintsMake it easier to compare and cross-reference the names of companies and people by applying strong normalisation.
Stars: ✭ 91 (-96.18%)
RltkRecord Linkage ToolKit (Find and link entities)
Stars: ✭ 71 (-97.02%)
RmlintExtremely fast tool to remove duplicates and other lint from your filesystem
Stars: ✭ 996 (-58.24%)
Fastcdc RsFastCDC implementation in Rust
Stars: ✭ 31 (-98.7%)
Dupandas📊 python package for performing deduplication using flexible text matching and cleaning in pandas dataframe
Stars: ✭ 20 (-99.16%)
BorgmaticSimple, configuration-driven backup software for servers and workstations
Stars: ✭ 902 (-62.18%)
JdupesA powerful duplicate file finder and an enhanced fork of 'fdupes'.
Stars: ✭ 790 (-66.88%)
RdedupData deduplication engine, supporting optional compression and public key encryption.
Stars: ✭ 690 (-71.07%)
TalismanStraightforward fuzzy matching, information retrieval and NLP building blocks for JavaScript.
Stars: ✭ 584 (-75.51%)
RecordlinkageA toolkit for record linkage and duplicate detection in Python
Stars: ✭ 532 (-77.69%)
KopiaCross-platform backup tool for Windows, macOS & Linux with fast, incremental backups, client-side end-to-end encryption, compression and data deduplication. CLI and GUI included.
Stars: ✭ 507 (-78.74%)
AlertmanagerPrometheus Alertmanager
Stars: ✭ 4,574 (+91.78%)
LibpostalA C library for parsing/normalizing street addresses around the world. Powered by statistical NLP and open geo data.
Stars: ✭ 3,312 (+38.87%)
lieuDedupe/batch geocode addresses and venues around the world with libpostal
Stars: ✭ 73 (-96.94%)
UMICollapseAccelerating the deduplication and collapsing process for reads with Unique Molecular Identifiers (UMI). Heavily optimized for scalability and orders of magnitude faster than a previous tool.
Stars: ✭ 31 (-98.7%)
record-linkage-resourcesResources for tackling record linkage / deduplication / data matching problems
Stars: ✭ 67 (-97.19%)
gencoreGenerate duplex/single consensus reads to reduce sequencing noises and remove duplications
Stars: ✭ 91 (-96.18%)
entity-embedPyTorch library for transforming entities like companies, products, etc. into vectors to support scalable Record Linkage / Entity Resolution using Approximate Nearest Neighbors.
Stars: ✭ 96 (-95.97%)
splinkImplementation of Fellegi-Sunter's canonical model of record linkage in Apache Spark, including EM algorithm to estimate parameters
Stars: ✭ 181 (-92.41%)
dduperFast block-level out-of-band BTRFS deduplication tool.
Stars: ✭ 108 (-95.47%)
acid-storeA library for secure, deduplicated, transactional, and verifiable data storage
Stars: ✭ 48 (-97.99%)
IntraArchiveDeduplicatorTool for managing data-deduplication within extant compressed archive files, along with a relatively performant BK tree implementation for fuzzy image searching.
Stars: ✭ 87 (-96.35%)
yadfYet Another Dupes Finder
Stars: ✭ 32 (-98.66%)
cargo-limitCargo with less noise: warnings are skipped until errors are fixed, Neovim integration, etc.
Stars: ✭ 105 (-95.6%)
zinggScalable identity resolution, entity resolution, data mastering and deduplication using ML
Stars: ✭ 655 (-72.54%)
zpaqfranzDeduplicating archiver with encryption and paranoid-level tests. Swiss army knife for the serious backup and disaster recovery manager. Ransomware neutralizer. Win/Linux/Unix
Stars: ✭ 86 (-96.39%)
mail-deduplicate📧 CLI to deduplicate mails from mail boxes.
Stars: ✭ 134 (-94.38%)
Neural-Scam-ArtistWeb Scraping, Document Deduplication & GPT-2 Fine-tuning with a newly created scam dataset.
Stars: ✭ 18 (-99.25%)
deduplicationFast multi-threaded content-dependent chunking deduplication for Buffers in C++ with a reference implementation in Javascript. Ships with extensive tests, a fuzz test and a benchmark.
Stars: ✭ 59 (-97.53%)
FrostA backup program that does deduplication, compression, encryption
Stars: ✭ 25 (-98.95%)
dedupsqlfsDeduplicating filesystem via Python3, FUSE and SQLite
Stars: ✭ 24 (-98.99%)
nomenklaturaFramework and command-line tools for integrating FollowTheMoney data streams from multiple sources
Stars: ✭ 158 (-93.38%)
LshLocality Sensitive Hashing using MinHash in Python/Cython to detect near duplicate text documents
Stars: ✭ 182 (-92.37%)
ResticFast, secure, efficient backup program
Stars: ✭ 15,105 (+533.33%)
KvdoA pair of kernel modules which provide pools of deduplicated and/or compressed block storage.
Stars: ✭ 168 (-92.96%)