entity-embedPyTorch library for transforming entities like companies, products, etc. into vectors to support scalable Record Linkage / Entity Resolution using Approximate Nearest Neighbors.
Stars: ✭ 96 (+43.28%)
splinkImplementation of Fellegi-Sunter's canonical model of record linkage in Apache Spark, including EM algorithm to estimate parameters
Stars: ✭ 181 (+170.15%)
stanceLearned string similarity for entity names using optimal transport.
Stars: ✭ 27 (-59.7%)
Merge-MachineMerge Dirty Data with Clean Reference Tables
Stars: ✭ 35 (-47.76%)
snowmanWelcome to Snowman App – a Data Matching Benchmark Platform.
Stars: ✭ 25 (-62.69%)
zinggScalable identity resolution, entity resolution, data mastering and deduplication using ML
Stars: ✭ 655 (+877.61%)
LibpostalA C library for parsing/normalizing street addresses around the world. Powered by statistical NLP and open geo data.
Stars: ✭ 3,312 (+4843.28%)
Dedupe🆔 A python library for accurate and scalable fuzzy matching, record deduplication and entity-resolution.
Stars: ✭ 3,241 (+4737.31%)
gencoreGenerate duplex/single consensus reads to reduce sequencing noises and remove duplications
Stars: ✭ 91 (+35.82%)
dduperFast block-level out-of-band BTRFS deduplication tool.
Stars: ✭ 108 (+61.19%)
conciliatorOpenRefine reconciliation services for VIAF, ORCID, and Open Library + framework for creating more.
Stars: ✭ 95 (+41.79%)
acid-storeA library for secure, deduplicated, transactional, and verifiable data storage
Stars: ✭ 48 (-28.36%)
whatisWhatIs.this: simple entity resolution through Wikipedia
Stars: ✭ 18 (-73.13%)
IntraArchiveDeduplicatorTool for managing data-deduplication within extant compressed archive files, along with a relatively performant BK tree implementation for fuzzy image searching.
Stars: ✭ 87 (+29.85%)
yadfYet Another Dupes Finder
Stars: ✭ 32 (-52.24%)
cargo-limitCargo with less noise: warnings are skipped until errors are fixed, Neovim integration, etc.
Stars: ✭ 105 (+56.72%)
zpaqfranzDeduplicating archiver with encryption and paranoid-level tests. Swiss army knife for the serious backup and disaster recovery manager. Ransomware neutralizer. Win/Linux/Unix
Stars: ✭ 86 (+28.36%)
mail-deduplicate📧 CLI to deduplicate mails from mail boxes.
Stars: ✭ 134 (+100%)
Neural-Scam-ArtistWeb Scraping, Document Deduplication & GPT-2 Fine-tuning with a newly created scam dataset.
Stars: ✭ 18 (-73.13%)
deduplicationFast multi-threaded content-dependent chunking deduplication for Buffers in C++ with a reference implementation in Javascript. Ships with extensive tests, a fuzz test and a benchmark.
Stars: ✭ 59 (-11.94%)
FrostA backup program that does deduplication, compression, encryption
Stars: ✭ 25 (-62.69%)
dedupsqlfsDeduplicating filesystem via Python3, FUSE and SQLite
Stars: ✭ 24 (-64.18%)
nomenklaturaFramework and command-line tools for integrating FollowTheMoney data streams from multiple sources
Stars: ✭ 158 (+135.82%)
LshLocality Sensitive Hashing using MinHash in Python/Cython to detect near duplicate text documents
Stars: ✭ 182 (+171.64%)
ResticFast, secure, efficient backup program
Stars: ✭ 15,105 (+22444.78%)
KvdoA pair of kernel modules which provide pools of deduplicated and/or compressed block storage.
Stars: ✭ 168 (+150.75%)
DupeguruFind duplicate files
Stars: ✭ 2,385 (+3459.7%)
DejavuQuickly detect already witnessed data.
Stars: ✭ 151 (+125.37%)
VdoUserspace tools for managing VDO volumes.
Stars: ✭ 138 (+105.97%)
Spark LucenerddSpark RDD with Lucene's query and entity linkage capabilities
Stars: ✭ 114 (+70.15%)
FingerprintsMake it easier to compare and cross-reference the names of companies and people by applying strong normalisation.
Stars: ✭ 91 (+35.82%)
RltkRecord Linkage ToolKit (Find and link entities)
Stars: ✭ 71 (+5.97%)
RmlintExtremely fast tool to remove duplicates and other lint from your filesystem
Stars: ✭ 996 (+1386.57%)
Fastcdc RsFastCDC implementation in Rust
Stars: ✭ 31 (-53.73%)
Dupandas📊 python package for performing deduplication using flexible text matching and cleaning in pandas dataframe
Stars: ✭ 20 (-70.15%)
BorgmaticSimple, configuration-driven backup software for servers and workstations
Stars: ✭ 902 (+1246.27%)
JdupesA powerful duplicate file finder and an enhanced fork of 'fdupes'.
Stars: ✭ 790 (+1079.1%)
RdedupData deduplication engine, supporting optional compression and public key encryption.
Stars: ✭ 690 (+929.85%)
TalismanStraightforward fuzzy matching, information retrieval and NLP building blocks for JavaScript.
Stars: ✭ 584 (+771.64%)
RecordlinkageA toolkit for record linkage and duplicate detection in Python
Stars: ✭ 532 (+694.03%)
KopiaCross-platform backup tool for Windows, macOS & Linux with fast, incremental backups, client-side end-to-end encryption, compression and data deduplication. CLI and GUI included.
Stars: ✭ 507 (+656.72%)
AlertmanagerPrometheus Alertmanager
Stars: ✭ 4,574 (+6726.87%)
lieuDedupe/batch geocode addresses and venues around the world with libpostal
Stars: ✭ 73 (+8.96%)
UMICollapseAccelerating the deduplication and collapsing process for reads with Unique Molecular Identifiers (UMI). Heavily optimized for scalability and orders of magnitude faster than a previous tool.
Stars: ✭ 31 (-53.73%)