ResticFast, secure, efficient backup program
Stars: ✭ 15,105 (+8199.45%)
KvdoA pair of kernel modules which provide pools of deduplicated and/or compressed block storage.
Stars: ✭ 168 (-7.69%)
DupeguruFind duplicate files
Stars: ✭ 2,385 (+1210.44%)
DejavuQuickly detect already witnessed data.
Stars: ✭ 151 (-17.03%)
VdoUserspace tools for managing VDO volumes.
Stars: ✭ 138 (-24.18%)
Spark LucenerddSpark RDD with Lucene's query and entity linkage capabilities
Stars: ✭ 114 (-37.36%)
FingerprintsMake it easier to compare and cross-reference the names of companies and people by applying strong normalisation.
Stars: ✭ 91 (-50%)
RltkRecord Linkage ToolKit (Find and link entities)
Stars: ✭ 71 (-60.99%)
RmlintExtremely fast tool to remove duplicates and other lint from your filesystem
Stars: ✭ 996 (+447.25%)
Fastcdc RsFastCDC implementation in Rust
Stars: ✭ 31 (-82.97%)
Dupandas📊 python package for performing deduplication using flexible text matching and cleaning in pandas dataframe
Stars: ✭ 20 (-89.01%)
BorgmaticSimple, configuration-driven backup software for servers and workstations
Stars: ✭ 902 (+395.6%)
JdupesA powerful duplicate file finder and an enhanced fork of 'fdupes'.
Stars: ✭ 790 (+334.07%)
RdedupData deduplication engine, supporting optional compression and public key encryption.
Stars: ✭ 690 (+279.12%)
TalismanStraightforward fuzzy matching, information retrieval and NLP building blocks for JavaScript.
Stars: ✭ 584 (+220.88%)
RecordlinkageA toolkit for record linkage and duplicate detection in Python
Stars: ✭ 532 (+192.31%)
KopiaCross-platform backup tool for Windows, macOS & Linux with fast, incremental backups, client-side end-to-end encryption, compression and data deduplication. CLI and GUI included.
Stars: ✭ 507 (+178.57%)
AlertmanagerPrometheus Alertmanager
Stars: ✭ 4,574 (+2413.19%)
LibpostalA C library for parsing/normalizing street addresses around the world. Powered by statistical NLP and open geo data.
Stars: ✭ 3,312 (+1719.78%)
lieuDedupe/batch geocode addresses and venues around the world with libpostal
Stars: ✭ 73 (-59.89%)
UMICollapseAccelerating the deduplication and collapsing process for reads with Unique Molecular Identifiers (UMI). Heavily optimized for scalability and orders of magnitude faster than a previous tool.
Stars: ✭ 31 (-82.97%)
record-linkage-resourcesResources for tackling record linkage / deduplication / data matching problems
Stars: ✭ 67 (-63.19%)
gencoreGenerate duplex/single consensus reads to reduce sequencing noises and remove duplications
Stars: ✭ 91 (-50%)
entity-embedPyTorch library for transforming entities like companies, products, etc. into vectors to support scalable Record Linkage / Entity Resolution using Approximate Nearest Neighbors.
Stars: ✭ 96 (-47.25%)
splinkImplementation of Fellegi-Sunter's canonical model of record linkage in Apache Spark, including EM algorithm to estimate parameters
Stars: ✭ 181 (-0.55%)
dduperFast block-level out-of-band BTRFS deduplication tool.
Stars: ✭ 108 (-40.66%)
acid-storeA library for secure, deduplicated, transactional, and verifiable data storage
Stars: ✭ 48 (-73.63%)
IntraArchiveDeduplicatorTool for managing data-deduplication within extant compressed archive files, along with a relatively performant BK tree implementation for fuzzy image searching.
Stars: ✭ 87 (-52.2%)
yadfYet Another Dupes Finder
Stars: ✭ 32 (-82.42%)
cargo-limitCargo with less noise: warnings are skipped until errors are fixed, Neovim integration, etc.
Stars: ✭ 105 (-42.31%)
zinggScalable identity resolution, entity resolution, data mastering and deduplication using ML
Stars: ✭ 655 (+259.89%)
zpaqfranzDeduplicating archiver with encryption and paranoid-level tests. Swiss army knife for the serious backup and disaster recovery manager. Ransomware neutralizer. Win/Linux/Unix
Stars: ✭ 86 (-52.75%)
mail-deduplicate📧 CLI to deduplicate mails from mail boxes.
Stars: ✭ 134 (-26.37%)
Neural-Scam-ArtistWeb Scraping, Document Deduplication & GPT-2 Fine-tuning with a newly created scam dataset.
Stars: ✭ 18 (-90.11%)
deduplicationFast multi-threaded content-dependent chunking deduplication for Buffers in C++ with a reference implementation in Javascript. Ships with extensive tests, a fuzz test and a benchmark.
Stars: ✭ 59 (-67.58%)
FrostA backup program that does deduplication, compression, encryption
Stars: ✭ 25 (-86.26%)
dedupsqlfsDeduplicating filesystem via Python3, FUSE and SQLite
Stars: ✭ 24 (-86.81%)
nomenklaturaFramework and command-line tools for integrating FollowTheMoney data streams from multiple sources
Stars: ✭ 158 (-13.19%)