62 Open source projects that are alternatives of or similar to IntraArchiveDeduplicator

Spark RDD with Lucene's query and entity linkage capabilities

Data deduplication engine, supporting optional compression and public key encryption.

Pytorch ReID: A tiny, friendly, strong pytorch implement of object re-identification baseline. Tutorial 👉https://github.com/layumi/Person_reID_baseline_pytorch/tree/master/tutorial

A pair of kernel modules which provide pools of deduplicated and/or compressed block storage.

RocketMQ消息幂等去重消费者，支持使用MySQL或者Redis做幂等表，开箱即用

⚡ A fast embedded library for approximate nearest neighbor search

FastCDC implementation in Rust

Code to download web-images

Prometheus Alertmanager

Anime Scene Search by Image

Locality Sensitive Hashing using MinHash in Python/Cython to detect near duplicate text documents

Implementation of Fellegi-Sunter's canonical model of record linkage in Apache Spark, including EM algorithm to estimate parameters

Deduplicating filesystem via Python3, FUSE and SQLite

Quickly detect already witnessed data.

No description or website provided.

Record Linkage ToolKit (Find and link entities)

An utility Discord bot written in C# using Discord.Net

Simple, configuration-driven backup software for servers and workstations

Yet Another Dupes Finder

A toolkit for record linkage and duplicate detection in Python

Python Script to download hundreds of images from 'Google Images'. It is a ready-to-run code!

Dedupe/batch geocode addresses and venues around the world with libpostal

Web Scraping, Document Deduplication & GPT-2 Fine-tuning with a newly created scam dataset.

Generate duplex/single consensus reads to reduce sequencing noises and remove duplications

Python Image Search Engine with OpenCV

A list of free data matching and record linkage software.

Fast block-level out-of-band BTRFS deduplication tool.

A backup program that does deduplication, compression, encryption

Fast, secure, efficient backup program

Deduplicating archiver with encryption and paranoid-level tests. Swiss army knife for the serious backup and disaster recovery manager. Ransomware neutralizer. Win/Linux/Unix

Find duplicate files

Reverse image search tool (SauceNao, ImgOps, trace.moe, and more)

Userspace tools for managing VDO volumes.

🧿 Pupyl is a really fast image search library which you can index your own (millions of) images and find similar images in milliseconds.

Make it easier to compare and cross-reference the names of companies and people by applying strong normalisation.

Framework and command-line tools for integrating FollowTheMoney data streams from multiple sources

Extremely fast tool to remove duplicates and other lint from your filesystem

微信小程序: 识图娘

📊 python package for performing deduplication using flexible text matching and cleaning in pandas dataframe

Cloud-native neural search framework for 𝙖𝙣𝙮 kind of data

A powerful duplicate file finder and an enhanced fork of 'fdupes'.

A site that allows you to reverse image search millions of furry images in under a second

Straightforward fuzzy matching, information retrieval and NLP building blocks for JavaScript.

SOTA Re-identification Methods and Toolbox

Cross-platform backup tool for Windows, macOS & Linux with fast, incremental backups, client-side end-to-end encryption, compression and data deduplication. CLI and GUI included.

📧 CLI to deduplicate mails from mail boxes.

A C library for parsing/normalizing street addresses around the world. Powered by statistical NLP and open geo data.

An open-source vector database for embedding similarity search and AI applications.

Accelerating the deduplication and collapsing process for reads with Unique Molecular Identifiers (UMI). Heavily optimized for scalability and orders of magnitude faster than a previous tool.

Cargo with less noise: warnings are skipped until errors are fixed, Neovim integration, etc.

Resources for tackling record linkage / deduplication / data matching problems

Anime Scene Search by Image

PyTorch library for transforming entities like companies, products, etc. into vectors to support scalable Record Linkage / Entity Resolution using Approximate Nearest Neighbors.

Fast multi-threaded content-dependent chunking deduplication for Buffers in C++ with a reference implementation in Javascript. Ships with extensive tests, a fuzz test and a benchmark.

人工智障本地图片检索工具 | An EfficientNet based image retrieval tool

Search IQDB from CLI

Mobile(iOS) Text-to-Image search powered by multimodal semantic representation models(e.g., OpenAI's CLIP). Accepted at NAACL 2022.

Scalable identity resolution, entity resolution, data mastering and deduplication using ML

Search photos on Unsplash based on OpenAI's CLIP model, support search with joint image+text queries and attention visualization.

🔎 A simple yet powerful module to retrieve organic search results and much more from Google.