All Projects → Navid2zp → dups

Navid2zp / dups

Licence: Apache-2.0 License
A CLI tool to find/remove duplicate files supporting multi-core and different algorithms (MD5, SHA256, and XXHash).

Programming Languages

go
31211 projects - #10 most used programming language

Projects that are alternatives of or similar to dups

hash-wasm
Lightning fast hash functions using hand-tuned WebAssembly binaries
Stars: ✭ 382 (+1719.05%)
Mutual labels:  md5, xxhash, sha256
Gtkhash
A cross-platform desktop utility for computing message digests or checksums
Stars: ✭ 167 (+695.24%)
Mutual labels:  md5, sha256
Merkle
Node.js module implementing Merkle tree algorithm
Stars: ✭ 123 (+485.71%)
Mutual labels:  md5, sha256
Checksum
Checksum calculation extensions for Swift
Stars: ✭ 28 (+33.33%)
Mutual labels:  md5, sha256
Blooddy crypto
ActionScript (AS3) library for processing binary data. This library contains MD5, SHA-1, SHA-2 ( SHA-224 и SHA-256 ), Base64, CRC32 algorithms, JSON encoder & decoder as well as PNG and JPEG encoders.
Stars: ✭ 83 (+295.24%)
Mutual labels:  md5, sha256
Hashcobra
HashCobra Hash Cracking tool.
Stars: ✭ 96 (+357.14%)
Mutual labels:  md5, sha256
Wjcryptlib
Public Domain C Library of Cryptographic functions. Including: MD5, SHA1, SHA256, SHA512, RC4, AES, AES-CTR, AES-OFB, AES-CBC
Stars: ✭ 250 (+1090.48%)
Mutual labels:  md5, sha256
Hash Buster
Crack hashes in seconds.
Stars: ✭ 981 (+4571.43%)
Mutual labels:  md5, sha256
BruteForce
A simple brute forcer written in GO for SHA1, SHA256, SHA512, MD5 and bcrypt
Stars: ✭ 49 (+133.33%)
Mutual labels:  md5, sha256
fhash
fHash - an open source files hash calculator for Windows and macOS
Stars: ✭ 222 (+957.14%)
Mutual labels:  md5, sha256
hash-checker
Fast and simple application that allows you to generate and compare hashes from files and text
Stars: ✭ 72 (+242.86%)
Mutual labels:  md5, sha256
Pure lua sha
SHA1, SHA2 and SHA3 functions written in pure Lua and optimized for speed
Stars: ✭ 78 (+271.43%)
Mutual labels:  md5, sha256
Digestif
Simple hash algorithms in OCaml
Stars: ✭ 69 (+228.57%)
Mutual labels:  md5, sha256
Hash Library
Portable C++ hashing library
Stars: ✭ 109 (+419.05%)
Mutual labels:  md5, sha256
Crypto Es
A cryptography algorithms library
Stars: ✭ 65 (+209.52%)
Mutual labels:  md5, sha256
Rufus
The Reliable USB Formatting Utility
Stars: ✭ 16,917 (+80457.14%)
Mutual labels:  md5, sha256
Openhashtab
📝 File hashing and checking shell extension
Stars: ✭ 599 (+2752.38%)
Mutual labels:  md5, sha256
Gensum
Powerful checksum generator!
Stars: ✭ 12 (-42.86%)
Mutual labels:  md5, sha256
crypto.js
base on crypto module
Stars: ✭ 13 (-38.1%)
Mutual labels:  md5, sha256
Hashrat
Hashing tool supporting md5,sha1,sha256,sha512,whirlpool,jh and hmac versions of these. Includes recursive file hashing and other features.
Stars: ✭ 46 (+119.05%)
Mutual labels:  md5, sha256

dups

dups

dups is a CLI tool to find and remove duplicate files using different hash algorithms (MD5, SHA256 and XXHash) with multi-core support.

Install

Download binaries:

Release Page

To use in a go project:

go get github.com/Navid2zp/dups

Usage

CLI

Available Commands:

Command Description
clean Finds duplicate files in a given path and deletes them.
scan Finds duplicate files in a given path but doesn't delete them.
help Help about any command

Flags:

flag Description
--algorithm string algorithm to use (md5/sha256/xxhash) (default "md5")
-f, --flat flat output, no extra info (only prints duplicate files)
-r, --full full search (search in sub-directories too)
--min-size int minimum file size to scan in bytes (default 10)
-s, --single-core use single cpu core

Examples:

Remove duplicates bigger than 1KB using multiple cpu cores:

dups clean path/to/directory --min-size 1024

Find duplicates and write them into file.txt:

dups scan path/to/directory -f >> file.txt

Find and list duplicates using single cpu core and XXHash algorithm:

dups scan path/to/directory -s --algorithm xxhash

Go code:

package main

import (
	"fmt"
	"github.com/Navid2zp/dups"
)

func main()  {
	// list all files including files in any sub-directory
	files, err := dups.GetFiles("path/to/directory", true)
	if err != nil {
		panic(err)
	}

        // group files based on their file size
        fileGroups, totalFiles := dups.GroupFiles(files, 128)

	// collect hashes for groups with more than one file
	// singleThread: use a single thread
	// flatt: don't print the process bar or any other information
	hashes := dups.CollectHashes(fileGroups, false, dups.XXHash, false, totalFiles)
	duplicates, filesCount, duplicatesCount := dups.GetDuplicates(hashes)
	fmt.Println("total number of files with duplicates:", filesCount)
	fmt.Println("total number of duplicate files:", duplicatesCount)

	freedSize, deletedCount, err := dups.RemoveDuplicates(duplicates)
	if err != nil {
		panic(err)
	}
	fmt.Println("remove", deletedCount, "files")
	fmt.Println("freed a total of ", freedSize, "bytes")
}

Notes:

  • Use single core option (-s) if files are big (depending on your disk type).
  • Use XXHash algorithm for fast scanning and MD5/SHA256 for safest scanning or if the number of files is huge.

Build from source:

go build -tags multicore if you are building using Go < 1.5 or edit runtime.GOMAXPROCS() manually to support multi-core.

License

Apache

Note that the project description data, including the texts, logos, images, and/or trademarks, for each open source project belongs to its rightful owner. If you wish to add or remove any projects, please contact us at [email protected].