All Projects → F483 → Dejavu

F483 / Dejavu

Licence: mit
Quickly detect already witnessed data.

Programming Languages

go
31211 projects - #10 most used programming language
golang
3204 projects

Projects that are alternatives of or similar to Dejavu

Desktoppr
Simple command line tool to set the desktop picture on macOS
Stars: ✭ 127 (-15.89%)
Mutual labels:  command-line-tool, command-line
Xioc
Extract indicators of compromise from text, including "escaped" ones.
Stars: ✭ 148 (-1.99%)
Mutual labels:  command-line-tool, command-line
Check It Out
A command line interface for Git Checkout. See branches available for checkout.
Stars: ✭ 127 (-15.89%)
Mutual labels:  command-line-tool, command-line
Hmap
hmap is a command line tool written in Swift to work with Clang header maps produced by Xcode.
Stars: ✭ 110 (-27.15%)
Mutual labels:  command-line-tool, command-line
Onhold
🔊 Play sounds while and after shell jobs complete
Stars: ✭ 146 (-3.31%)
Mutual labels:  command-line-tool, command-line
Dynein
DynamoDB CLI written in Rust.
Stars: ✭ 126 (-16.56%)
Mutual labels:  command-line-tool, command-line
Git Tidy
Tidy up stale git branches.
Stars: ✭ 137 (-9.27%)
Mutual labels:  command-line-tool, command-line
Tooling
Advancing Node.js as a framework for writing great tools
Stars: ✭ 98 (-35.1%)
Mutual labels:  command-line-tool, command-line
Gossed
Push the standard output of ANY program to browsers as Server Sent Events
Stars: ✭ 138 (-8.61%)
Mutual labels:  command-line-tool, command-line
Fblog
Small command-line JSON Log viewer
Stars: ✭ 137 (-9.27%)
Mutual labels:  command-line-tool, command-line
Awesome Cli
A curated list of awesome resources for building immersive CLI experiences.
Stars: ✭ 104 (-31.13%)
Mutual labels:  command-line-tool, command-line
Artisan Menu
📝 Artisan Menu - Use Artisan via an elegant console GUI
Stars: ✭ 141 (-6.62%)
Mutual labels:  command-line-tool, command-line
Csv2db
The CSV to database command line loader
Stars: ✭ 102 (-32.45%)
Mutual labels:  command-line-tool, command-line
Typin
Declarative framework for interactive CLI applications
Stars: ✭ 126 (-16.56%)
Mutual labels:  command-line-tool, command-line
Ask Cli
Alexa Skills Kit Command Line Interface
Stars: ✭ 100 (-33.77%)
Mutual labels:  command-line-tool, command-line
Asciigraph
Go package to make lightweight ASCII line graph ╭┈╯ in command line apps with no other dependencies.
Stars: ✭ 1,805 (+1095.36%)
Mutual labels:  command-line, command-line-tool
Forge Node App
🛠📦🎉 Generate Node.js boilerplate with optional libraries & tools
Stars: ✭ 90 (-40.4%)
Mutual labels:  command-line-tool, command-line
Terminal layout
The project help you to quickly build layouts in terminal,cross-platform(一个跨平台的命令行ui布局工具)
Stars: ✭ 98 (-35.1%)
Mutual labels:  command-line-tool, command-line
Brotab
Control your browser's tabs from the command line
Stars: ✭ 137 (-9.27%)
Mutual labels:  command-line-tool, command-line
Pueue
🌠 Manage your shell commands.
Stars: ✭ 2,471 (+1536.42%)
Mutual labels:  command-line-tool, command-line

Build Status Issues Go Report Card License GoDoc Donate PayPal Donate Bitcoin Avaivable For Hire

Déjà vu

Quickly detect already witnessed data, ideal for deduplication.

Limited memory of witnessed data, oldest are forgotten. Library is thread safe. Offers deterministic and probabilistic (over an order of magnitude less memory consuming) implementation. The probabilistic implementation uses bloom filters, meaning false positives are possible but not false negatives.

Installation

Download binary release

Compiled binaries for many platforms are available and can be downloaded for the latest release.

Extract the binary for your platform and add it to your system path.

Compile from source

Requires golang environment/workspace.

# compile and install library
go get github.com/f483/dejavu

# compile and install binary
go install github.com/f483/dejavu/dejavu

Command line usage

$ dejavu -h
Usage: dejavu [OPTION]... [FILE]...

Concatenate FILE(s) and filter or output duplicate lines.

With no FILE, or when FILE is -, read standard input.

Options:
  -D	use deterministic mode instead of probabilistic
	WARNING requires order of magnitude more memory
  -d	output only duplicates instead of filtering
  -f float
    	chance of false positive, between 0.0 and 1.0
	only for probabilistic mode (default 1e-06)
  -l uint
    	limit after which entries are forgotton (default 1000000)
  -o string
    	output file, defaults to stdout
  -v	output version information and exit

Examples:
  dejavu
	default probabilistic deduplication from stdin to std out with
	1mil entry limit and 1/1mil chance of false positive (~8M mem usage)
  dejavu -o s f - g
	deduplicat f, then stdin, then g, to output s
  dejavu -l 10000000 -fp 0.000000001
	probabilistic deduplication with 10mil entry limit
	and 1/1bil chance of false positive (~70M mem usage)
  dejavu -d -D -l 65536
	output duplicates and avoid false positives with deterministic mode
	lower entry limit to avoid excessive memory usage

Implementation:
  Efficient probabilistic and deterministic duplicate detection with O(1) 
  detection time and O(n) memory usage in relation to entry limit. Default
  probabilistic implementation uses bloom filters, meaning false
  positives are possible but not false negatives.

Author: Fabian Barkhau <[email protected]>
Project: https://github.com/f483/dejavu
License: MIT https://raw.githubusercontent.com/f483/dejavu/master/LICENSE

Library usage (golang)

Probabilistic example

package main

import (
	"fmt"
	"github.com/f483/dejavu"
)

func main() {

	// probably remembers last 65536 with 0.000001 chance of false positive
	p := dejavu.NewProbabilistic(65536, 0.000001)

	fmt.Println(p.Witness([]byte("bar"))) // entry added
	fmt.Println(p.Witness([]byte("bar"))) // probably remembers entry
}

Deterministic example

package main

import (
	"fmt"
	"github.com/f483/dejavu"
)

func main() {

	// always remembers last 1024 entries
	d := dejavu.NewDeterministic(1024)

	fmt.Println(d.Witness([]byte("foo"))) // entry added
	fmt.Println(d.Witness([]byte("foo"))) // remembers entry
}

Performance

Linear memory usage: O(n)

Probabilistic

0.000001 chance of false positive.

Benchmark Memory

Deterministic

Benchmark Memory

Constant witness time: O(1)

Probabilistic

0.000001 chance of false positive.

Benchmark Time

Deterministic

Benchmark Time

Note that the project description data, including the texts, logos, images, and/or trademarks, for each open source project belongs to its rightful owner. If you wish to add or remove any projects, please contact us at [email protected].