All Projects → prashnts → pybloomfiltermmap3

prashnts / pybloomfiltermmap3

Licence: MIT license
Fast Python Bloom Filter using Mmap

Programming Languages

c
50402 projects - #5 most used programming language
python
139335 projects - #7 most used programming language
cython
566 projects
Makefile
30231 projects

Projects that are alternatives of or similar to pybloomfiltermmap3

Python Hashes
Interesting (non-cryptographic) hashes implemented in pure Python.
Stars: ✭ 213 (+144.83%)
Mutual labels:  bloom-filter
blex
Fast Bloom filter with concurrent accessibility, powered by :atomics module.
Stars: ✭ 34 (-60.92%)
Mutual labels:  bloom-filter
libfilter
High-speed Bloom filters and taffy filters for C, C++, and Java
Stars: ✭ 23 (-73.56%)
Mutual labels:  bloom-filter
Abyss
🔬 Assemble large genomes using short reads
Stars: ✭ 219 (+151.72%)
Mutual labels:  bloom-filter
ntEdit
✏️ultra fast and scalable genome assembly polishing
Stars: ✭ 56 (-35.63%)
Mutual labels:  bloom-filter
redisbloom-go
Go Client for RedisBloom probabilistic module
Stars: ✭ 74 (-14.94%)
Mutual labels:  bloom-filter
Sketchy
Sketching Algorithms for Clojure (bloom filter, min-hash, hyper-loglog, count-min sketch)
Stars: ✭ 138 (+58.62%)
Mutual labels:  bloom-filter
bloom filter
Bloom filter implementation in Crystal lang
Stars: ✭ 33 (-62.07%)
Mutual labels:  bloom-filter
bloom
Probabilistic set data structure
Stars: ✭ 77 (-11.49%)
Mutual labels:  bloom-filter
cuckoo filter
High-performance, concurrent, and mutable Cuckoo Filter for Erlang and Elixir
Stars: ✭ 31 (-64.37%)
Mutual labels:  bloom-filter
raptor
A fast and space-efficient pre-filter for querying very large collections of nucleotide sequences.
Stars: ✭ 37 (-57.47%)
Mutual labels:  bloom-filter
fever
fast, extensible, versatile event router for Suricata's EVE-JSON format
Stars: ✭ 47 (-45.98%)
Mutual labels:  bloom-filter
exor filter
Erlang nif for xor_filter. 'Faster and Smaller Than Bloom and Cuckoo Filters'.
Stars: ✭ 29 (-66.67%)
Mutual labels:  bloom-filter
Libbloom
A simple and small bloom filter implementation in plain C.
Stars: ✭ 215 (+147.13%)
Mutual labels:  bloom-filter
hackernews-button
Privacy-preserving Firefox extension linking to Hacker News discussion; built with Bloom filters and WebAssembly
Stars: ✭ 73 (-16.09%)
Mutual labels:  bloom-filter
Redis Cuckoofilter
Hashing-function agnostic Cuckoo filters for Redis
Stars: ✭ 158 (+81.61%)
Mutual labels:  bloom-filter
Doramon
个人工具汇总:一致性哈希工具,Bitmap工具,布隆过滤器参数生成器,Yaml和properties互转工具,一键式生成整个前后端工具,单机高性能幂等工具,zookeeper客户端工具,分布式全局id生成器,时间转换工具,Http封装工具
Stars: ✭ 53 (-39.08%)
Mutual labels:  bloom-filter
PharoPDS
Probabilistic data structures in Pharo Smalltalk.
Stars: ✭ 28 (-67.82%)
Mutual labels:  bloom-filter
bloomfilter
Bloom filters for Java
Stars: ✭ 53 (-39.08%)
Mutual labels:  bloom-filter
crlite
WebPKI-level Certificate Revocation via Multi-Level Bloom Filter Cascade
Stars: ✭ 52 (-40.23%)
Mutual labels:  bloom-filter

pybloomfiltermmap3

pybloomfiltermmap3 is a Python 3 compatible fork of pybloomfiltermmap by @axiak.

The goal of pybloomfiltermmap3 is simple: to provide a fast, simple, scalable, correct library for Bloom filters in Python.

Build Status Documentation Status PyPI PyPI PyPI

Why pybloomfiltermmap3?

There are a couple reasons to use this module:

  • It natively uses mmaped files.
  • It is fast (see benchmarks).
  • It natively does the set things you want a Bloom filter to do.

Quickstart

After you install, the interface to use is a cross between a file interface and an ste interface. As an example:

    >>> import pybloomfilter
    >>> fruit = pybloomfilter.BloomFilter(100000, 0.1, '/tmp/words.bloom')
    >>> fruit.update(('apple', 'pear', 'orange', 'apple'))
    >>> len(fruit)
    3
    >>> 'mike' in fruit
    False
    >>> 'apple' in fruit
    True

To create an in-memory filter, simply omit the file location:

    >>> cakes = pybloomfilter.BloomFilter(10000, 0.1)

Caveat: it is currently not possible to persist this filter later.

Docs

Current docs are available at pybloomfiltermmap3.rtfd.io.

Install

To install:

    $ pip install pybloomfiltermmap3

and you should be set.

Note to Python 2 to < 3.5 users

This library is specifically meant for Python 3.5 and above. As of 2020, we strongly advise you to switch to an actively maintained distribution of Python 3. If for any reason your current environment is restricted to Python 2, please see pybloomfiltermmap. Please note that the latter is not actively maintained and will lack bug fixes and new features.

History and Future

pybloomfiltermmap is an excellent Bloom filter implementation for Python 2 by @axiak and contributors. I, @prashnts, made initial changes to add support for Python 3 sometime in 2016 as the current pybloomfiltermmap3 on PyPI. Since then, with the help of contributors, there have been incremental improvements and bug fixes while maintaining the API from versions 0.4.x and below.

Some new features and changes were first introduced in version 0.5.0. From this point on, the goal is to reach stability, as well as add a few more APIs to expand upon the use cases. While we can't guarantee that we won't change the current interface, the transition from versions 0.4.x and below should be quick one liners. Please open an issue if we broke your build!

Suggestions, bug reports, and / or patches are welcome!

Contributions and development

When contributing, you should set up an appropriate Python 3 environment and install the dependencies listed in requirements-dev.txt. Package installation depends on a generated pybloomfilter.c file, which requires Cython module to be in your current environment.

Maintainers

License

See the LICENSE file. It's under the MIT License.

Note that the project description data, including the texts, logos, images, and/or trademarks, for each open source project belongs to its rightful owner. If you wish to add or remove any projects, please contact us at [email protected].