absolute8511 / Zanredisdb
Licence: mit
Yet another distributed kvstore support redis data and index. moved to: https://github.com/youzan/ZanRedisDB
Stars: ✭ 64
Programming Languages
go
31211 projects - #10 most used programming language
Projects that are alternatives of or similar to Zanredisdb
Elasticell
Elastic Key-Value Storage With Strong Consistency and Reliability
Stars: ✭ 453 (+607.81%)
Mutual labels: redis, key-value, distributed-database, raft
Tikv
Distributed transactional key-value database, originally created to complement TiDB
Stars: ✭ 10,403 (+16154.69%)
Mutual labels: key-value, rocksdb, raft
Etcd
Distributed reliable key-value store for the most critical data of a distributed system
Stars: ✭ 38,238 (+59646.88%)
Mutual labels: key-value, distributed-database, raft
raft-rocks
A simple database based on raft and rocksdb
Stars: ✭ 38 (-40.62%)
Mutual labels: rocksdb, key-value, raft
Tidis
Distributed transactional NoSQL database, Redis protocol compatible using tikv as backend
Stars: ✭ 1,182 (+1746.88%)
Mutual labels: redis, rocksdb, raft
Ardb
A redis protocol compatible nosql, it support multiple storage engines as backend like Google's LevelDB, Facebook's RocksDB, OpenLDAP's LMDB, PerconaFT, WiredTiger, ForestDB.
Stars: ✭ 1,707 (+2567.19%)
Mutual labels: redis, key-value, rocksdb
Dynomite
A generic dynamo implementation for different k-v storage engines
Stars: ✭ 3,830 (+5884.38%)
Mutual labels: redis, key-value, distributed-database
Citus
Distributed PostgreSQL as an extension
Stars: ✭ 5,580 (+8618.75%)
Mutual labels: sharding, distributed-database
Finn
Fast Raft framework using the Redis protocol for Go
Stars: ✭ 534 (+734.38%)
Mutual labels: redis, raft
Kvrocks
the kvrocks(kv-rocks) is SSD NoSQL which based on rocksdb, and compatible with the Redis protocol, intention to decrease the cost of memory and increase the capability
Stars: ✭ 471 (+635.94%)
Mutual labels: redis, rocksdb
Pickledb
pickleDB is an open source key-value store using Python's json module.
Stars: ✭ 549 (+757.81%)
Mutual labels: json, key-value
Middleware development learning
中间件、高性能服务器、分布式存储等(redis、memcache、nginx、大容量redis pika、rocksdb、mongodb、wiredtiger存储引擎、高性能代理中间件)二次开发、性能优化,逐步整理文档说明并配合demo指导--每周末定时更新2-3篇技术文章及程序demo--(技术交流QQ群:568892619)
Stars: ✭ 449 (+601.56%)
Mutual labels: redis, rocksdb
Hraftd
A reference use of Hashicorp's Raft implementation
Stars: ✭ 732 (+1043.75%)
Mutual labels: key-value, raft
Dble
A High Scalability Middle-ware for MySQL Sharding
Stars: ✭ 774 (+1109.38%)
Mutual labels: sharding, distributed-database
Fastonosql
FastoNoSQL is a crossplatform Redis, Memcached, SSDB, LevelDB, RocksDB, UnQLite, LMDB, ForestDB, Pika, Dynomite, KeyDB GUI management tool.
Stars: ✭ 1,001 (+1464.06%)
Mutual labels: redis, rocksdb
Apicache
Simple API-caching middleware for Express/Node.
Stars: ✭ 957 (+1395.31%)
Mutual labels: json, redis
ZanRedisDB
What is ZanRedisDB
ZanRedisDB is a distributed redis cluster with strong consistency.
- Compatible with redis protocol: most redis command is supported in ZanRedisDB, so you can replace Redis with ZanRedisDB and get a more powerful cluster with unlimited data.
- Namespace and set supported: You can use the namespace and set as prefix to isolate different data to make operation more easily.
- Strong consistence and persistence: Use raft and rocksdb to make sure all the data have the consistence replication and persistent on stable storage.
- Scalability: You can easily increase the cluster capacity just by adding more machines.
- Multi-DC support: multi data centers deployment can be supported with rackaware feature.
Build
Install the compress library
yum install snappy-devel (for CentOS) apt-get install libsnappy1 libsnappy-dev (for Debian/Ubuntu) brew install snappy (for Mac)
Build the rocksdb
git clone https://github.com/absolute8511/rocksdb.git cd rocksdb USE_SSE=1 make static_lib
Install the dependency:
CGO_CFLAGS="-I/path/to/rocksdb/include" CGO_LDFLAGS="-L/path/to/rocksdb -lrocksdb -lstdc++ -lm -lsnappy -lrt" go get github.com/absolute8511/gorocksdb CGO_CFLAGS="-I/path/to/rocksdb/include" CGO_LDFLAGS="-L/path/to/rocksdb -lrocksdb -lstdc++ -lm -lsnappy" go get github.com/absolute8511/gorocksdb (for MacOS)
use the gpm
to install other dependencies
wget https://raw.githubusercontent.com/pote/gpm/v1.4.0/bin/gpm && chmod +x gpm && sudo mv gpm /usr/local/bin gpm get
Build zankv and placedriver from the source (only support go version 1.8+, gcc 4.9+ or xcode-command-line-tools on Mac):
make
If you want package the binary release run the scripts
./pre-dist.sh ./dist.sh
Deploy
- Deploy the rsync daemon which is needed on all server node to transfer the snapshot data for raft
- Deploy etcd cluster which is needed for the meta data for the namespaces
- Deploy the placedriver which is used for data placement:
placedriver -config=/path/to/config
- Deploy the zankv for data storage server
zankv -config=/path/to/config
API
placedriver has several HTTP APIs to manager the namespace
- list the namespace:
GET /namespaces
- list the data nodes:
GET /datanodes
- list the placedriver nodes:
GET /listpd
- query the namespace meta info:
GET /query/namespace_name
- create the namespace (handle only by leader) :
POST /cluster/namespace/create?namespace=test_p16&partition_num=16&replicator=3
- delete the namespace (handle only by leader):
POST /cluster/namespace/delete?namespace=test_p16&partition=**
storage server HTTP APIs for stats:
- namespace stats :
GET /stats
- namespace raft internal stats :
GET /raft/stats
- optimize the data storage :
POST /kv/optimize
- get the raft leader of the namespace partition:
GET /cluster/leader/namespace-partition
storage server also support the redis apis for read/write :
- KV:
- Hash Set:
- List:
- Sorted Set:
- ZSet:
Client
Golang client SDK : client-sdk , a redis proxy can be deployed based on this golang sdk if you want use the redis client in other language.
Architechture
Roadmap
- Redis data structures
- [x] KV
- [x] Hash
- [x] List
- [x] Set
- [x] Sorted Set
- [x] GeoHash
- [x] Expires
- [x] HyperLogLog
- [x] JSON
- Distributed system
- [x] Raft based replication
- [x] Partitions
- [x] Auto balance and migrate
- [x] Support namespace
- [x] High available
- [x] Distributed scan on table
- [x] Rackaware deployment used for across data center deployment
- Searchable and Indexing
- [ ] Secondary index support on Hash fields
- [ ] Secondary index support for json kv
- [ ] Full text search support
- Operation
- [x] Backup and restore for cluster
- [ ] More stats for read/write performance and errors.
- Client
- [x] High available for redis commands (Retry on fail)
- [ ] Extand redis commands to support index and search
- [x] Extand redis commands for advance scan
- Others (maybe)
- [ ] Support configure for Column storage friendly for OLAP
- [ ] BoltDB as storage engine (read/range optimize)
- [ ] Lua scripts support
- [ ] Support export data to other systems
Note that the project description data, including the texts, logos, images, and/or trademarks,
for each open source project belongs to its rightful owner.
If you wish to add or remove any projects, please contact us at [email protected].