All Projects → wenweihu86 → distkv

wenweihu86 / distkv

Licence: other
Distributed KV Storage System based on Raft and RocksDB, can be use to store small files, like images.

Programming Languages

java
68154 projects - #9 most used programming language
shell
77523 projects
HTML
75241 projects
javascript
184084 projects - #8 most used programming language

Projects that are alternatives of or similar to distkv

Sharkstore
distributed key - value persisted storage system
Stars: ✭ 165 (+230%)
Mutual labels:  rocksdb, raft, distributed-storage
Dragonboat
Dragonboat is a high performance multi-group Raft consensus library in pure Go.
Stars: ✭ 3,983 (+7866%)
Mutual labels:  raft, distributed-storage
Seaweedfs
SeaweedFS is a fast distributed storage system for blobs, objects, files, and data lake, for billions of files! Blob store has O(1) disk seek, cloud tiering. Filer supports Cloud Drive, cross-DC active-active replication, Kubernetes, POSIX FUSE mount, S3 API, S3 Gateway, Hadoop, WebDAV, encryption, Erasure Coding.
Stars: ✭ 13,380 (+26660%)
Mutual labels:  distributed-storage, distributed-file-system
Kites
🪁 A consistency, partition tolerance completed distributed KV store, implementation of the Raft distributed consensus protocol and Kotlin.
Stars: ✭ 41 (-18%)
Mutual labels:  rocksdb, raft
distmq
Distributed Message Queue based on Raft
Stars: ✭ 32 (-36%)
Mutual labels:  raft, distributed-storage
Leofs
The LeoFS Storage System
Stars: ✭ 1,439 (+2778%)
Mutual labels:  distributed-storage, distributed-file-system
Braft
An industrial-grade C++ implementation of RAFT consensus algorithm based on brpc, widely used inside Baidu to build highly-available distributed systems.
Stars: ✭ 2,964 (+5828%)
Mutual labels:  raft, distributed-storage
Go Fastdfs
go-fastdfs 是一个简单的分布式文件系统(私有云存储),具有无中心、高性能,高可靠,免维护等优点,支持断点续传,分块上传,小文件合并,自动同步,自动修复。Go-fastdfs is a simple distributed file system (private cloud storage), with no center, high performance, high reliability, maintenance free and other advantages, support breakpoint continuation, block upload, small file merge, automatic synchronization, automatic r…
Stars: ✭ 2,923 (+5746%)
Mutual labels:  distributed-file-system, distributed-file-storage
Zanredisdb
Yet another distributed kvstore support redis data and index. moved to: https://github.com/youzan/ZanRedisDB
Stars: ✭ 64 (+28%)
Mutual labels:  rocksdb, raft
Tidis
Distributed transactional NoSQL database, Redis protocol compatible using tikv as backend
Stars: ✭ 1,182 (+2264%)
Mutual labels:  rocksdb, raft
fastdfs
FastDFS is an open source high performance distributed file system (DFS). It's major functions include: file storing, file syncing and file accessing, and design for high capacity and load balance. Wechat/Weixin public account (Chinese Language): fastdfs
Stars: ✭ 8,173 (+16246%)
Mutual labels:  distributed-file-system, distributed-file-storage
Raft Java
Raft Java implementation which is simple and easy to understand.
Stars: ✭ 783 (+1466%)
Mutual labels:  raft, distributed-storage
raft-rocks
A simple database based on raft and rocksdb
Stars: ✭ 38 (-24%)
Mutual labels:  rocksdb, raft
Tikv
Distributed transactional key-value database, originally created to complement TiDB
Stars: ✭ 10,403 (+20706%)
Mutual labels:  rocksdb, raft
Vasto
A distributed key-value store. On Disk. Able to grow or shrink without service interruption.
Stars: ✭ 206 (+312%)
Mutual labels:  rocksdb, distributed-storage
balboa
server for indexing and querying passive DNS observations
Stars: ✭ 42 (-16%)
Mutual labels:  rocksdb
paper
Computer Foundations Practices
Stars: ✭ 17 (-66%)
Mutual labels:  raft
engine
Online 4X Grand Strategy RPG Engine
Stars: ✭ 19 (-62%)
Mutual labels:  rocksdb
kedis
Kedis-Server is a Redis-Protocol compatible persistance NoSQL with RocksDB as its storage engine
Stars: ✭ 82 (+64%)
Mutual labels:  rocksdb
little-raft
The lightest distributed consensus library. Run your own replicated state machine! ❤️
Stars: ✭ 316 (+532%)
Mutual labels:  raft

distkv

分布式KV存储系统,也适用于存储小文件,比如图片。

架构图

distkv架构图

各模块功能介绍

  • distkv-store-server:存储实际key/value数据,是多组raft集群,每组是一个分片。
  • distkv-meta-server:存储key对应到distkv-store-server的哪个分片上。
  • distkv-proxy-server:client请求代理层,负责和distkv-meta-server以及distkv-store-server通信。

主要接口的实现

写入接口实现

  • client发送set请求给proxy-server
  • proxy-server根据store-server分片数量,给请求key分配一个分片id(shardingIndex)
  • proxy-server请求meta-server,将key和分片id写入到meta-server中。
  • proxy-server请求分片id对应的store-server集群,将key和value写入到该分片集群中。
  • proxy-server返回给client成功。

读取接口实现

  • client发送get请求给proxy-server
  • proxy-server请求meta-server,获取key对应的分片id
  • proxy请求分片id对应的store-server集群,获取key对应的value
  • proxy将value返回给client

部署

  • 执行mvn clean package,然后将target下的zip包解压到部署目录,执行./bin/run.sh即可。
  • 对于一个模块多节点部署的,需要更改conf和run.sh中的端口,保证端口不要重复。
  • meta-server和store-server都是raft集群,建议部署三节点以上。
  • proxy-server是无状态的,可以部署任意个节点。
  • example模块是spring boot应用,提供了基本的key/value写入/读取使用方法,同时也提供了图片写入/读取的使用方法。
Note that the project description data, including the texts, logos, images, and/or trademarks, for each open source project belongs to its rightful owner. If you wish to add or remove any projects, please contact us at [email protected].