GitPlanet

Cheap and reliable Node.js hosting starts at $3/month, and $1/month static HTML hosting

Created with love in Canada, visit hostnodejs.com today

Feel like to post an Ad? Learn Details
All Projects → tiglabs → Sharkstore

tiglabs / Sharkstore

Licence: apache-2.0
distributed key - value persisted storage system

Labels

raftrocksdbdistributed-storage

Projects that are alternatives of or similar to Sharkstore

distkv
Distributed KV Storage System based on Raft and RocksDB, can be use to store small files, like images.
Stars: ✭ 50 (-69.7%)
Mutual labels:  rocksdb, raft, distributed-storage
Vasto
A distributed key-value store. On Disk. Able to grow or shrink without service interruption.
Stars: ✭ 206 (+24.85%)
Mutual labels:  rocksdb, distributed-storage
Braft
An industrial-grade C++ implementation of RAFT consensus algorithm based on brpc, widely used inside Baidu to build highly-available distributed systems.
Stars: ✭ 2,964 (+1696.36%)
Mutual labels:  distributed-storage, raft
distmq
Distributed Message Queue based on Raft
Stars: ✭ 32 (-80.61%)
Mutual labels:  raft, distributed-storage
Kites
🪁 A consistency, partition tolerance completed distributed KV store, implementation of the Raft distributed consensus protocol and Kotlin.
Stars: ✭ 41 (-75.15%)
Mutual labels:  rocksdb, raft
Tidis
Distributed transactional NoSQL database, Redis protocol compatible using tikv as backend
Stars: ✭ 1,182 (+616.36%)
Mutual labels:  rocksdb, raft
raft-rocks
A simple database based on raft and rocksdb
Stars: ✭ 38 (-76.97%)
Mutual labels:  rocksdb, raft
Raft Java
Raft Java implementation which is simple and easy to understand.
Stars: ✭ 783 (+374.55%)
Mutual labels:  distributed-storage, raft
Dragonboat
Dragonboat is a high performance multi-group Raft consensus library in pure Go.
Stars: ✭ 3,983 (+2313.94%)
Mutual labels:  distributed-storage, raft
Zanredisdb
Yet another distributed kvstore support redis data and index. moved to: https://github.com/youzan/ZanRedisDB
Stars: ✭ 64 (-61.21%)
Mutual labels:  rocksdb, raft
Tikv
Distributed transactional key-value database, originally created to complement TiDB
Stars: ✭ 10,403 (+6204.85%)
Mutual labels:  rocksdb, raft
Deepsea
A collection of Salt files for deploying, managing and automating Ceph.
Stars: ✭ 140 (-15.15%)
Mutual labels:  distributed-storage
Srchx
A standalone lightweight full-text search engine built on top of blevesearch and Go with multiple storage (scorch, boltdb, leveldb, badger)
Stars: ✭ 118 (-28.48%)
Mutual labels:  rocksdb
Benchmarks
Benchmark of open source, embedded, memory-mapped, key-value stores available from Java (JMH)
Stars: ✭ 116 (-29.7%)
Mutual labels:  rocksdb
Logcabin
LogCabin is a distributed storage system built on Raft that provides a small amount of highly replicated, consistent storage. It is a reliable place for other distributed systems to store their core metadata and is helpful in solving cluster management issues.
Stars: ✭ 1,576 (+855.15%)
Mutual labels:  raft
Seaweedfs
SeaweedFS is a fast distributed storage system for blobs, objects, files, and data lake, for billions of files! Blob store has O(1) disk seek, cloud tiering. Filer supports Cloud Drive, cross-DC active-active replication, Kubernetes, POSIX FUSE mount, S3 API, S3 Gateway, Hadoop, WebDAV, encryption, Erasure Coding.
Stars: ✭ 13,380 (+8009.09%)
Mutual labels:  distributed-storage
Rocksdb
Pure C++ Node.js RocksDB binding. An abstract-leveldown compliant store.
Stars: ✭ 138 (-16.36%)
Mutual labels:  rocksdb
Raft.net
Implementation of RAFT distributed consensus algorithm among TCP Peers on .NET / .NETStandard / .NETCore / dotnet
Stars: ✭ 112 (-32.12%)
Mutual labels:  raft
X0
Xzero HTTP Application Server
Stars: ✭ 111 (-32.73%)
Mutual labels:  raft
Bayard
A full-text search and indexing server written in Rust.
Stars: ✭ 1,555 (+842.42%)
Mutual labels:  raft
View All Similar Projects ➔

SHARKSTORE

中文版本

SharkStore is a distributed, persistent key-value store, whose database layer is based on the rocksdb and the replication works are based on raft.

Modules

  • master-server(golang) -- metadata server
  • data-server(c++) -- database server
  • gateway-server(golang) -- proxy
  • console(golang) -- web administration

Directory

|-- README.md
|-- console                 web administration tool
|-- data-server             business data storage service, where data is stored
|-- glide.yaml
|-- master-server           metadata service, where you get metadata
|-- model
|-- pkg-go                  common module
|-- proxy
    |-- gateway-server      sql/restful proxy

Architecture Design

See arch.md .

Install

See INSTALL.md .

Performance

We created a table named 'metric' witch 12 columns on 3 data servers with NVMe disk, and each of the columns is integer type. The first four columns(named salt, key, host, ts) are made a unified index together as the key. On the other hand, the table is pre-splitted to 100 ranges with 3 replicas running on raft.

  • Batch Insert

    100-row records inserted each time, 61,000,000,000 records inserted totally.

    average response time: 54ms, best tps: 800,000

  • Select

    • select one record by salt+key+host+ts

      average response time: 5ms

    • select 10-100 records by salt+key+host

      average response time: 50ms(before caching) VS 8ms(after caching)

Features

  • SQL syntax compatible and restful api supported
    Users can access by sql client directly, and also by restful api.

  • Dynamic table scheme
    Table columns is allowed to be added or renamed dynamically.

  • Pre-sharding
    Pre-sharding is supported when creating a new table.

  • Globally sorted data
    User can scan tables by primary key globally.

  • Strong consistency.
    Strong consistency is guaranteed by the data replication works on raft group.

  • Online scalability | Auto failover | Auto rebalance | Auto schedule

  • NVMe+SPDK Supported
    Under testing...

License

Under the Apache 2.0 license. See the LICENSE file for details.

Thanks

Many designs are inspired by them. Thanks for their great work.

Note that the project description data, including the texts, logos, images, and/or trademarks, for each open source project belongs to its rightful owner. If you wish to add or remove any projects, please contact us at [email protected].