ToplingDB: A Persistent Key-Value Store for External Storage
ToplingDB is developed and maintained by Topling Inc. It is built with RocksDB.
ToplingDB has much more key features than RocksDB:
- SidePlugin enables users to write a json(or yaml) to define DB configs
- Embeded Http Server enables users to view almost all DB info on web, this is a component of SidePlugin
- Embeded Http Server enables users to online change db/cf options and all db meta objects(such as MemTabFactory, TableFactory, WriteBufferManager ...) without restart the running process
- Many improves and refactories on RocksDB, aimed for performance and extendibility
- ToplingCSPPMemTab(CSPP is Crash Safe Parallel Patricia trie) MemTab, which outperforms SkipList on all aspects: 3x lower memory usage, 7x single thread performance, perfect multi-thread scaling
- ToplingFastTable is an SST implementation optimized for speed, aimed for MemTable flush and L0->L1 compaction.
- ToplingZipTable is an SST implementation optimized for RAM and SSD space, aimed for L2+ level compaction, which used dedicated searchable in-memory data compression algorithms.
- Distributed Compaction for offload compactions on elastic computing clusters, this is more general than RocksDB Compaction Service.
- Builtin SidePlugins for existing RocksDB components(Cache, Comparator, TableFactory, MemTableFactory...)
- Builtin Prometheus metrics support, this is based on Embeded Http Server
- Many bugfixes for RocksDB, a small part of such fixes was Pull Requested to upstream RocksDB
ToplingDB cloud native services
- Todis(Redis on ToplingDB), Todis on aliyun
- ToplingSQL(MySQL on ToplingDB), comming soon...
ToplingDB Components
With SidePlugin mechanics, plugins/components can be physically seperated from core toplingdb
- Compiled to a seperated dynamic lib and loaded at runtime
- User code need not any changes, just change json/yaml files
- Topling's non-open-source enterprise plugins/components are delivered in this way
| Repository | Permission | Description (and components) |
|---|---|---|
| ToplingDB | public | Top repositry, forked from RocksDB with our fixes, refactories and enhancements |
| rockside | public | This is a submodule, contains:
|
| cspp-memtable | private | Auto clone in Makefile, open for partner. Usage doc |
| topling-rocks | private | Auto clone in Makefile, contains: |
private repos are auto cloned in ToplingDB's Makefile, community users has no access permission to these private repos, so the auto clone in Makefile will fail, thus ToplingDB is built without private components, this is so called community version.
Run db_bench
ToplingDB requires gcc 8.4 or newer, or new clang(in near 3 years).
Even without Topling performance components, ToplingDB is much faster than upstream RocksDB:
sudo yum -y install git libaio-devel gcc-c++ gflags-devel zlib-devel bzip2-devel
git clone https://github.com/topling/toplingdb
cd toplingdb
make -j`nproc` db_bench DEBUG_LEVEL=0
cp sideplugin/rockside/src/topling/web/{style.css,index.html} ${/path/to/dbdir}
cp sideplugin/rockside/sample-conf/lcompact_community.yaml .
export LD_LIBRARY_PATH=`find sideplugin -name lib_shared`
# change ./lcompact_community.yaml
# 1. path items (search /dev/shm), if you have no fast disk(such as on a cloud server), use /dev/shm
# 2. change max_background_compactions to your cpu core num
# command option -json can accept json and yaml files, here use yaml file for more human readable
./db_bench -json lcompact_community.yaml -num 10000000 -disable_wal=true -value_size 2000 -benchmarks=fillrandom,readrandom -batch_size=10
# you can access http://127.0.0.1:8081 to see webview
# you can see this db_bench is much faster than RocksDBLicense
We disallow bytedance using this software, other terms are identidal with upstream rocksdb license, see LICENSE.Apache, COPYING and LICENSE.leveldb.
RocksDB: A Persistent Key-Value Store for Flash and RAM Storage
RocksDB is developed and maintained by Facebook Database Engineering Team. It is built on earlier work on LevelDB by Sanjay Ghemawat ([email protected]) and Jeff Dean ([email protected])
This code is a library that forms the core building block for a fast key-value server, especially suited for storing data on flash drives. It has a Log-Structured-Merge-Database (LSM) design with flexible tradeoffs between Write-Amplification-Factor (WAF), Read-Amplification-Factor (RAF) and Space-Amplification-Factor (SAF). It has multi-threaded compactions, making it especially suitable for storing multiple terabytes of data in a single database.
Start with example usage here: https://github.com/facebook/rocksdb/tree/main/examples
See the github wiki for more explanation.
The public interface is in include/. Callers should not include or
rely on the details of any other header files in this package. Those
internal APIs may be changed without warning.
Questions and discussions are welcome on the RocksDB Developers Public Facebook group and email list on Google Groups.
License
RocksDB is dual-licensed under both the GPLv2 (found in the COPYING file in the root directory) and Apache 2.0 License (found in the LICENSE.Apache file in the root directory). You may select, at your option, one of the above-listed licenses.