GitPlanet

Cheap and reliable Node.js hosting starts at $3/month, and $1/month static HTML hosting

Created with love in Canada, visit hostnodejs.com today

Feel like to post an Ad? Learn Details
All Projects → malbrain → database

malbrain / database

Licence: BSD-2-Clause license
Key-Value/Document store database library with btree and ARTree indexing methods, SSN-MVCC concurrency

Programming Languages

c
50402 projects - #5 most used programming language
Batchfile
5799 projects

Labels

nosqldocument-databasedatabase-libraryartful-indexbtree-index

Projects that are alternatives of or similar to database

Rethinkdb.driver
🎧 A NoSQL C#/.NET RethinkDB database driver with 100% ReQL API coverage.
Stars: ✭ 350 (+422.39%)
Mutual labels:  nosql, document-database
Azure Cosmos Js
@azure/cosmos has moved to a new repo https://github.com/Azure/azure-sdk-for-js
Stars: ✭ 201 (+200%)
Mutual labels:  nosql, document-database
Orientdb
OrientDB is the most versatile DBMS supporting Graph, Document, Reactive, Full-Text and Geospatial models in one Multi-Model product. OrientDB can run distributed (Multi-Master), supports SQL, ACID Transactions, Full-Text indexing and Reactive Queries. OrientDB Community Edition is Open Source using a liberal Apache 2 license.
Stars: ✭ 4,394 (+6458.21%)
Mutual labels:  nosql, document-database
skytable
Skytable is an extremely fast, secure and reliable real-time NoSQL database with automated snapshots and TLS
Stars: ✭ 696 (+938.81%)
Mutual labels:  nosql, document-database
uptasticsearch
An Elasticsearch client tailored to data science workflows.
Stars: ✭ 47 (-29.85%)
Mutual labels:  nosql, document-database
Ftserver Cs
Lightweight iBoxDB Full Text Search Server for C#
Stars: ✭ 81 (+20.9%)
Mutual labels:  nosql, document-database
Arangodb
🥑 ArangoDB is a native multi-model database with flexible data models for documents, graphs, and key-values. Build high performance applications using a convenient SQL-like query language or JavaScript extensions.
Stars: ✭ 11,880 (+17631.34%)
Mutual labels:  nosql, document-database
Ftserver
Lightweight Embeddable iBoxDB Full Text Search Server for Java
Stars: ✭ 219 (+226.87%)
Mutual labels:  nosql, document-database
Tiedot
A rudimentary implementation of a basic document (NoSQL) database in Go
Stars: ✭ 2,643 (+3844.78%)
Mutual labels:  nosql, document-database
Mongodb Quickstart Course
Course demos and handout material for Talk Python's MongoDB Quickstart course
Stars: ✭ 220 (+228.36%)
Mutual labels:  nosql, document-database
NoSQLDataEngineering
NoSQL Data Engineering
Stars: ✭ 25 (-62.69%)
Mutual labels:  nosql, document-database
Ravendb
ACID Document Database
Stars: ✭ 2,870 (+4183.58%)
Mutual labels:  nosql, document-database
docs
Source code of the ArangoDB online documentation
Stars: ✭ 18 (-73.13%)
Mutual labels:  nosql, document-database
soda-for-java
SODA (Simple Oracle Document Access) for Java is an Oracle library for writing Java apps that work with JSON (and not only JSON!) in the Oracle Database. SODA allows your Java app to use the Oracle Database as a NoSQL document store.
Stars: ✭ 61 (-8.96%)
Mutual labels:  nosql
SimpleSQLite
SimpleSQLite is a Python library to simplify SQLite database operations: table creation, data insertion and get data as other data formats. Simple ORM functionality for SQLite.
Stars: ✭ 116 (+73.13%)
Mutual labels:  database-library
unqlite.rs
UnQLite wrapper 1.0 is avaliable for Rust
Stars: ✭ 99 (+47.76%)
Mutual labels:  nosql
workshop-intro-to-cassandra
Learn Apache Cassandra fundamentals in this hands-on workshop
Stars: ✭ 208 (+210.45%)
Mutual labels:  nosql
chainDB
A noSQL database based on blockchain technology
Stars: ✭ 13 (-80.6%)
Mutual labels:  nosql
location-api-sl
This API can be use to all developers to get location details of Sri Lanka 🇱🇰 including major cities, sub areas, districts and Provinces. ⛳️
Stars: ✭ 35 (-47.76%)
Mutual labels:  nosql
gorm-neo4j
GORM for Neo4j
Stars: ✭ 16 (-76.12%)
Mutual labels:  nosql
View All Similar Projects ➔

malbrain/database

A working project for High-concurrency B-tree/ARTree Database source code in C. This project was created as a sub-module for the /www.github.com/malbrain/javascript-database project, but it can also be used by itself as a database/indexing library. The API is documented on the Wiki pages.

The testfiles directory exercises the basic functionality of the database, and can be used as a sample database application. For a more complete example of how MVCC and transactions can be implemented on the database core functionality, please see the javascript-database project, partcularly the js_db*.c files.

Clone the database project:

git clone --recursive https://github.com/malbrain/database.git

Compile the database library and standalone test module with ./build or build.bat.

Usage: standalone db_name -cmds=[wdf] -summary=[csrvikdn] -idxType=[012] -bits=# -xtra=# -inMem -stats -prng=# -debug -monitor -uniqueKeys -noDocs -noIdx -keyLen=# -minKey=abcd -maxKey=abce -drop -idxBinary=. -pipeline src_file1 src_file2 ... ]
  where db_name is the prefix name of the database document and index files
  cmds is a string of (w)rite/(d)elete/(f)ind commands, to run sequentially on each input src_file.
  summary scan is a string of (c)ount/(r)everse scan/(s)can/(v)erify/(i)terate/(k)ey list(d)ump doc(n)umber flags for a final scan after all threads have quit
  pennysort creates random 100 byte B64 input lines, sets keyLen to 10, line count from the file name
  idxType is the type of index: 0 = ART, 1 = btree1, 2 = btree2
  keyLen is key size, zero for whole input file line
  bits is the btree page size in bits
  xtra is the btree leaf page extra bits
  inMem specifies no disk files
  noDocs specifies keys only
  noIdx specifies documents only
  minKey specifies beginning cursor key
  maxKey specifies ending cursor key
  drop will initially drop database
  idxBinary utilize length counted fields separated by  the given deliminator in keys
  use prng number stream [012] for pennysort keys
  stats keeps performance and debugging totals and prints them
  uniqueKeys ensure keys are unique
  run cmds in a single threaded  pipeline using one input file at a time
  src_file1 thru src_filen are files of keys/documents separated by newline

Linux compilation command:

[karl@test7x64 xlink]# cc -std=c11 -O2 -g -o standalone standalone.c base64.c db*.c artree/*.c btree1/*.c btree2/*.c mutex/mutex.c rwlock/readerwriter.c -lpthread

**********************************************************************
** Visual Studio 2019 Developer Command Prompt v16.2.3
** Copyright (c) 2019 Microsoft Corporation
**********************************************************************
[vcvarsall.bat] Environment initialized for: 'x64'


c:\Users\Owner\Source\Repos\malbrain\database>testfiles\test1
Alpha testing of binary string fields
expect 13 sorted keys

standalone testdb -cmds=w -summary=skn -idxBinary=: testfiles/test1

thrd:0 cmd:w ARTree: begin
thrd:0 cmd:w file:testfiles/test1 load records
thrd:0 cmd:w file:testfiles/test1 records processed: 13
thrd:0 cmd:w end

 real 0m0.002s
 user 0m0.000s
 sys  0m0.000s

Index ARTree summary scan:
 forward index cursor
*}-Wz1;TD-
0fssx}~[oB
5HA\z%qt{%
AsfAGHM5om
Q)JN)R9z-L
abc:def
abcd:ef
abcde:f
my+=5r7(N|
mz4VCN@a#"
o4FoBkqERn
uI^EYm8s=|
~sHd0jDv6X
 Index scan complete
 Total keys 13

 real 0m0.005s
 user 0m0.000s
 sys  0m0.015s

Total memory allocated: 3.616 MB
Bytes per key: 291692

ARTree Index type 1 blks allocated: 00000002
ARTree Index type 2 blks allocated: 00000001
ARTree Index type 5 blks allocated: 00000016
ARTree Index type 6 blks allocated: 00000013
ARTree Index type 8 blks allocated: 00000022
ARTree Index type 9 blks allocated: 00000010

ARTree Index type 1 blks freed    : 00000001
ARTree Index type 8 blks freed    : 00000001
ARTree Index type 9 blks freed    : 00000002


**********************************************************************


c:\Users\Owner\Source\Repos\malbrain\database>testfiles\test2
"40M random 10 byte keys, single threaded insertions into each index type"

standalone.exe testdb -stats -cmds=w -summary=vc -idxType=2 -bits=16  -inMem -noDocs -pennysort 40000000

thrd:0 cmd:w Btree2: begin
thrd:0 cmd:w random keys:40000000
thrd:0 cmd:w file:40000000 records processed: 40000000
thrd:0 cmd:w end

 real 1m14.259s
 user 1m13.843s
 sys  0m0.406s

Index Btree2 summary scan:
 forward index cursor
 key order verification
 index key count
 Index scan complete
 Total keys 40000000

 real 0m2.508s
 user 0m2.500s
 sys  0m0.000s

Total memory allocated: 2154.231 MB
Bytes per key: 56

standalone.exe testdb -stats -cmds=w -summary=vc -idxType=1 -bits=16  -inMem -noDocs -pennysort 40000000

thrd:0 cmd:w Btree1: begin
thrd:0 cmd:w random keys:40000000
thrd:0 cmd:w file:40000000 records processed: 40000000
thrd:0 cmd:w end

 real 0m53.759s
 user 0m53.531s
 sys  0m0.218s

Index Btree1 summary scan:
 forward index cursor
 key order verification
 index key count
 Index scan complete
 Total keys 40000000

 real 0m2.383s
 user 0m2.375s
 sys  0m0.000s

Total memory allocated: 1027.947 MB
Bytes per key: 26


**********************************************************************


c:\Users\Owner\Source\Repos\malbrain\database>testfiles\test3
"40M random 10 byte keys, multi-threaded insertions into Adaptive Radix Tree index""

standalone testdb -prng=2 -stats -cmds=w -summary=vc -idxType=0 -bits=16  -inMem -noDocs -pennysort -threads=4 10000000
thread 0 launched for file 10000000 cmds w
thread 1 launched for file 10000000 cmds w
thread 2 launched for file 10000000 cmds w
thread 3 launched for file 10000000 cmds w

thrd:3 cmd:w ARTree: begin
thrd:3 cmd:w random keys:10000000
thrd:3 cmd:w file:10000000 records processed: 10000000
thrd:3 cmd:w end

 real 0m8.588s
 user 0m33.812s
 sys  0m0.546s

thrd:1 cmd:w ARTree: begin
thrd:1 cmd:w random keys:10000000
thrd:1 cmd:w file:10000000 records processed: 10000000
thrd:1 cmd:w end

 real 0m8.592s
 user 0m33.812s
 sys  0m0.546s

thrd:0 cmd:w ARTree: begin
thrd:0 cmd:w random keys:10000000
thrd:0 cmd:w file:10000000 records processed: 10000000
thrd:0 cmd:w end

 real 0m8.596s
 user 0m33.812s
 sys  0m0.546s

thrd:2 cmd:w ARTree: begin
thrd:2 cmd:w random keys:10000000
thrd:2 cmd:w file:10000000 records processed: 10000000
thrd:2 cmd:w end

 real 0m8.615s
 user 0m33.843s
 sys  0m0.546s

Index ARTree summary scan:
 forward index cursor
 key order verification
 index key count
 Index scan complete
 Total keys 40000000

 real 0m18.537s
 user 0m18.532s
 sys  0m0.000s

Total memory allocated: 3164.795 MB
Bytes per key: 82

ARTree Index type 1 blks allocated: 12468681
ARTree Index type 2 blks allocated: 01688726
ARTree Index type 3 blks allocated: 00266305
ARTree Index type 6 blks allocated: 40000000
ARTree Index type 8 blks allocated: 92558615
ARTree Index type 9 blks allocated: 00000065

ARTree Index type 1 blks freed    : 01688726
ARTree Index type 2 blks freed    : 00266305
ARTree Index type 8 blks freed    : 12468616
ARTree Index type 9 blks freed    : 00000065


**********************************************************************


c:\Users\Owner\Source\Repos\malbrain\database>testfiles\test4
"40M random 10 byte keys, multi-threaded insertions into btree1 index"

standalone testdb -prng=2 -stats -cmds=w -summary=vc -idxType=1 -bits=16  -inMem -noDocs -pennysort -threads=4 10000000
thread 0 launched for file 10000000 cmds w
thread 1 launched for file 10000000 cmds w
thread 2 launched for file 10000000 cmds w
thread 3 launched for file 10000000 cmds w

thrd:1 cmd:w Btree1: begin
thrd:1 cmd:w random keys:10000000
thrd:1 cmd:w file:10000000 records processed: 10000000
thrd:1 cmd:w end

 real 0m16.332s
 user 1m5.046s
 sys  0m0.250s

thrd:0 cmd:w Btree1: begin
thrd:0 cmd:w random keys:10000000
thrd:0 cmd:w file:10000000 records processed: 10000000
thrd:0 cmd:w end

 real 0m16.359s
 user 1m5.140s
 sys  0m0.250s

thrd:3 cmd:w Btree1: begin
thrd:3 cmd:w random keys:10000000
thrd:3 cmd:w file:10000000 records processed: 10000000
thrd:3 cmd:w end

 real 0m16.383s
 user 1m5.171s
 sys  0m0.250s

thrd:2 cmd:w Btree1: begin
thrd:2 cmd:w random keys:10000000
thrd:2 cmd:w file:10000000 records processed: 10000000
thrd:2 cmd:w end

 real 0m16.460s
 user 1m5.250s
 sys  0m0.265s

Index Btree1 summary scan:
 forward index cursor
 key order verification
 index key count
 Index scan complete
 Total keys 40000000

 real 0m2.383s
 user 0m2.375s
 sys  0m0.000s

Total memory allocated: 1032.647 MB
Bytes per key: 27


**********************************************************************


c:\Users\Owner\Source\Repos\malbrain\database>testfiles\test5
"10M random 10 byte keys, insertions then searches in ARTree index"

standalone testdb -prng=2 -stats -cmds=wf -summary=vc -idxType=0 -bits=16  -inMem -noDocs -pennysort 10000000

thrd:0 cmd:w ARTree: begin
thrd:0 cmd:w random keys:10000000
thrd:0 cmd:w file:10000000 records processed: 10000000
thrd:0 cmd:w end

 real 0m5.872s
 user 0m5.750s
 sys  0m0.125s

thrd:0 cmd:f ARTree: begin
thrd:0 cmd:f random keys:10000000
thrd:0 cmd:f file:10000000 records processed: 10000000
thrd:0 cmd:f end

 real 0m4.297s
 user 0m4.297s
 sys  0m0.000s

Index ARTree summary scan:
 forward index cursor
 key order verification
 index key count
 Index scan complete
 Total keys 10000000

 real 0m3.682s
 user 0m3.672s
 sys  0m0.000s

Total memory allocated: 868.290 MB
Bytes per key: 91

ARTree Index type 1 blks allocated: 02309998
ARTree Index type 2 blks allocated: 00271819
ARTree Index type 3 blks allocated: 00266275
ARTree Index type 6 blks allocated: 10000000
ARTree Index type 8 blks allocated: 22336227
ARTree Index type 9 blks allocated: 00000065

ARTree Index type 1 blks freed    : 00271819
ARTree Index type 2 blks freed    : 00266275
ARTree Index type 8 blks freed    : 02309933
ARTree Index type 9 blks freed    : 00000065


**********************************************************************


c:\Users\Owner\Source\Repos\malbrain\database>testfiles\test5
"10M random 10 byte keys, insertions then searches in ARTree index"

standalone testdb -prng=2 -stats -cmds=wf -summary=vc -idxType=0 -bits=16  -inMem -noDocs -pennysort 10000000

thrd:0 cmd:w ARTree: begin
thrd:0 cmd:w random keys:10000000
thrd:0 cmd:w file:10000000 records processed: 10000000
thrd:0 cmd:w end

 real 0m5.901s
 user 0m5.703s
 sys  0m0.203s

thrd:0 cmd:f ARTree: begin
thrd:0 cmd:f random keys:10000000
thrd:0 cmd:f file:10000000 records processed: 10000000
thrd:0 cmd:f end

 real 0m4.336s
 user 0m4.343s
 sys  0m0.000s

Index ARTree summary scan:
 forward index cursor
 key order verification
 index key count
 Index scan complete
 Total keys 10000000

 real 0m4.278s
 user 0m4.266s
 sys  0m0.000s

Total memory allocated: 868.853 MB
Bytes per key: 91

ARTree Index type 1 blks allocated: 02309998
ARTree Index type 2 blks allocated: 00271819
ARTree Index type 3 blks allocated: 00266275
ARTree Index type 6 blks allocated: 10000000
ARTree Index type 8 blks allocated: 22336227
ARTree Index type 9 blks allocated: 00000065

ARTree Index type 1 blks freed    : 00271819
ARTree Index type 2 blks freed    : 00266275
ARTree Index type 8 blks freed    : 02309933
ARTree Index type 9 blks freed    : 00000065


**********************************************************************


c:\Users\Owner\Source\Repos\malbrain\database>testfiles\test6
"40M random 10 byte keys, multi-threaded insertions then searches in btree1 index"

standalone testdb -prng=2 -stats -cmds=w -summary=vc -idxType=0 -bits=16 -noDocs -pennysort -threads=4 10000000
thread 0 launched for file 10000000 cmds w
thread 1 launched for file 10000000 cmds w
thread 2 launched for file 10000000 cmds w
thread 3 launched for file 10000000 cmds w

thrd:1 cmd:w ARTree: begin
thrd:1 cmd:w random keys:10000000
thrd:1 cmd:w file:10000000 records processed: 10000000
thrd:1 cmd:w end

 real 0m9.490s
 user 0m34.671s
 sys  0m2.562s

thrd:0 cmd:w ARTree: begin
thrd:0 cmd:w random keys:10000000
thrd:0 cmd:w file:10000000 records processed: 10000000
thrd:0 cmd:w end

 real 0m9.514s
 user 0m34.718s
 sys  0m2.562s

thrd:2 cmd:w ARTree: begin
thrd:2 cmd:w random keys:10000000
thrd:2 cmd:w file:10000000 records processed: 10000000
thrd:2 cmd:w end

 real 0m9.522s
 user 0m34.750s
 sys  0m2.562s

thrd:3 cmd:w ARTree: begin
thrd:3 cmd:w random keys:10000000
thrd:3 cmd:w file:10000000 records processed: 10000000
thrd:3 cmd:w end

 real 0m9.578s
 user 0m34.781s
 sys  0m2.578s

Index ARTree summary scan:
 forward index cursor
 key order verification
 index key count
 Index scan complete
 Total keys 40000000

 real 0m18.846s
 user 0m18.828s
 sys  0m0.015s

Total memory allocated: 3165.380 MB
Bytes per key: 82

ARTree Index type 1 blks allocated: 12468681
ARTree Index type 2 blks allocated: 01688726
ARTree Index type 3 blks allocated: 00266305
ARTree Index type 6 blks allocated: 40000000
ARTree Index type 8 blks allocated: 92558615
ARTree Index type 9 blks allocated: 00000065

ARTree Index type 1 blks freed    : 01688726
ARTree Index type 2 blks freed    : 00266305
ARTree Index type 8 blks freed    : 12468616
ARTree Index type 9 blks freed    : 00000065


standalone testdb -prng=2 -stats -cmds=f -summary=vc -idxType=0 -bits=16 -noDocs -pennysort -threads=4 10000000
thread 0 launched for file 10000000 cmds f
thread 1 launched for file 10000000 cmds f
thread 2 launched for file 10000000 cmds f
thread 3 launched for file 10000000 cmds f

thrd:1 cmd:f ARTree: begin
thrd:1 cmd:f random keys:10000000
thrd:1 cmd:f file:10000000 records processed: 10000000
thrd:1 cmd:f end

 real 0m6.206s
 user 0m23.578s
 sys  0m1.234s

thrd:2 cmd:f ARTree: begin
thrd:2 cmd:f random keys:10000000
thrd:2 cmd:f file:10000000 records processed: 10000000
thrd:2 cmd:f end

 real 0m6.289s
 user 0m23.812s
 sys  0m1.234s

thrd:0 cmd:f ARTree: begin
thrd:0 cmd:f random keys:10000000
thrd:0 cmd:f file:10000000 records processed: 10000000
thrd:0 cmd:f end

 real 0m6.357s
 user 0m23.937s
 sys  0m1.234s

thrd:3 cmd:f ARTree: begin
thrd:3 cmd:f random keys:10000000
thrd:3 cmd:f file:10000000 records processed: 10000000
thrd:3 cmd:f end

 real 0m6.393s
 user 0m23.984s
 sys  0m1.234s

Index ARTree summary scan:
 forward index cursor
 key order verification
 index key count
 Index scan complete
 Total keys 40000000

 real 0m18.281s
 user 0m18.281s
 sys  0m0.000s

Total memory allocated: 0.443 MB
Bytes per key: 0


**********************************************************************


c:\Users\Owner\Source\Repos\malbrain\database>testfiles\test7
"40M random 10 byte keys, multi-threaded insertions then subset cursor over btree1 index"

standalone testdb -prng=2 -stats -cmds=w -summary=k -idxType=1 -bits=16 -minKey=aaaA -maxKey=aaaK -noDocs -pennysort -threads=4 10000000
thread 0 launched for file 10000000 cmds w
thread 1 launched for file 10000000 cmds w
thread 2 launched for file 10000000 cmds w
thread 3 launched for file 10000000 cmds w

thrd:3 cmd:w Btree1: begin
thrd:3 cmd:w random keys:10000000
thrd:3 cmd:w file:10000000 records processed: 10000000
thrd:3 cmd:w end

 real 0m16.527s
 user 1m5.546s
 sys  0m0.562s

thrd:0 cmd:w Btree1: begin
thrd:0 cmd:w random keys:10000000
thrd:0 cmd:w file:10000000 records processed: 10000000
thrd:0 cmd:w end

 real 0m16.563s
 user 1m5.640s
 sys  0m0.562s

thrd:1 cmd:w Btree1: begin
thrd:1 cmd:w random keys:10000000
thrd:1 cmd:w file:10000000 records processed: 10000000
thrd:1 cmd:w end

 real 0m16.583s
 user 1m5.703s
 sys  0m0.562s

thrd:2 cmd:w Btree1: begin
thrd:2 cmd:w random keys:10000000
thrd:2 cmd:w file:10000000 records processed: 10000000
thrd:2 cmd:w end

 real 0m16.675s
 user 1m5.796s
 sys  0m0.562s

Index Btree1 summary scan:
 forward index cursor
 min key: <aaaA>
 max key: <aaaK>
aaaBCw`d42
aaaBONxqOK
aaaBm90ePd
aaaCctIYQw
aaaCgdnoHA
aaaCpkDRZ6
aaaD@tGrtK
aaaDL4uFgu
aaaDihSWuY
aaaDyhKhZg
aaaE0tam4P
aaaFiGHVmr
aaaG5S2Vod
aaaHQtKXpk
aaaHyG0MNI
aaaI4R9OlN
aaaIJp7aPf
aaaJBwd`ex
aaaJjg1Fhs
aaaKQGRIk0
aaaKcxeoHS
aaaKfOd4i3
aaaKshGObR
 Index scan complete
 Total keys 23

 real 0m0.019s
 user 0m0.000s
 sys  0m0.000s

Total memory allocated: 1031.715 MB
Bytes per key: 47036151
Note that the project description data, including the texts, logos, images, and/or trademarks, for each open source project belongs to its rightful owner. If you wish to add or remove any projects, please contact us at [email protected].