coleifer / greendb

greendb

async server frontend for symas lmdb.

description

greendb is a lightweight server (and Python client) for symas lmdb. The server uses the new Redis RESPv3 protocol.

greendb supports multiple independent databases, much like Redis. Values are serialized using msgpack, so the server is capable of storing all the data-types supported by msgpack. greendb also provides per-database configuration of multi-value support, allowing you to efficiently store multiple values at a given key, in sorted order (e.g. for secondary indexes).

With greendb, database keys are always bytestrings. Values may be:

• dict
• list
• set
• bytestrings
• unicode strings
• integers
• floating-point
• boolean
• None

installing

$ pip install greendb

Alternatively, you can install from git:

$ git clone https://github.com/coleifer/greendb
$ cd greendb
$ python setup.py install

Dependencies:

• gevent
• lmdb
• msgpack-python

running

$ greendb.py -h

Usage: greendb.py [options]

Options:
  -h, --help            show this help message and exit
  -c CONFIG, --config=CONFIG
                        Config file (default="config.json")
  -D DATA_DIR, --data-dir=DATA_DIR
                        Directory to store db environment and data.
  -d, --debug           Log debug messages.
  -e, --errors          Log error messages only.
  -H HOST, --host=HOST  Host to listen on.
  -l LOG_FILE, --log-file=LOG_FILE
                        Log file.
  -m MAP_SIZE, --map-size=MAP_SIZE
                        Maximum size of memory-map used for database. The
                        default value is 256M and should be increased. Accepts
                        value in bytes or file-size using "M" or "G" suffix.
  --max-clients=MAX_CLIENTS
                        Maximum number of clients.
  -n MAX_DBS, --max-dbs=MAX_DBS
                        Number of databases in environment. Default=16.
  -p PORT, --port=PORT  Port to listen on.
  -r, --reset           Reset database and config. All data will be lost.
  -s, --sync            Flush system buffers to disk when committing a
                        transaction. Durable but much slower.
  -u DUPSORT, --dupsort=DUPSORT
                        db index(es) to support dupsort
  -M, --no-metasync     Flush system buffers to disk only once per
                        transaction, omit the metadata flush.
  -W, --writemap        Use a writeable memory map.
  -A, --map-async       When used with "--writemap" (-W), use asynchronous
                        flushes to disk.

Complete config file example with default values:

{
  "host": "127.0.0.1",
  "port": 31337,
  "max_clients": 1024,
  "path": "data",  // Directory for data storage, default is "data" in CWD.
  "map_size": "256M",  // Default map size is 256MB. INCREASE THIS!
  "read_only": false,  // Open the database in read-only mode.
  "metasync": true,  // Sync metadata changes (recommended).
  "sync": false,  // Sync all changes (durable, but much slower).
  "writemap": false,  // Use a writable map (probably safe to do).
  "map_async": false,  // Asynchronous writable map.
  "meminit": true,  // Initialize new memory pages to zero.
  "max_dbs": 16,  // Maximum number of DBs.
  "max_spare_txns": 64,
  "lock": true,  // Lock the database when opening environment.
  "dupsort": false  // Either a boolean or a list of DB indexes.
}

Example custom configuration:

• 1GB max database size
• dupsort enabled on databases 13, 14 and 15
• data stored in /var/lib/greendb/data

{
  "map_size": "1G",
  "dupsort": [13, 14, 15],
  "path": "/var/lib/greendb/data/"
}

Equivalent configuration using command-line arguments:

$ greendb.py -m 1G -u 13 -u 14 -u 15 -D /var/lib/greendb/data/

client

A Python client is included in the greendb module. All server commands are implemented as client methods using the lower-case command name, for example:

from greendb import Client
client = Client(host='10.0.0.3')

# Execute the ENVINFO command.
print(client.envinfo())

# Set multiple key/value pairs, read a key, then delete two keys.
client.mset({'k1': 'v1', 'k2': 'v2'})  # MSET
print(client.get('k1'))  # GET
client.mdelete(['k1', 'k2'])  # MDELETE

Additionally, the Client implements much of the Python dict interface, such as item get/set/delete, iteration, length, contains, etc.

If an error occurs, either due to a malformed command (e.g., missing required parameters) or for any other reason (e.g., attempting to write to a read-only database), then a CommandError will be raised by the client with a message indicating what caused the error.

A note about connections: the greendb client will automatically connect the first time you issue a command to the server. The client maintains its own thread-safe (and greenlet-safe) connection pool. If you wish to explicitly connect, the client may be used as a context manager.

In a multi-threaded or multi-greenlet application (e.g. a web app), the client will maintain a separate connection for each thread/greenlet.

command reference

Below is the list of supported commands. Commands are available on the client using the lower-case command name as the method name.

protocol

The protocol is the Redis RESPv3 protocol. I have extended the protocol with one additional data-type, a dedicated type for representing UTF8-encoded unicode text (notably absent from RESPv3). This type is denoted by the leading ^ byte in responses.

Details can be found here: https://github.com/antirez/RESP3/blob/master/spec.md