BIGDIS README

Welcome to Bigdis! The Redis ugly brother...

Bigdis is a weekend experiment about writing a Redis "clone" implementing a very strict subset of Redis commands (plain key-value basically) and using the file system as back end, that is: every key is represented as a file.

What is the goal of such a monster you may ask? Short answer: storing very large values.

Many kind of DBs are not well suited for storing large "files" as values. I mean things like images, or videos, and so forth. Still in the web-scale era it is very convenient to be able to access this kind of objects in a distributed fashion, with a networking layer, possibly with a protocol that contains already a large number of tested implementations.

Such a protocol can be indeed the Redis protocol: it's simple to implement and good enough to transfer large data files without too much overhead.

So I wrote this simple proof-of-concept implementation of "Bigdis".

WARNING: this is just an hack I wrote in a few hours in a weekend where there was too wind to go to the sea. Don't use it in production without testing it and possibly fixing it properly. I'll likely don't have time to continue hacking on it so if you are interested please drop me an email.

HOW IT WORKS

Bigdis implements the new Redis multi-bulk protocol, so it does not work with old client libraries.

Every key is stored as a file having as name the hexadecimal SHA1 of the key name itself.

For instance the key "foo" will be stored as a file named:

0beec7b5ea3f0fdbc95d0dd47f3c5bc275da8a33

The first four bytes, in groups of two, are used to locate the directory where the file is located, so the full path for the above key is actually:

/db-root/0b/ee/0beec7b5ea3f0fdbc95d0dd47f3c5bc275da8a33

So there are a total of 65536 directories holding the actual keys. This is useful as filesystems may not handle very well a lot of files inside a single directory.

Keys are created in an atomic fashion using the rename(2) command.

During transfers of bulk values, a temp file is used, so even if you transfer a multi-gigabyte value, there is no need to use a lot of RAM in the server running Bigdis.

USING BIGDIS

Edit bigdis.conf to setup the location of your DB, then execute bigdis with the following command:

tclsh bigdis.tcl

Use any Redis client supporting the new protocol to talk with it.

FUTURE

Do you think this idea is useful? Are you interested in developing it further? Drop me an email. Maybe it can be interesting to port it into Ruby/EM so that there will be more developers that can hack on it.