FDBSwift v4

Fine. I'll do it myself.

Episode IV: A New Swift

This is FoundationDB client for Swift. It's quite low-level, (almost) Foundationless and can into Swift-NIO.

Installation

Obviously, you need to install FoundationDB first. Download it from official website. Next part is tricky because module CFDBSwift (C bindings) won't link libfdb_c library on its own, and FoundationDB doesn't yet ship pkg-config during installation. Therefore you must install it yourself. Run

chmod +x ./scripts/install_pkgconfig.sh
./scripts/install_pkgconfig.sh

or copy scripts/libfdb.pc (choose your platform) to /usr/local/lib/pkgconfig/ on macOS or /usr/lib/pkgconfig/libfdb.pc on Linux.

Migration to v4

v4 isn't a major API upgrade, still breaking and quite massive internally.

The main change is introduction of AnyFDB and AnyFDBTransaction protocols with all public API. Existing FDB and FDB.Transaction classes now adopt these two protocols respectively. All your code that states FDB.Transaction as an argument or return type should be refactored to AnyFDBTransaction its_not_that_hard_steve_brule.jpg

This is done in order to enable advanced testability and mocking (Barbara Liskov as in SOLID, bruh).

Also FDBSwift now supports Swift 5.1 (may require in future). Additionally, FoundationDB 6.2.7 is now supported (hence your FDB servers should be updated in order to use this package).

Everything else is the same as in v3. For now, at least.

Usage

Root concepts

By default (and in the very core) this wrapper, as well as C API, operates with byte keys and values (not pointers, but Array<UInt8>). See Keys, tuples and subspaces section for more details.

Values are always bytes (typealias Bytes = [UInt8]) (or nil if key not found). Why not Data you may ask? I'd like to stay Foundationless for as long as I can (srsly, import half of the world just for Data object which is a fancy wrapper around NSData which is a fancy wrapper around [UInt8]?) (Hast thou forgot that you need to wrap all your Data objects with autoreleasepool or otherwise you get fancy memory leaks?) (except for Linux tho, yes), you can always convert bytes to Data with Data(bytes: myBytes) initializer (why would you want to do that? oh yeah, right, JSON... ok, but do it yourself plz, extensions to the rescue).

First let's deal with synchronous API (there is also asynchronous, using Swift-NIO, see below).

Connection

// Default cluster file path depending on your OS
let fdb = FDB()
// OR
let fdb = FDB(clusterFile: "/usr/local/etc/foundationdb/fdb.cluster")

Optionally you may pass network stop timeout.

Keep in mind that at this point connection has not yet been established, it's automatically established on first actual database operation. If you would like to explicitly connect to database and catch possible errors, you should just call:

try fdb.connect()

Disconnection is automatic, on deinit. But you may also call disconnect() method directly. Be warned that if anything goes wrong during disconnection, you will get uncatchable fatal error. It's not that bad because disconnection should happen only once, when your application shuts down (and you shouldn't really care about fatal errors at that point). Also you very ought to ensure that FDB really disconnected before actual shutdown (trap SIGTERM signal and wait for disconnect to finish), otherwise you might experience undefined behaviour (I personally haven't really encountered that yet, but it's not phantom menace; when you don't follow FoundationDB recommendations, things get quite messy indeed).

Before you connected to FDB cluster you may set network options:

try fdb.setOption(.TLSCertPath(path: "/opt/fdb/tls/chain.pem"))
try fdb.setOption(.TLSPassword(password: "changeme"))
try fdb.setOption(.buggifyEnable)

See FDB+NetworkOptions.swift file for complete set of network options.

Keys, tuples and subspaces

All keys are AnyFDBKey which is a protocol:

public protocol AnyFDBKey {
    func asFDBKey() -> Bytes
}

This protocol is adopted by String, StaticString, Tuple, Subspace and Bytes (aka Array<UInt8>), so you may freely use any of these types, or adopt this protocol in your custom types.

Since you would probably like to have some kind of namespacing in your application, you should stick to Subspace which is an extremely useful instrument for creating namespaces. Under the hood it utilizes the Tuple concept. You oughtn't really bother delving into it, just remember that currently subspaces accept String, Int, Float (aka Float32), Double, Bool, UUID, Tuple (hence FDBTuplePackable), FDB.Null (why would you do that?) and Bytes as arguments.

// dump subspace if you would like to see how it looks from the inside
let rootSubspace = FDB.Subspace("root")
// also check Subspace.swift for more details and usecases
let childSubspace = rootSubspace.subspace("child", "subspace")
// OR
let childSubspace = rootSubspace["child"]["subspace"]
// OR
let childSubspace = rootSubspace["child", "subspace"]

// Talking about tuples:
let tuple = FDB.Tuple(
    Bytes([0, 1, 2]),
    322,
    -322,
    FDB.Null(),
    "foo",
    FDB.Tuple("bar", 1337, "baz"),
    FDB.Tuple(),
    FDB.Null()
)
let packed: Bytes = tuple.pack()
let unpacked: FDB.Tuple = try FDB.Tuple(from: packed)
let tupleBytes: Bytes? = unpacked.tuple[0] as? Bytes
let tupleInt: Int? = unpacked.tuple[1] as? Int
// ...
let tupleEmptyTuple: FDB.Tuple? = unpacked.tuple[6] as? FDB.Tuple
let tupleNull: FDB.Null? = unpacked.tuple[7] as? FDB.Null
if tupleNull is FDB.Null || unpacked.tuple[7] is FDB.Null {}
// you get the idea

Setting values

Simple as that:

try fdb.set(key: "somekey", value: someBytes)
// OR
try fdb.set(key: Bytes([0, 1, 2, 3]), value: someBytes)
// OR
try fdb.set(key: FDB.Tuple("foo", FDB.Null(), "bar", FDB.Tuple("baz", "sas"), "lul"), value: someBytes)
// OR
try fdb.set(key: Subspace("foo").subspace("bar"), value: someBytes)

Getting values

Value is always Bytes? (nil if key not found), you should unwrap it before use. Keys are, of course, still AnyFDBKeys.

let value = try fdb.get(key: "someKey")

Range get (multi get)

Since FoundationDB keys are lexicographically ordered over the underlying bytes, you can get all subspace values (or even from whole DB) by querying range from key somekey\x00 to key somekey\xFF (from byte 0 to byte 255). You shouldn't do it manually though, as Subspace object has a shortcut that does it for you.

Additionally, get(range:) (and its versions) method returns not Bytes, but special structure FDB.KeyValuesResult which holds an array of FDB.KeyValue structures and a flag indicating whether DB can provide more results (pagination, kinda):

public extension FDB {
    /// A holder for key-value pair
    public struct KeyValue {
        public let key: Bytes
        public let value: Bytes
    }
    
    /// A holder for key-value pairs result returned from range get
    public struct KeyValuesResult {
        /// Records returned from range get
        public let records: [FDB.KeyValue]

        /// Indicates whether there are more results in FDB
        public let hasMore: Bool
    }
}

If range call returned zero records, it would result in an empty FDB.KeyValuesResult struct (not nil).

let subspace = FDB.Subspace("root")
let range = subspace.range
/*
  these three calls are completely equal (can't really come up with case when you need second form,
  but whatever, I've seen worse whims)
*/
let result: FDB.KeyValuesResult = try fdb.get(range: range)
let result: FDB.KeyValuesResult = try fdb.get(begin: range.begin, end: range.end)
let result: FDB.KeyValuesResult = try fdb.get(subspace: subspace)

// though call below is not equal to above one because `key(subspace:)` overload implicitly loads range
// this one will load bare subspace key
let result: FDB.KeyValuesResult = try fdb.get(key: subspace)

result.records.forEach {
    dump("\($0.key) - \($0.value)")
    return
}

Clearing values

Clearing (removing, deleting, you name it) records is simple as well.

try fdb.clear(key: childSubspace.subspace("concrete_record"))
// OR
try fdb.clear(key: childSubspace["concrete_record"])
// OR
try fdb.clear(key: rootSubspace["child"]["subspace"]["concrete_record"])
// OR EVEN
try fdb.clear(key: rootSubspace["child", "subspace", "concrete_record"])
// OR EVEN (this is not OK, but still possible :)
try fdb.clear(key: rootSubspace["child", FDB.Null, FDB.Tuple("foo", "bar"), "concrete_record"])

// clears whole subspace, including "concrete_record" key
try fdb.clear(range: childSubspace.range)

Don't forget that actual clearing is not performed until transaction commit.

Atomic operations

FoundationDB also supports atomic operations like ADD, AND, OR, XOR and stuff like that (please refer to docs). You can perform any of these operations with atomic(op🔑value:) method:

try fdb.atomic(.Add, key: key, value: 1)

Knowing that most popular atomic operation is increment (or decrement), I added handy syntax sugar:

try fdb.increment(key: key)
// OR returning incremented value, which is always Int64
let result: Int64 = try fdb.increment(key: key)
// OR
let result = try fdb.increment(key: key, value: 2)

However, keep in mind that example above isn't atomic anymore.

And decrement, which is just a proxy for increment(key:value:), just inverting the value:

let result = try fdb.decrement(key: key)
// OR
let result = try fdb.decrement(key: key, value: 2)

Transactions

All previous examples are utilizing FDB object methods which are implicitly transactional. If you would like to perform more than one operation within one transaction (and experience all delights of ACID), you should first begin transaction using begin() method on FDB object context and then do your stuff (just don't forget to commit() it in the end, by default transactions roll back if not committed explicitly, or after timeout of 5 seconds):

let transaction = try fdb.begin()

// By default transactions are NOT committed, you must do it explicitly or pass optional arg `commit`
try transaction.set(key: "someKey", value: someBytes, commit: true)

try transaction.commit()
// OR
transaction.reset()
// OR
transaction.cancel()
// Or you can just leave transaction object in place and it resets & destroys itself on `deinit`.
// Consider it auto-rollback.
// Please refer to official docs on reset and cancel behaviour:
// https://apple.github.io/foundationdb/api-c.html#c.fdb_transaction_reset

It's not really necessary to commit readonly transaction though :)

Additionally you may set transaction options using transaction.setOption(_:) method:

let transaction: AnyFDBTransaction = ...
try transaction.setOption(.transactionLoggingEnable(identifier: "debuggable_transaction"))
try transaction.setOption(.snapshotRywDisable)

See Transaction+Options.swift file for complete set of options.

Asynchronous API

If your application is NIO-based (pure Swift-NIO or Vapor), you would definitely want (need) to utilize EventLoopFutures, otherwise you are in a great danger of deadlocks which are exceptionally tricky to debug (I've once spent whole weekend debugging my first deadlock, don't repeat my mistakes; thin ice, big time).

If you would like to know Swift-NIO Futures better, please refer to docs.

In order to utilize Futures, you must first have a reference to current EventLoop. If you use Swift-NIO directly, it's available within ChannelHandler.channelRead method, as ChannelHandlerContexts argument property eventLoop. If you use Vapor (starting from version 3.0), it's available from req argument within each action. Please refer to official docs.

All asynchronous stuff (basically mirror methods for all synchronous methods, see Transaction+Sync.swift) is located in Transaction class (see Transaction+NIO.Swift file for complete API), but it all starts with creating a new transaction with EventLoop:

let transactionFuture: EventLoopFuture<Transaction> = fdb.begin(eventLoop: currentEventLoop)

Transaction here is wrapped with an EventLoopFuture because this call may fail (no connection, something is wrong, etc.).

This transaction now supports asynchronous methods (if you try to call asynchronous method on transaction created without EventLoop, you will instantly get failed EventLoopPromise, so take care). Keep in mind that almost all async transaction methods returns not just EventLoopFuture with result (or Void) inside, but a tuple of result and this very same transaction, because if you'd like to commit it yourself, you must have a reference to it, and it's your job to pass this transaction further while you need it.

Conflicts, retries and withTransaction

Since FoundationDB is quite a transactional database, sometimes commits might not succeed due to serialization failures. This can happen when two or more transactions create overlapping conflict ranges. Or, simply speaking, when they try to access or modify same keys (unless they are not in snapshot read mode) at the same time. This is expected (and, in a way, welcomed) behaviour because this is how ACID is achieved.

In these [not-so-rare] cases transaction is allowed to be replayed again. How do you know if transaction can be replayed? It's failed with a special FDB.Error case .transactionRetry(AnyFDBTransaction) which holds current transaction as an associated value. If your transaction (or its respective EventLoopFuture) is failed with this particular error, it means that the transaction has already been rolled back to its initial state and is ready to be executed again.

You can implement this retry logic manually or you can just use FDB instance method withTransaction. This function, as always, comes with two flavors: synchronous and NIO. Following example should be self-explanatory:

let maybeString: String? = try fdb.withTransaction { transaction in
    guard let bytes: Bytes = try transaction.get(key: key) else {
        return nil
    }
    try transaction.commitSync()
    return String(bytes: bytes, encoding: .ascii)
}

// OR

let maybeStringFuture: EventLoopFuture<String?> = fdb.withTransaction(on: myEventLoop) { transaction in
    return transaction
        .get(key: key, commit: true)
        .map { maybeBytes, transaction in
            guard let bytes = maybeBytes else {
                return nil
            }
            return String(bytes: bytes, encoding: .ascii)
        }
}

Thus your block of code will be gently retried until transaction is successfully committed.

Writing to Versionstamped Keys

As a special type of atomic operation, values can be written to special keys that are guaranteed to be unique. These keys make use of an incomplete versionstamp within their tuples, which will be completed by the underlying cluster when data is written. The Versionstamp that was used within a transaction can then be retrieved so it can be referenced elsewhere.

An incomplete versionstamp can be created and added to tuples using the FDB.Versionstamp() initializer. The userData field is optional, and serves to further order keys if multiple are written within the same transaction.

Within a transaction's block, the set(versionstampedKey: key, value: value) method can be used to write to keys with incomplete versionstamps. This method will search the key for an incomplete versionstamp, and if one is found, will flag it to be replaced by a complete versionstamp once it's written to the cluster. If an incomplete versionstamp was not found, a FDB.Error.missingIncompleteVersionstamp error will be thrown.

If you need the complete versionstamp that was used within the key, you can call getVersionstamp() before the transaction is committed. Note that this method must be called within the same transaction that a versionstamped key was written in, otherwise it won't know which versionstamp to return. Also note that this versionstamp does not include any user data that was associated with it, since it will be the same versionstamp no matter how many versionstamped keys were written.

let keyWithVersionstampPlaceholder = self.subspace[FDB.Versionstamp(userData: 42)]["anotherKey"]
let valueToWrite: String = "Hello, World!"

let versionstampedKeyFuture: EventLoopFuture<FDB.Versionstamp> = fdb.withTransaction(on: self.eventLoop) { transaction in
    transaction
        .set(versionstampedKey: keyWithVersionstampPlaceholder, value: Bytes(valueToWrite.utf8))
        .flatMap { _ in
            return transaction.getVersionstamp(commit: true)
        }
}

// When the versionstamp eventually resolves…
versionstampedKeyFuture.whenSuccess { versionstamp in
    var versionstamp = versionstamp
    versionstamp.userData = 42
    
    let actualKey = self.subspace[versionstamp]["anotherKey"]
    
    // … return it to user, save it as a reference to another entry, etc…
}

Complete example

let key = FDB.Subspace("1337")["322"]

let future: EventLoopFuture<String> = fdb.withTransaction(on: myEventLoop) { transaction in
    return transaction
        .setOption(.timeout(milliseconds: 5000))
        .flatMap { transaction in
            transaction.setOption(.snapshotRywEnable)
        }
        .flatMap { transaction in
            transaction.set(key: key, value: Bytes([1, 2, 3]))
        }
        .flatMap { transaction in
            transaction.get(key: key, snapshot: true)
        }
        .flatMapThrowing { (maybeBytes, transaction) -> (String, AnyFDBTransaction) in
            guard let bytes = maybeBytes else {
                throw MyApplicationError.Something("Bytes are not bytes")
            }
            guard let string = String(bytes: bytes, encoding: .ascii) else {
                throw MyApplicationError.Something("String is not string")
            }
            return (string, transaction)
        }
        .flatMap { string, transaction in
            transaction
                .commit()
                .map { _ in string }
        }
}

future.whenSuccess { (resultString: String) in
    print("My string is '\(resultString)'")
}
future.whenFailure { (error: Error) in
    print("Error :C '\(error)'")
}

// OR (you only use wait method outside of main thread or eventLoop thread, because it's blocking)
let string: String = try future.wait()

Of course, in most cases it's much easier and cleaner to just pass commit: true argument into set(key:value:commit:) method (or its siblings), and it will do things for you.

Debugging/logging

FDBSwift supports official community Swift-Log library, therefore you might plug your custom logger into FDB class:

FDB.logger = myCustomLogger

// or project-wise

LoggingSystem.bootstrap(MyLogHandler.init)

See Swift-Log docs for more details on custom loggers.

By default FDBSwift uses very basic factory stdout logger with .info default log level (shouldn't be flooded). If something goes wrong and/or you're not sure what's happening, you just change log level to .debug, just like that:

FDB.logger.logLevel = .debug

Troubleshooting

Q: I cannot compile my project, something like "Undefined symbols for architecture" and tons of similar crap. Pls halp.

A: You haven't properly installed pkg-config for FoundationDB, see Installation section.

Q: I'm getting strange error on second operation: "API version already set". What's happening?

A: You tried to create more than one instance of FDB object, which is a) prohibited b) not needed at all since one instance is just enough for any application (if not, consider horizontal scaling, FDB absolutely shouldn't be a bottleneck of your application). Philosophically speaking it's not very ok, there should be a way of creating more than one of FDB connection in a runtime, and I will definitely try to make it possible. Still, I don't think that FDB connection pooling is a good idea, it already does everything for you.

Q: My application/server just stuck, it stopped responding and dispatching requests. The heck?

A: It's called deadlock. You blocked main/event loop thread. You never block main thread (or event loop thread). It happened because you did some blocking disk or network operation within flatMap/map future closure (probably, while requesting the very same application instance over network). Do your blocking (IO/network) operation within DispatchQueue.async context, resolve it with EventLoopPromise and return future result as EventLoopFuture.

Warnings

Though I aim for full interlanguage compatibility of Tuple layer, I don't guarantee it. During development I refered to Python implementation, but there might be slight differences (like unicode string and byte string packing, see design doc on strings and my comments on that). In general it's should be quite compatible already. Probably one day I'll spend some time on ensuring packing compatibility, but that's not high priority for me.

TODOs

• Enterprise support, vendor WSDL, rewrite on Java Scala Kotlin Java 10
• Drop enterprise support, rewrite on golang using react-native (pretty sure it will be a thing by that time)
• Blockchain? ICO? VR? AR?
• Rehab
• ✅ Proper errors
• ✅ Transactions rollback
• ✅ Tuples
• ✅ Tuples pack
• ✅ Tuples unpack
• ✅ Integer tuples
• ✅ Ranges
• ✅ Subspaces
• ✅ Atomic operations
• ✅ Tests
• ✅ Properly test on Linux
• ✅ 🎉 Asynchronous methods (Swift-NIO)
• ✅ More verbose
• ✅ Even more verbose
• ✅ Transaction options
• ✅ Network options
• ✅ Docblocks and built-in documentation
• ✅ Auto transaction retry if allowed and appropriate
• ✅ 🎉 Even morer verbose (Swift-Log)
• ✅ The rest of tuple pack/unpack (only floats, I think?) (also Bool and UUID)
• More sugar for atomic operations
• The rest of C API (watches?)
• Directories
• Drop VR support