Mytosis

A peer-to-peer data sync framework

Maintenance Notice

⚠️ None of this is maintained anymore.

It was a proof of concept. I learned a lot, and I'm slowly designing a more robust and verifyable system (see here), but in the meantime you might as well regard it as abandoned.

Introduction

Mytosis organizes data as one massive object which contains other objects. The root is called the graph, and its children are called nodes.

Graph {
  node1
  node2
  node3
  ...
}

A key concept is that nodes cannot contain other nodes. To express relationships, Mytosis uses pointers, sometimes called "edges". Those edges are automatically resolved, allowing you to structure your data as a tree, as tables, as key-value pairs, or an interconnected mesh.

Graph {
  book {
    title: "Hitchhiker's Guide"
    author: Pointer(author)
  }

  author {
    name: "Douglas Adams"
    authored: Pointer(books)
  }

  books {
    book: Pointer(book)
  }
}

To mutate the graph, you declare all fields which changed along with their new values. Mytosis will generate a patch update and give it to your storage and network plugins.

Installing

This package is available on npm.

# Install with npm
npm install mytosis --save

# Install with yarn
yarn add mytosis

Now you can import it into your project.

// ES Modules
import database from 'mytosis'

// CommonJS
const database = require('mytosis').default

API

The main API surface is intentionally small. It consists of only a few methods.

If you need more fine-grained control, you might find the graph-crdt documentation useful.
Database and Context inherit from Graph and Node (respectively).

database(...configs)

Creates a new database. It can be used with or without the new keyword.

By default, the database operates as an in-memory cache, and can be extended by passing in plugins. There are tons of options, so we'll cover those later.

const db = database()

Options

Most methods accept an options object as the last parameter, which supports the following:

• options.storage - Overrides the storage defaults. Can be a storage plugin or null.
• options.network - Overrides the network defaults. Can be a ConnectionGroup or null.

db.read(key[, options])

Reads a Node from the database, resolved through a promise.

Node

An object containing only primitives. Nested objects are represented as pointers.

const stats = await db.read('stats')

console.log('Stats:', stats)

If the node can't be found, null is returned.

Options

Same options as db.nodes().

db.nodes([...keys][, options])

Reads many keys simultaneously, resolving with an array of nodes.

Note: This isn't just Promise.all. It's more performant than db.read, especially if you're querying the network.

// Reads both `timeline` and `profile/<userid>`.
const [timeline, profile] = await db.nodes(['timeline', `profile/${userId}`])

console.log('Profile:', profile.snapshot())

If a node can't be found, it's value is null.

Options

All standard options are supported. In addition...

• options.force - Ignore the in-memory cache and force a read from the plugins. Expects a boolean value.

db.write(key, patch[, options])

Updates properties on a node. If the node doesn't exist, it's created.

db.write('preferences', {
  notificationsDisabled: true,
  theme: 'dark',
})

Everything in Mytosis is a patch update. You only need to declare the properties you're changing.

db.branch()

Clones the current database into a new in-memory store. Nothing you change will have an effect on the source database until you choose to commit() them.

Hooks and API extensions are shared with branches, but storage and network plugins are ignored.

Some use cases are:

• Describing a collection of changes which should be applied simultaneously
• Making edits which might be later cancelled

const update = db.branch()

// These writes won't affect `db`.
await update.write('settings', {
  bio: 'It all started when...',
  theme: 'Monokai',
})

await update.write('contact', {
  email: '[email protected]',
})

// Write all the changes at once.
await db.commit(update)

Note: Changes to the source database will not affect branches.
You may want to rebase() your changes before calling commit().

db.commit(changes[, options])

Applies a collection of changes all at once, represented as a graph. Useful for committing changes pushed from other replicas or branched databases.

It's used under the hood by node.write and db.write.

import { Graph, Node } from 'mytosis'

const graph = new Graph()
const node = new Node({ uid: 'change' })

node.merge({ changes: 'eh, probably' })
graph.merge({ change: node })

await db.commit(graph)

Events

Each mutation will emit an "update" event, passing a graph containing only the changes. There's also a "history" event when properties are overwritten. If you keep track of these deltas, you can roll time backwards and forwards.

node.read(key[, options])

Reads a primitive value from the node. If the value is a pointer to another node, Mytosis will automatically resolve it.

const weather = await db.write('weather', {
  temperature: 25,
})

const temperature = await weather.read('temperature')

node.fields([...keys][, options])

Read a list of fields in a single request. Pointers are resolved using db.nodes for efficient read batching.

const [name, friends] = await user.fields(['name', 'friends'])

Standard options are accepted.

node.write(key, value[, options])

Writes a value to the node. The value can be any primitive.

const company = await db.write('company', {})

await company.write('phone', '456-123-8970')

To create a pointer to another node, you can write the reference.

// Create two nodes...
const dave = await db.write('user', { name: 'dave' })
const company = await db.write('company', { name: 'Enterprise Inc.' })

// Link one to the other.
await dave.write('workplace', company)

// Prints "Enterprise Inc."
dave.read('company')
  .then(company => company.read('name'))
  .then(name => console.log('Name:', name))

Events

Each node inherits from an event emitter. Any mutation triggers "update", passing the changes. This is useful for observing real-time data and reacting to changes.

const usage = await db.read('usage-stats')

usage.on('update', (changes) => {
  console.log('Fields changed:', ...changes)
})

Streaming

You can read everything in storage as a stream of nodes. Streams use JavaScript async iteration, and completely ignore the in-memory cache, going straight to the storage plugin.

for await (const node of db) {
  console.log('Node ID:', String(node))
  console.log('Value:', node.snapshot())
}

Naturally those queries can be quite expensive, so by default none of the network plugins are used, the query is only executed on the machine which requested it.

If you don't have a storage plugin, or if your storage plugin doesn't support streaming, you'll get an error.

Config

Mytosis is designed to be highly extensible through plugins. You can use it with any storage backend, sync it over the network using any connection prototcol (such as websockets, http, webrtc, or a mix of all three), intercept and transform reads and writes, filter incoming reads or writes, and extend the core API.

These plugins are all contained in the config, and must be defined when the database is instantiated.

You can have more than one config though, as the database function accepts several and will merge them all together. This allows you to create presets, or groupings of plugins and share them as a single unit.

const db = database(config1, config2, config3)

Note: Currently, there are no published plugins.

Hooks

Used to intercept reads & writes. For example, you could use hooks to:

• Reject writes of the wrong data type
• Listen for write events and update a database visualization
• Keep track of how often a value is read for an LRU policy
• Transform values to a different format (like a Date object)

A hook is a function which takes a read or write action and returns the transformed action.

const writeHook = (action) => action

When you write a value, mytosis generates an action object which represents the write. It contains details like what network & storage plugins should be called, what the update graph is, what the state will look like when the write has finished, and any merge deltas after completion.

Hooks can change every bit of it.

For example, here's a hook which adds a prefix to every read.

const readHook = (readAction) => ({
  ...readAction,

  keys: readAction.keys.map((key) => `my-prefix/${key}`),
})

const db = database({
  hooks: {
    before: {
      read: {
        nodes: readHook,
      },
    },
  },
})

Hooks can be asynchronous. If a promise is returned, the entire pipeline is put on pause until it resolves.

// Delay all writes by one second.
const writeHook = async (writeAction) => {
  await Promise.delay(1000)

  return writeAction
}

As a best practice, never mutate the action. Return a new action instead.

These are all the hooks you can register:

const hook = (action) => action

database({
  hooks: {
    before: {
      write: hook,
      read: {
        nodes: hook,
        field: hook,
      },
    },

    after: {
      write: hook,
      read: {
        node: hook,
        field: hook,
      },
    },
  }
})

Note: You may be tempted to implement security using hooks. Don't. Do it in the router instead.

Storage

Used to save data to a persistent cache and read it back later. If a plugin is provided, the first read will attempt to pull from storage, and its return value will be cached in memory.

database({
  storage: new StoragePlugin(),
})

Writing a storage plugin will look a bit like this...

class StoragePlugin {
  async read (action) {} // Read a value
  async write (action) {} // Write a set of values
  async remove (action) {} // Delete a value
}

Note: Mytosis may require .query() and .list() methods in the future.

Network

Used to send updates and request data from remote sources.

database({
  network: new NetworkPlugin(),
})

By default, Mytosis will not use your network plugins. Instead, it interfaces with another plugin type, called the router. The router is responsible for answering requests and sending out data.

Network plugins are objects with a .send() method and a message event stream. Each connection declares its connection type and its unique identifier.

Here's an example of a network plugin...

import { Stream } from 'mytosis'

class NetworkPlugin {
  type = 'websocket'
  id = uuid()

  send (message) {}

  messages = new Stream(emit => {
    myNetworkInterface.on('message', emit)
  })
}

Mytosis may specify connection and disconnection event streams in the future, as well as offline and ephemeral flags.

Router

The router is responsible for sending read requests, pushing out updates, and handling requests from others.

At its core, a router looks like this:

const router = (db, config) => {
  config.network.messages.forEach((message) => {
    console.log('Incoming message:', message)
  })

  return {
    async push ({ network, update }) {
      network.send({ /* your update */ })
    }

    async pull ({ network, key }) {
      network.send({ /* your request */ })
    }
  }
}

database({ router })

push() is invoked for writes, while pull() is called for reads. You're given a group of network connections and complete creative freedom. Whatever the push/pull resolve value is will be given to the user (after merging with whatever storage returns).

Security

Mytosis doesn't have a stance on read or write permissions. Your router is responsible for how you answer unauthorized requests. If a message comes in asking for data they're not authorized to read, pull() (or push()) should reject.

Ultimately, the security mechanism is completely up to you.

Note: Mytosis may publish a routing framework in the near future designed to address permissions and security.

Roadmap

• Better offline editing
• Query support (plugin integration)
• Ability to delete
• Low-level streaming API