nestybox / Sysbox

Introduction

Sysbox is an open-source container runtime (aka runc), originally developed by Nestybox, that enables Docker containers to act as virtual servers capable of running software such as Systemd, Docker, and Kubernetes in them, easily and with proper isolation. This allows you to use containers in new ways, and provides a faster, more efficient, and more portable alternative to virtual machines in many scenarios.

Prior to Sysbox, running such software in a container required you to create complex images, custom entrypoints, special volume mounts, and use very unsecure privileged containers. With Sysbox, this is as simple as:

$ docker run --runtime=sysbox-runc -it some-image

In this container, you can now run Systemd, Docker, Kubernetes, etc., just like you would on a physical host or virtual machine. You can launch inner containers (and even inner privileged containers), knowing that the outer container is strongly isolated from the underlying host (via the Linux user-namespace as well as other features setup by Sysbox). No more complex docker images or docker run commands, and no need for unsecure privileged containers.

In order to do this, Sysbox uses many OS-virtualization features of the Linux kernel and complements these with OS-virtualization techniques implemented in user-space. These include using all Linux namespaces (in particular the user-namespace), partial virtualization of procfs and sysfs, selective syscall trapping, and more. Due to this, Sysbox requires a fairly recent Linux kernel (see the supported distros below).

Sysbox was forked from the excellent OCI runc in early 2019, and has undergone significant changes since then. It's written in Go, and it is currently composed of three components: sysbox-runc, sysbox-fs, and sysbox-mgr. More on Sysbox's design can be found in the Sysbox user guide.

Sysbox sits below OCI-compatible container managers such as Docker / containerd, allowing you to use these well known tools to deploy the containers. No need to learn new tools.

The complete list of features is here.

Contents

• License
• Audience
• System Containers
• Supported Distros
• Host Requirements
• Installing Sysbox
• Using Sysbox
• Docker Default Runtime Config
• Documentation
• Sysbox Features
• Integration with Container Managers & Orchestrators
• Sysbox does not use hardware virtualization
• Comparison to related technologies
• Contributing
• Filing Issues
• Troubleshooting & Support
• Roadmap
• Relationship to Nestybox
• Uninstallation
• Contact
• Thank You

License

Sysbox is an open-source project, licensed under the Apache License, Version 2.0. See the LICENSE file for details.

Audience

The Sysbox project is intended for anyone looking to experiment, invent, learn, and build systems using system containers. It's cutting-edge OS virtualization, and contributions are welcomed.

The Sysbox project is not meant for people looking for a commercially supported solution. For such a solution, use the Sysbox Enterprise Edition (Sysbox-EE).

Sysbox-EE uses Sysbox at its core, but complements it with enterprise-level features for improved security and efficiency as well as Nestybox support (see next section).

For info on where to find and download Sysbox-EE, refer to the Nestybox website.

Sysbox Features

The table below summarizes the key features of the Sysbox container runtime. It also provides a comparison between the Sysbox Community Edition (i.e., this repo) and the Sysbox Enterprise Edition (Sysbox EE).

More on the features below.

If you have questions, you can reach us here.

System Containers

We call the containers deployed by Sysbox system containers, to highlight the fact that they can run not just micro-services (as regular containers do), but also system software such as Docker, Kubernetes, Systemd, inner containers, etc.

More on system containers here.

Supported Distros

Sysbox relies on functionality available only in relatively recent Linux kernel releases.

See the distro compatibility doc for information about the supported Linux distributions and the required kernel releases.

We plan to add support for more distros in the near future.

Installing Sysbox

1. Download the latest Sysbox community-edition package from the release page:

$ wget https://github.com/nestybox/sysbox/releases/download/v0.2.1/sysbox_0.2.1-0.ubuntu-focal_amd64.deb

2. Verify that the checksum of the downloaded file fully matches the expected/published one. For example:

$ sha256sum sysbox_0.2.1-0.ubuntu-focal_amd64.deb
126e4963755cdca440579d81409b3f4a6d6ef4c6c2d4cf435052a13468e2c250  sysbox_0.2.1-0.ubuntu-focal_amd64.deb

3. Stop and eliminate all running Docker containers. Refer to the detailed installation process for information on how to avoid impacting existing containers.

$ docker rm $(docker ps -a -q) -f

... if an error is returned, it simply indicates that no existing containers were found.

4. Install the Sysbox package and follow the installer instructions:

$ sudo apt-get install ./sysbox_0.2.1-0.ubuntu-focal_amd64.deb -y

More information on the installation process can be found here. If you run into problems during installation, see the troubleshooting doc.

Alternatively, you can build Sysbox binaries yourself. It's a very simple process as we have a 'make' target that builds Sysbox inside a Docker container, and another 'make' target that installs Sysbox on your host. See the Sysbox developer's guide for more on this.

Using Sysbox

Once Sysbox is installed, you use it as follows:

$ docker run --runtime=sysbox-runc --rm -it --hostname my_cont debian:latest
[email protected]_cont:/#

This launches a system container. It looks very much like a regular container, but it's different under the hood. Within it can run application or system-level software (systemd, dockerd, K8s) inside just as you would in a VM. The container is strongly isolated via the Linux user-namespace.

The Sysbox Quickstart Guide has many usage examples. You should start there to get familiarized with the use cases enabled by Sysbox.

Note that if you omit the --runtime option, Docker will use its default runc runtime to launch regular containers (rather than system containers). It's perfectly fine to run system containers launched with Docker + Sysbox alongside regular Docker containers; they won't conflict and can co-exist side-by-side.

Docker Default Runtime Config

Optionally, you can make Sysbox the default runtime for Docker. To do this, add the default-runtime config to /etc/docker/daemon.json. It should look similar to this:

{
  "default-runtime": "sysbox-runc",
  "runtimes": {
     "sysbox-runc": {
        "path": "/usr/local/sbin/sysbox-runc"
     }
  }
}

Then restart Docker (e.g., sudo systemctl restart docker). With this setup, you can omit the --runtime=sysbox-runc flag when using docker run to create containers with Sysbox.

Documentation

We strive to provide good documentation; it's a key component of the Sysbox project.

We have several documents to help you get started and get the best out of Sysbox.

• Sysbox Quick Start Guide

  • Provides many examples for using system containers. New users should start here.

• Sysbox User Guide

  • Provides more detailed information on Sysbox features and design.

• Sysbox Distro Compatibility Doc

  • Distro compatibility requirements.

• Nestybox blog site

  • Articles on using Sysbox to solve real-life problems.

Sysbox Feature Description

OCI-based

• Integrates with OCI compatible container managers (e.g., Docker).

• Currently we only support Docker/containerd, but plan to add support for more managers / orchestrators (e.g., K8s) soon.

• Sysbox is ~90% OCI-compatible. See here for more on this.

Systemd-in-Docker [ +v0.1.2 ]

• Run Systemd inside a Docker container easily, without complex container configurations.

• Enables you to containerize apps that rely on Systemd (e.g., legacy apps).

Docker-in-Docker

• Run Docker inside a container easily and without unsecure privileged containers.

• Full isolation between the Docker inside the container and the Docker on the host.

Kubernetes-in-Docker [ +v0.2.0 ]

• Deploy Kubernetes (K8s) inside containers with proper isolation (no privileged containers), using simple Docker images and Docker run commands (no need for custom Docker images with tricky entrypoints).

• Deploy directly with docker run commands for full flexibility, or using a higher level tool (e.g., such as kindbox).

Strong container isolation

• Root user in the system container maps to a fully unprivileged user on the host.

• The procfs and sysfs exposed in the container are fully namespaced.

• Programs running inside the system container (e.g., Docker, Kubernetes, etc) are limited to using the resources given to the system container itself.

• The system container's initial mounts are immutable (can't be changed from inside the container, even though processes in the container can setup other mounts).

• See the security section of the User Guide for details.

Fast & Efficient

• Sysbox uses host resources optimally and starts containers in a few seconds.

Inner Container Image Preloading

• You can create a system container image that includes inner container images using a simple Dockerfile or via Docker commit. See here for more.

Please see our Roadmap for a list of features we are working on.

Integration with Container Managers & Orchestrators

Though Sysbox is OCI-based (and thus compatible with OCI container managers), it's currently only tested with Docker / containerd.

In particular, we don't yet support using Kubernetes to deploy system containers with Sysbox (though we plan to).

Performance

Sysbox is fast and efficient, as described in this Nestybox blog post.

The system containers created by Sysbox have similar performance to those created by Docker's default runtime (the OCI runc).

Even containers deployed inside the system containers have excellent performance, thought there is a slight overhead for network IO (as expected since packets emitted by inner Docker containers go through an additional Docker network inside the system container).

Now, if you use Sysbox to deploy system containers that replace VMs, then the performance and efficiency gains are significant: you can deploy twice as many system containers as VMs on the same server and get the same performance, and do this with a fraction of the memory and storage consumption. The blog post referenced above has more on this.

Sysbox does not use hardware virtualization

Though the containers generated by Sysbox resemble virtual machines in some ways (e.g., you can run as root, run multiple services, and deploy Docker and K8s inside), Sysbox does not use hardware virtualization. It's purely an OS-virtualization technology meant to create containers that can run applications as well as system-level software, easily and securely.

This makes the containers created by Sysbox very fast, efficient, and portable. Isolation wise, it's fair to say that they provide stronger isolation than regular Docker containers (by virtue of using the Linux user-namespace), but weaker isolation than VMs (by sharing the Linux kernel among containers).

Comparison to related technologies

Sysbox is pretty unique: it is, to the best of our knowledge, the only OCI-based container runtime that allows Docker and similar tools to deploy "VM-like" containers capable of running systemd, Docker, K8s, etc., with ease and strong isolation from the underlying host (i.e., no privileged containers).

See this blog post for a high-level comparison between Sysbox and related technologies such as LXD, K8s.io KinD, Ignite, Kata Containers, rootless Docker, and more.

Contributing

We welcome contributions to Sysbox, whether they are small documentation changes, bug fixes, or feature additions. Please see the contribution guidelines and developer's guide for more info.

Filing Issues

We apologize for any problems in Sysbox or its documentation, and we appreciate people filing issues that help us improve the software.

Please see the contribution guidelines for info on how to report issues.

Troubleshooting & Support

Refer to the Troubleshooting document and to the issues for help.

Reach us at our slack channel for any questions.

Uninstallation

Prior to uninstalling Sysbox, make sure all system containers are removed. There is a simple shell script to do this here.

The method for uninstalling Sysbox depends on how you installed it.

1. If you used the packaged version from the Nestybox website, follow the uninstallation instructions in the associated documentation.

2. If you built it from source and installed it manually, follow these instructions to uninstall it.

Roadmap

The following is a list of features in the Sysbox roadmap.

We list these here so that our users can get a better idea of where we are going and can give us feedback on which of these they like best (or least).

Here is a short list; the Sysbox issue tracker has many more.

• Support for more Linux distros.

• Support for deploying system containers with Kubernetes.

• More improvements to procfs and sysfs virtualization.

• Continued improvements to container isolation.

• Exposing host devices inside system containers with proper permissions.

Relationship to Nestybox

Sysbox was initially developed by Nestybox, and Nestybox is the main sponsor of the Sysbox open-source project.

Having said this, we encourage participation from the community to help evolve and improve it, with the goal of increasing the use cases and benefits it enables. External maintainers and contributors are welcomed.

Nestybox uses Sysbox as the core of it's Sysbox enterprise product (Sysbox-EE), which consists of Sysbox plus proprietary features meant for enterprise use.

To ensure synergy between the Sysbox project and commercial entities such as Nestybox, we use the following criteria when considering adding functionality to Sysbox:

Any features that mainly benefit individual practitioners are made part of the Sysbox open-source project. Any features that mainly address enterprise-level needs are reserved for the Sysbox Enterprise Edition.

The Sysbox project maintainers will make this determination on a feature by feature basis, with total transparency. The goal is to create a balanced approach that enables the Sysbox open-source community to benefit and thrive while creating opportunities for commercial entities to create a healthy viable business around the technology.

Contact

Slack: Nestybox Slack Workspace

Email: [email protected]

We are available from Monday-Friday, 9am-5pm Pacific Time.

Thank You

We thank you very much for using and/or contributing to Sysbox. We hope you find it interesting and that it helps you use containers in new and more powerful ways.