docker-onload

docker-onload provides a Dockerfile which installs Solarflare's OpenOnload into various OS flavors. Find it on the Docker Hub: https://hub.docker.com/r/neomantra/onload/

See changes in the CHANGELOG.

The onload image is built with ONLOAD_WITHZF set, thus without support for TCPDirect (aka ZF). This is due to licensing restrictions.

Supported Docker Hub tags for image neomantra/onload and respective Dockerfile links

These unversioned image tags currently map to 7.1.3.202:

• centos7 (centos7/Dockerfile)
• centos8 (centos8/Dockerfile)
• stretch (stretch/Dockerfile)
• buster (buster/Dockerfile)
• trusty (trusty/Dockerfile)
• xenial (xenial/Dockerfile)
• bionic (bionic/Dockerfile)
• focal (focal/Dockerfile)
• jammy (jammy/Dockerfile)

The following versioned tags are available:

• For 7.1.3.202
  • 7.1.3.202-bionic
  • 7.1.3.202-bullseye
  • 7.1.3.202-buster
  • 7.1.3.202-centos7
  • 7.1.3.202-centos8
  • 7.1.3.202-focal
  • 7.1.3.202-jammy
  • 7.1.3.202-stretch
  • 7.1.3.202-trusty
  • 7.1.3.202-xenial
• For 7.1.2.141
  • 7.1.2.141-bionic
  • 7.1.2.141-bullseye
  • 7.1.2.141-buster
  • 7.1.2.141-centos7
  • 7.1.2.141-centos8
  • 7.1.2.141-focal
  • 7.1.2.141-jammy
  • 7.1.2.141-stretch
  • 7.1.2.141-trusty
  • 7.1.2.141-xenial

Launching Onload-enabled containers

Onload-enabled contaiers require exposing the host network and onload devices, so run like so:

docker run --net=host --device=/dev/onload --device=/dev/onload_epoll -it ONLOAD_ENABLED_IMAGE_ID [COMMAND] [ARG...]

The OpenOnload 201606 series also requires --device=/dev/onload_cplane. Using ef_vi or TCPDirect requires --device=/dev/sfc_char.

Here's a bash one-liner for extracting the OpenOnload version year: onload --version | awk 'NR == 1 {print substr($2, 1, 4)}'

Cavets

• Host networking must be used: --net=host

• The following devices must be exported: --device=/dev/onload --device=/dev/onload_epoll.

• The OpenOnload 201606 series also requires --device=/dev/onload_cplane.

• Using ef_vi or TCPDirect requires --device=/dev/sfc_char.

• The host's onload --version must be the same as the container's.

• Stack Sharing: If a container and the host must share an Onload stack, both should use EF_SHARE_WITH=-1 to avoid a current limitation in OpenOnload. Note this disables the stack sharing security feature.

• Due to a current limitation with OpenOnload, you should run with EF_USE_HUGE_PAGES=0 if you share Onload stacks.

• Some libraries, such as jemalloc need to invoke syscalls at startup. This can cause infinite loops because the OpenOnload acceleration also needs malloc (via dlsym); see jemalloc issues 443 and 1426. This can be alleviated by setting ONLOAD_DISABLE_SYSCALL_HOOK=1; note you will also need to set ONLOAD_USERSPACE_ID to match the unpatched driver version. NOTE: This may have been fixed in 7.0.0.176: "SF-122792-KI/bug62297: avoid hang at app startup with jemalloc".

• These OpenOnload builds default to using -march and -mtune based on the CPU-type of the build machine. This might not be optimial or runnable on your runtime platform. A future release will allow this to be specified as Docker build arguments.

TCPDirect

In OpenOnload 201606-u1, Solarflare introducted a new kernel-bypass networking API named TCPDirect.

To run TCPDirect applications in a container, an addition device must be exported: --device=/dev/sfc_char

TCPDirect is under a different license than OpenOnload; its binaries may not be distributed. Thus, the onload public image on Docker Hub does not have TCPDirect support.

You are free to build and deploy TCPDirect-enabled images yourself with the regular Dockerfiles. To do so, set the build argument ONLOAD_WITHZF to a non-empty string (the Dockerfile checks [ -z ${ONLOAD_WITHZF} ]). For example:

git clone https://github.com/neomantra/docker-onload.git
cd docker-onload
docker build --build-arg ONLOAD_WITHZF=1 -f xenial/Dockerfile -t neomantra/onload:201606-u1-xenial .

Image Building Helper Script

The Ruby script build_onload_image.rb helps generate command lines for building Onload images.

$ ./build_onload_image.rb --help
build_onload_image.rb [options]

ACTIONS
    --versions                show list of onload version name (use with -v to show all fields)
    --flavors                 show list of image flavors
    --gettag     <prefix>     show the autotag name of --autotag <prefix>

    --build                   show docker build command
    --execute                 execute docker build command

OPTIONS
    --flavor   -f  <flavor>   specify build <flavor> (required for --build or --execute)
    --onload   -o  <version>  specify onload <version> to build (default is 'latest')

    --url      -u <url>       Override URL for "packaged" versions.

    --tag      -t <tag>       tag image as <tag>
    --autotag  -a <prefix>    tag image as <prefix><version>-<flavor>[-nozf]. 
                              <prefix> is optional, but note without a <prefix> with colon,
                              the autotag will be a name not an image-name:tag

    --zf          <truthy>    build with TCPDirect (zf)  (or not, if optional <truthy> is '0' or 'false')

    --arg          <arg>      pass '--build-arg <arg>' to "docker build"

    --quiet    -q             build quietly (pass -q to "docker build")
    --no-cache                pass --no-cache to "docker build"

    --execute  -x             also execute the build line

    --push     -p             push the built image

    --verbose  -v             verbose output
    --help     -h             show this help

Example usage:

./build_onload_image.rb -o 7.1.1.75 --arg foo -f buster --zf --execute

There are also build_all_flavors.sh and build_all_images.sh.

Customized Image Building

Dockerfiles are provided for the following base systems, selecting the Dockerfile path with -f:

• CentOS 7 (centos7/Dockerfile)
• CentOS 8 (centos8/Dockerfile)
• Debian Stretch (stretch/Dockerfile)
• Debian Buster (buster/Dockerfile)
• Ubuntu Trusty (trusty/Dockerfile)
• Ubuntu Xenial (xenial/Dockerfile)
• Ubuntu Bionic (bionic/Dockerfile)
• Ubuntu Focal (focal/Dockerfile)
• Ubuntu Jammy (jammy/Dockerfile)

Each system folder has a Dockerfile.

The following are the available build-time options. They can be set using the --build-arg CLI argument, like so:

docker build --build-arg ONLOAD_VERSION="201509" --build-arg ONLOAD_MD5SUM="b093ea9f3a534c9c9fe9da6c2b6ccb7a" -f trusty/Dockerfile .

The Dockerfile downloads specific versions from openonload.org using the following ARG settings:

ONLOAD_PACKAGE_URL defaults to https://support-nic.xilinx.com/wp/onload?sd=SF-109585-LS-37&pe=SF-122921-DH-6

ONLOAD_LEGACY_URL defaults to https://www.openonload.org/download/openonload-${ONLOAD_VERSION}.tgz. If you want to build from a legacy (non-packaged) URL, you must also set ONLOAD_PACKAGE_URL to '' (empty string).

If you change the ONLOAD_VERSION, you must also change ONLOAD_MD5SUM to match. Note that Docker is only supported by OpenOnload since version 201502.

If you patch OpenOnload, you must specify ONLOAD_USERSPACE_ID to match the ID of the driver. It can be found in the build tree at ./build/gnu_x86_64/lib/transport/ip/uk_intf_ver.h. The following are driver interface IDs we have recorded:

Building from Onload Git repo

An initial bullseye Dockerfile is available for building from upstream GitHub, rather than pre-packaged releases:

docker build --build-arg ONLOAD_GIT_REF=master -f bullseye/Dockerfile .

License

Copyright (c) 2015-2022 Neomantra BV

Released under the MIT License, see LICENSE.txt