Firecracker Task Driver
nomad task driver for creating Firecracker micro-vms.
- Website: https://www.nomadproject.io
Requirements
- Nomad 0.9+
- Go 1.11
- Linux 4.14+ Firecracker currently supports physical Linux x86_64 and aarch64 hosts, running kernel version 4.14 or later. However, the aarch64 support is not feature complete (alpha stage).
- KVM enabled in your Linux kernel, and you have read/write access to /dev/kvm.
- tun module loaded
- ip6tables package
- Container networking plugins (Firecracker branch)
- tc-redirect-tap
- Firecracker binary
Note: The last version of firecracker that works with this nomad plugin is : 0.25.2, more work is needed to make it work with latest releases.
Installation
Install(and compile) the firecracker-task-driver binary and put it in plugin_dir and then add a plugin "firecracker-task-driver" {} line in your nomad config file.
go get github.com/cneira/firecracker-task-driver
cp $GOPATH/bin/firecracker-task-driver YOURPLUGINDIRThen in your nomad config file, set
plugin "firecracker-task-driver" {}In developer/test mode(nomad agent -dev) , plugin_dir is unset it seems, so you will need to mkdir plugins and then copy the firecracker-task-driver binary to plugins and add a plugins_dir = "path/to/plugins" to the above config file.
then you can run it like:
nomad agent -dev -config nomad.config
For more details see the nomad docs.
Container network configuration
- Build cni plugins (Firecracker branch) and tc-redirect-tap and copy them /opt/cni/bin
- Create a network configuration to be used by micro-vms on /etc/cni/conf.d/, for example: default.conflist.
{
"name": "default",
"cniVersion": "0.4.0",
"plugins": [
{
"type": "ptp",
"ipMasq": true,
"ipam": {
"type": "host-local",
"subnet": "192.168.127.0/24",
"resolvConf": "/etc/resolv.conf"
}
},
{
"type": "firewall"
},
{
"type": "tc-redirect-tap"
}
]
}Example : exposing port 27960 on micro-vm
{
"name": "microvms2",
"cniVersion": "0.4.0",
"plugins": [
{
"type": "ptp",
"ipMasq": true,
"ipam": {
"type": "host-local",
"subnet": "192.168.127.0/24",
"resolvConf": "/etc/resolv.conf"
}
},
{
"type": "firewall"
},
{
"type": "portmap",
"capabilities": {"portMappings": true},
"runTimeConfig": {
"portMappings":
[ { "hostPort": 27960, "containerPort": 27960, "protocol": "udp" }
] }
},
{
"type": "tc-redirect-tap"
}
]
}In this example with outside world connectivity for your vms. The name of this network is default and this name is the parameter used in Network on the task driver job spec. Also the filename must match the name of the network, and the suffix .conflist.
Creating a rootfs and kernel image for firecracker
We need to an ext4 root filesystem to use as disk and an uncompressed vmlinux image, the process on how to generate them is described here.
Using ZFS zvols to create a rootfs for microvms
Leveraging ZFS zvols to expose rootfs to firecracker is really simple, and zfs has a lot of benefits.
First download a template image, for example from OpenVZCentos7
Now create a ZVOL to host this tarball
$ zfs create -V 1G zpool/centos7vm
$ mkfs.ext4 /dev/zvol/zpool/centos7vm
$ mount -t ext4 /dev/zvol/zpool/centos7vm /mnt
$ tar xfvz centos-7-x86_64-minimal.tar.gz -C /mnt
$ zfs snapshot zpool/centos7vm@final Now just use your new zvol as your BootDisk For example:
job "example3" {
datacenters = ["dc1"]
type = "service"
group "test" {
restart {
attempts = 0
mode = "fail"
}
task "test01" {
driver = "firecracker-task-driver"
config {
Vcpus = 1
Mem = 128
KernelImage= "/home/cneira/kernel-images/vmlinux.bin"
BootDisk = "/dev/zvol/vms/centos7vm"
Network = "default"
}
}
}
}
Firecracker task driver options
KernelImage (not required, default: vmlinux )
- kernel image to be used on the micro-vm, if this option is omitted it expects a vmlinux file in the allocation dir.
BootOptions (not required, default: "ro console=ttyS0 reboot=k panic=1 pci=off nomodules")
- Kernel command line.
BootDisk (not required, default: rootfs.ext4)
- ext4 rootfs to use, if this is omitted it expects a rootfs called rootfs.ext4 in the allocation dir.
Disks (not required)
- Additional disks to add to the micro-vm, must use the suffix :ro or :rw, can be specified multiple times.
Network (not required)
- Network name if using CNI
Vcpus (not required, default: 1)
- Number of cpus to assign to micro-vm.
Cputype (not required)
- The CPU Template defines a set of flags to be disabled from the microvm so that the features exposed to the guest are the same as in the selected instance type. templates available are C3 or T2.
Mem (not required, default: 512)
- Amount of memory in Megabytes to assign to micro-vm.
Firecracker (not required, default: "/usr/bin/firecracker")
- Location of the firecracker binary, the option could be omitted if the environment variable FIRECRACKER_BIN is set.
Log (not required)
- Where to write logs from micro-vm.
DisableHt (not required, default: false)
- Disable CPU Hyperthreading.
When the microvm starts a file will be created in /tmp/ with the following name -, for example : /tmp/test01-785f9472-52a7-3dbf-8305-d482b1f7dc6f will contain the following info :
{
"AllocId": "590983f4-499a-380f-420e-e5be4d5f46d9",
"Ip": "192.168.127.62/24",
"Serial": "/dev/pts/3",
"Pid": "237216",
"Vnic": "veth05fb4547vm"
}
- AllocId (given by nomad)
- Ip (Ip address assigned by cni configuration)
- Serial (tty where a serial console is setup for the vm)
- Pid ( Pid for the firecracker process that started the vm)
- Vnic (virtual interface on the host linked to the vm)
Examples:
Omitting KernelImage and BootDisk
Don't specifying KernelImage and BootDisk it will default to rootfs.ext4 and vmlinux in the allocation directory.
job "example" {
datacenters = ["dc1"]
type = "service"
group "test" {
restart {
attempts = 0
mode = "fail"
}
task "test01" {
artifact {
source = "https://firecracker-kernels.s3-sa-east-1.amazonaws.com/vmlinux-5.4.0-rc5.tar.gz"
destination = "."
}
artifact {
source = "https://firecracker-rootfs.s3-sa-east-1.amazonaws.com/ubuntu16.04.rootfs.tar.gz"
destination = "."
}
driver = "firecracker-task-driver"
config {
Vcpus = 1
Mem = 128
Network = "default"
}
}
}
}CNI network configuration
job "cni-network-configuration-example" {
datacenters = ["dc1"]
type = "service"
group "test" {
restart {
attempts = 0
mode = "fail"
}
task "test01" {
driver = "firecracker-task-driver"
config {
KernelImage = "/home/build/firecracker/hello-vmlinux.bin"
Firecracker = "/home/build/firecracker/firecracker"
Vcpus = 1
Mem = 128
BootDisk = "/home/build/firecracker/hello-rootfs.ext4"
Network = "fcnet"
}
}
}
}Additional Disks configuration
job "neverwinter" {
datacenters = ["dc1"]
type = "service"
task "nwn-server" {
driver = "firecracker-task-driver"
config {
Vcpus = 1
KernelImage = "/home/cneira/Development/vmlinuxs/vmlinux"
BootDisk= "/home/cneira/Development/rootfs/ubuntu/18.04/nwnrootfs.ext4"
Disks = [ "/home/cneira/Development/disks/disk0.ext4:rw" ]
Mem = 1000
Network = "default"
}
}
}Accessing the microvm using serial console
The firecracker-task-driver exposes the serial console as this option is handy to troubleshoot network issues. Each microvm generates a state file on the /tmp/ directory, named using the job name + allocation id. For example:
-rw-r--r--. 1 root root 152 May 12 14:07 /tmp/test01-590983f4-499a-380f-420e-e5be4d5f46d9The contents of the state file should be like the following:
{
"AllocId": "590983f4-499a-380f-420e-e5be4d5f46d9",
"Ip": "192.168.127.62/24",
"Serial": "/dev/pts/3",
"Pid": "237216",
"Vnic": "veth05fb4547vm"
}Using the serial now we know which serial port is expose and it's a matter of connect to it. You could use SCREEN(1) to connect to the serial console.
$ sudo screen /dev/pts/3
Started Update UTMP about System Runlevel Changes.
CentOS Linux 7 (Core)
Kernel 4.14.225 on an x86_64
192 login:
Demo
Support
It's also possible to support the project on Patreon
I work on this project on my free time and my country is not on the list available for github sponsors so any help for me continue working on this is appreciated.