fpgasystems / Coyote

OS for FPGAs

Framework providing operating system abstractions and a range of shared networking (RDMA, TCP/IP) and memory services to common modern heterogeneous platforms.

Some of Coyote's features:

• Multiple isolated virtualized vFPGA regions
• Dynamic reconfiguration
• RTL and HLS user logic coding support
• Unified host and FPGA memory with striping across virtualized DRAM channels
• TCP/IP service
• RDMA service
• HBM support
• Runtime scheduler for different host user processes

Prerequisites

Full Vivado/Vitis suite is needed to build the hardware side of things. Hardware server will be enough for deployment only scenarios. Coyote runs with Vivado 2022.1. Previous versions can be used at one's own peril.

Following AMD platforms are supported: vcu118, Alveo u50, Alveo u55c, Alveo u200, Alveo u250 and Alveo u280. Coyote is currently being developed on the HACC cluster at ETH Zurich. For more information and possible external access check out the following link: https://systems.ethz.ch/research/data-processing-on-modern-hardware/hacc.html

CMake is used for project creation. Additionally Jinja2 template engine for Python is used for some of the code generation. The API is writen in C++, 17 should suffice (for now).

System HW

The following picture shows the high level overview of Coyote's hardware architecture.

System SW

Coyote contains the following software layers, each adding higher level of abstraction and parallelisation potential:

1. cService - Coyote daemon, targets a single vFPGA. Library of loadable functions and scheduler for submitted user tasks.
2. cProc - Coyote process, targets a single vFPGA. Multiple cProc objects can run within a single vFPGA.
3. cThread - Coyote thread, running on top of cProc. Allows the exploration of task level parallelisation.
4. cTask - Coyote task, arbitrary user variadic function with arbitrary parameters executed by cThreads.

Init

$ git clone https://github.com/fpgasystems/Coyote.git

Build HW

Create a build directory :

$ cd hw && mkdir build && cd build

Enter a valid system configuration :

$ cmake .. -DFDEV_NAME=u250 <params...>

Following configuration options are provided:

Create the shell and the project :

$ make shell

The project is created once the above command completes. Arbitrary user logic can then be inserted. If any of the existing examples are chosen, nothing needs to be done at this step.

User logic wrappers can be found under build project directory in the hdl/config_X where X represents the chosen PR configuration. Both HLS and HDL wrappers are placed in the same directories.

If multiple PR configurations are present it is advisable to put the most complex configuration in the initial one (config_0). Additional configurations can always be created with make dynamic. Explicit floorplanning should be done manually after synthesis (providing default floorplanning generally makes little sense).

Project can always be managed from Vivado GUI, for those more experienced with FPGA design flows.

When the user design is ready, compilation can be started with the following command :

$ make compile

Once the compilation finishes the initial bitstream with the static region can be loaded to the FPGA via JTAG. All compiled bitstreams, including partial ones, can be found in the build directory under bitstreams.

User logic can be simulated by creating the testbench project :

$ make sim

The logic integration, stimulus generation and scoreboard checking should be adapted for each individual DUT.

Driver

After the bitstream has been loaded, the driver can be compiled on the target host machine :

$ cd driver && make

Insert the driver into the kernel (don't forget privileges) :

$ insmod fpga_drv.ko

Restart of the machine might be necessary after this step if the util/hot_reset.sh script is not working (as is usually the case).

Build SW

Available sw projects (as well as any other) can be built with the following commands :

$ cd sw && mkdir build && cd build
$ cmake ../ -DTARGET_DIR=<example_path>
$ make

Publication

If you use Coyote, cite us :

@inproceedings{coyote,
    author = {Dario Korolija and Timothy Roscoe and Gustavo Alonso},
    title = {Do {OS} abstractions make sense on FPGAs?},
    booktitle = {14th {USENIX} Symposium on Operating Systems Design and Implementation ({OSDI} 20)},
    year = {2020},
    pages = {991--1010},
    url = {https://www.usenix.org/conference/osdi20/presentation/roscoe},
    publisher = {{USENIX} Association}
}

Repository structure

├── driver
│   └── eci
│   └── pci
├── hw
│   └── ext/network
│   └── ext/eci
│   └── hdl
│   └── ip
│   └── scripts
│   └── sim
├── sw
│   └── examples
│   └── include
│   └── src
├── util
├── img

Additional requirements

If networking services are used, to generate the design you will need a valid UltraScale+ Integrated 100G Ethernet Subsystem license set up in Vivado/Vitis.