eBPF CO-RE code example

This eBPF example:

• has a BCC compatible example BRANCH that can be found HERE with an example in C and other in Python

• has a very similar example using Go (libbpfgo) that can be found HERE with a good blog post describing it.

• has a compatible example version for really old (to eBPF) kernels - v4.15 - HERE

Intent of this repository was to create a simple eBPF code that would serve the reader to learn how to create your own eBPF C code using libbpf and making it portable among different kernel versions.

When doing this, I initially had ipsetaudit code in my mind - a tool capable of auditing ipset calls - and then the conntracker - a firewall rule sniffer using conntracker and eBPF - I created later on.

To be honest, I'm now using this as a base for all eBPF projects I start because of the Makefile example AND the very simple kprobe linking example. I'll probably add other BPF MAP examples to it, haven't done before because I was keeping it compatible to v4.15 kernels, but now I did another branch just for the older kernels.

eBPF libraries introduction

There are currently 2 main eBPF libraries:

1. BCC: compiles the eBPF code during code execution, loads the eBPF bytecode in kernel and makes userland app to get shared information from the eBPF code through BPF MAPS.

   • BCC project
   • BCC reference guide
   • BCC python developer

2. libbpf: compiles the eBPF (and user-land) code before the execution. The library is responsible for making eBPF bytecode symbol relocations BEFORE loading it into kernel, making the binary portable among different kernels. For that, libbpf needs kernel to support BTF (latest kernels support).

   • libbpf project
   • BPF portability and CO-RE
   • BCC to BPF conversion

This example is now ONLY USING libbpf. Check the BCC README if you want to explore it further.

3. Other resources you might be interested in:

   • Diving into BPF
   • Kernel eBPF features by version
   • Older kernels and eBPF global data

The file tree explained

Makefile

It is a Makefile you can use as a base file for your own project. It will compile libbpf statically from a local git submodule (./libbpf) and link it statically with your CO-RE libbpf binary file.

mine-static and mine-dynamic

The distributions do have libbpf library package nowadays but, before it gets stablized into 1.0, it is likely that they won't be able to catch up with the needed features of recent eBPF code.

Because of that, the Makefile creates 2 application binaries:

• mine-static is statically linked with libbpf from upstream
• mine-dynamic is linked with your distribution libbpf.so

Those 2 files are the same regarding functionality: they are eBPF (libbpf based) binary files that will execute together: the userland and kernel eBPF program.

Because the way libbpf works, you don't need to worry about compiling the eBPF object and loading it separately into the kernel because libbpf library will help your userland code to do it for you (check mine.c code).

mine.bpf.c

This is the eBPF source code. It is the source code that will be compiled to a eBPF architecture binary that will be just-in-time compiled by the kernel to be executed inside it.

Good thing about libbpf is that it allows you, together with the Makefile of this project - embbed this binary as a bytecode into your final userland application. It will then load the eBPF program(s) into the kernel and allow your userland code to talk to them through eBPF MAPS.

There are many different eBPF program types to be explored, and this example is only showing one. Some of the program types are:

• BPF_PROG_TYPE_SOCKET_FILTER: a network packet filter
• BPF_PROG_TYPE_KPROBE: determine whether a kprobe should fire or not
• BPF_PROG_TYPE_SCHED_CLS: a network traffic-control classifier
• BPF_PROG_TYPE_SCHED_ACT: a network traffic-control action
• BPF_PROG_TYPE_TRACEPOINT: determine whether a tracepoint should fire or not
• BPF_PROG_TYPE_XDP: a network packet filter run from the device-driver receive path
• BPF_PROG_TYPE_PERF_EVENT: determine whether a perf event handler should fire or not
• BPF_PROG_TYPE_CGROUP_SKB: a network packet filter for control groups
• BPF_PROG_TYPE_CGROUP_SOCK: a network packet filter for control groups that is allowed to modify socket options
• BPF_PROG_TYPE_SK_SKB: a network packet filter for forwarding packets between sockets
• BPF_PROG_CGROUP_DEVICE: determine if a device operation should be permitted or not

And the one being shown by this example is BPF_PROG_TYPE_KPROBE (or BPF_PROG_TYPE_PERF_EVENT, as libbpf will configure the perf event as being a kprobe).

We are adding a probe to the "tcp_connect" kernel function and linking this probe to the eBPF program called "tcp_connect". Whenever the kernel executes this function, our eBPF program will run and will call the "tcp_connect_enter()" inline function that will populate an eBPF MAP that is accessible by userland code.

mine.c

This code is responsible to get events out of the eBPF file descriptor, and read the eBPF MAP that was used as an IPC mechanism in between this userland code and the eBPF program running inside the eBPF VM of the kernel.

mine.h

This is header file that is shared among the 2 codes: the eBPF C code, in mine.bpf.c, and the userland C code, in mine.c.

If you are using libbpf only, then this file won't have much but the structs responsible to describe the BPF maps you want to share among kernel and userland.

vmlinux.h

This file is generated by doing:

tools/bpftool btf dump file /sys/kernel/btf/vmlinux format c > vmlinux.h

in a kernel supporting BTF.

Without entering into how binary relocation works, the CO-RE eBPF binaries need to be relocatable to run in different kernels. That would require that they were compiled together with the kernel or compiled and loaded as dynamic kernel modules.

In fact one may be able to compile a libbpf object file for a specific kernel version that won't support relocations, for example.

This example focuses in CO-RE only libbpf based programs. With that, it is mandatory that libbpf learns how to do the *symbol relocation BEFORE loading the eBPF program inside the kernel.

The Linux kernel engineers solved this by creating a BTF (BPF type format) file that you can obtain from /sys/kernel/btf/vmlinux. This file gives all needed information so libbpf can do binary relocations and allow your binary to run in multiple kernel versions (and not only the one you compiled your binary in).

I'm currently working in a way to load external BTF files for kernels that don't support it. Visit BTFHUB for more information.

patches/*

This is an on-going work of mine (and others) to make libbpf to work with legacy kprobes interface (when the link in between the perf event probe and the bpf program was made by different interface).

If you want to run this code in older kernels you might need those patches applied on top of libbpf/.

FAQ

I got the following question about this write up:

One thing I have not understand yet is where you mapped, in userspace, the eBPF map you created.

In my example I'm using a very simple way of eBPF communication because one of my original intents was to demonstrate how eBPF could be used in older kernels. The idea, from now on, here, is to explore new eBPF features, like ring buffers, and to show how they can be used... and also some other forms of sharing data between kernel and userland through the constant use of eBPF maps.

Nevertheless, the explanation for this question is this:

I'm creating a perf buffer by using the file descriptor of an eBPF map. The following line:

pb = perf_buffer__new(bpf_map__fd(mine->maps.events), 16 /* BUFFER PAGES */, &pb_opts);

Creates a new perf buffer and uses the following structure:

struct mine_bpf {
	struct bpf_object_skeleton *skeleton;
	struct bpf_object *obj;
	struct {
		struct bpf_map *events;
	} maps;
	struct {
		struct bpf_program *tcp_connect;
		struct bpf_program *tcp_connect_ret;
	} progs;
	struct {
		struct bpf_link *tcp_connect;
		struct bpf_link *tcp_connect_ret;
	} links;
};

That is automatically generated by the bpftool skel feature (as I'm demonstrating how to generate embedded CO-RE (portable) eBPF code together with a single compiled binary.

Make sure not to delete the .skel.h autogenerated file if you want to navigate through it and understand how libbpf skeleton works.

With all that said, the inifite loop with:

err = perf_buffer__poll(pb, 100)

will call the callback function:

pb_opts.sample_cb = handle_event;
pb_opts.lost_cb = handle_lost_events;

each time there is a "perf event". This perf event works like a 'software performance event sampling' (and uses the perf infrastructure within the kernel, thus the name).

One might be asking: but what about the macros to update or delete items from the eBPF map we have (events). That is managed by the libbpf '__perf_buffer__new()' function logic. In this example, the eBPF map only serves to exchange the pointer from which userland will mmap() data from kernel each time there is a perf event happening.