KBIO, added IO-Uring in tokio. Currently, kbio is just used in KuiBaDB, it is only a prototype and has not been tested fully.
-
Our experience using C++20 coroutines in Hologres tells us that Worker Steal is very necessary. So we build kbio based on tokio.
-
We run one IO-Uring instance per Worker. Before that, all Workers in Tokio used an Epoll instance, mainly because once a file descriptor is bound to an Epoll instance in Tokio, then the file descriptor will never switch to other Epoll instances, which will affect the effect of Worker Steal.
-
When a Worker needs to submit SQE, it will give priority to the Uring instance corresponding to the current Worker, and will also switch to other Uring instances when there is no free space in the current Uring instance.
-
KBIO is only a transitional project. We expect Tokio to support iouring in the end.
Usage
Most of them are fully compatible with tokio. See integrate kbio into memc for another example.
Benchmark
We use monoio-benchmark to do the benchmark, and use the example above to replace tokio-server/src/main.rs..
See integrate kbio into memc for another example.
[[email protected]]$ taskset -c 6-7 ./tokio-server --cores 6 7
[[email protected] 1k]$ ./client -c 0 1 2 3 -n 150
Running ping pong client.
Packet size: 1024
Connection count per core: 150; Global connection count: 600
QPS limit per core: 0; Global QPS limit: 0
Target: 127.0.0.1:40000
CPU slot: 0,1,2,3
1.000: NAdd: 138103; NSum: 138103; NAverage: 138093.969, LatencyAverage: 1831.181 us
2.000: NAdd: 179485; NSum: 317588; NAverage: 158781.531, LatencyAverage: 2535.254 us
3.000: NAdd: 187939; NSum: 505527; NAverage: 168495.281, LatencyAverage: 2779.394 us
4.000: NAdd: 187813; NSum: 693340; NAverage: 173320.438, LatencyAverage: 2891.172 us
5.000: NAdd: 187757; NSum: 881097; NAverage: 176204.344, LatencyAverage: 2959.997 us
6.001: NAdd: 186244; NSum: 1067341; NAverage: 177874.719, LatencyAverage: 3002.718 us
7.001: NAdd: 190705; NSum: 1258046; NAverage: 179705.047, LatencyAverage: 3023.978 us
8.001: NAdd: 192710; NSum: 1450756; NAverage: 181329.328, LatencyAverage: 3036.164 us
^C
[[email protected] 1k]$ ./monoio-server --cores 6 7
[[email protected] 1k]$ ./client -c 0 1 2 3 -n 150
Running ping pong client.
Packet size: 1024
Connection count per core: 150; Global connection count: 600
QPS limit per core: 0; Global QPS limit: 0
Target: 127.0.0.1:40000
CPU slot: 0,1,2,3
1.000: NAdd: 199135; NSum: 199135; NAverage: 199119.969, LatencyAverage: 2629.738 us
2.000: NAdd: 231118; NSum: 430253; NAverage: 215108.594, LatencyAverage: 2612.171 us
3.000: NAdd: 242953; NSum: 673206; NAverage: 224382.906, LatencyAverage: 2560.102 us
4.000: NAdd: 244618; NSum: 917824; NAverage: 229436.391, LatencyAverage: 2531.557 us
5.000: NAdd: 242257; NSum: 1160081; NAverage: 231997.594, LatencyAverage: 2519.936 us
6.000: NAdd: 247475; NSum: 1407556; NAverage: 234573.359, LatencyAverage: 2502.973 us
7.001: NAdd: 250774; NSum: 1658330; NAverage: 236884.219, LatencyAverage: 2486.379 us
8.001: NAdd: 240513; NSum: 1898843; NAverage: 237327.906, LatencyAverage: 2487.242 us
^C
[[email protected] 1k]$ ./glommio-server --cores 6 7
[[email protected] 1k]$ ./client -c 0 1 2 3 -n 150
Running ping pong client.
Packet size: 1024
Connection count per core: 150; Global connection count: 600
QPS limit per core: 0; Global QPS limit: 0
Target: 127.0.0.1:40000
CPU slot: 0,1,2,3
1.000: NAdd: 108135; NSum: 108135; NAverage: 108127.938, LatencyAverage: 4766.189 us
2.000: NAdd: 127786; NSum: 235921; NAverage: 117951.289, LatencyAverage: 4727.146 us
3.000: NAdd: 127098; NSum: 363019; NAverage: 120996.281, LatencyAverage: 4725.291 us
4.000: NAdd: 127328; NSum: 490347; NAverage: 122576.320, LatencyAverage: 4721.638 us
5.000: NAdd: 127299; NSum: 617646; NAverage: 123518.625, LatencyAverage: 4719.515 us
^C