how-to-optimize-gemm

English | 简体中文

row-major matmul optimization

All backends and corresponding tutorials

Build and run

Usage is similar for all backends:

1. Open the backend directory to be used, and change the OLD and NEW of makefile to the same implementation for the first run, for example

$ cd aarch64
$ cat makefile
OLD    := MMult_4x4_10
NEW   := MMult_4x4_10
..

2. makefilewill compile and run the implementation whichNEWpoint at, and copyoutput_MMult_4x4_10.mtooutput_new.m`

$ make run
$ cat output_new.m

3. It may not be intuitive to look at the numbers directly, so draw a line chart

$ python3 -m pip install -r ../requirements.txt
$ python3 plot.py

Differences between backends

Specific to each hardware, there are subtle differences:

• NEW may choose a different name
• vulkan/int4 needs prerequisitions

1. armv7 and aarch64

A. Prepare armv7/aarch64 linux development environment, Raspberry Pi/rk3399/aws arm server are all fine.

B. By default ARCH := native, build and run directly

$ cd armv8 && make run

2. aarch64 int8

chgemm is an int8 gemm library.

• blue line is chgemm implementation
• orange line is aarch64 fp32 peak

Compared to the code in this tutorial, the differences are:

1. Dealing with the boundary problem, unlike the tutorial where only multiples of 4 are considered;
2. Int8 reaches a maximum of 18.6 gflops (relative to the theoretical limit of fp32 is only 14.3 on RK3399, gemmlowp is about 12-14gflops);
3. Based on symmetric quantization, input value range must be in [-127, +127], and -128 cannot appear;
4. Built-in small example about how to integrate into android studio

chgemm has been merged into ncnn INT8 convolution implementation.

3. x86 original

flame referenced by x86 is the original implementation, with some differences from this repo:

1. The original is column-major x86 SSE version
2. Both are tutorials, and the MMult_4x4_17.c written now can reach 70% of the armv8.1 CPU peak
3. The boundary problem is not dealt with now, only the case where MNK is a multiple of 4 is considered; sub_kernel also only writes the simplest kind of assembly. Practical needs a simple adjustment;
4. In terms of drawing, octave was discarded (it is too troublesome to configure the environment once for embedded devices), and python was used instead.

4. CUDA

This version is faster than NVIDIA cuBLAS

• green line is MMult_cuda_12 without tensorcore
• blue line is cuBLAS without tensorcore

1. Need to install cuda driver and nvcc by yourself
2. CPU OpenBLAS is required to do the baseline to verify that the values are correct

5. Vulkan

1. vulkan build depends on kompute API packaging, see vulkan build documentation for details

2. More about how to learn compute shader

6. CUDA int4

WIP

Introduce some tools

• megpeak: For measuring hardware limit performance, support arm/x86/OCL..
• perf: Available in linux system tools, for system-level performance analysis and disassembly
• YHs_Sample: dalao 's implementation