Android Camera Api2 / CameraX YUV to ByteBuffer
OUTDATED!
7 years passed since CameraApi 2 was introduced in Android 5. So Google decided that it's time to make convenient way to get RGB images from it.
ImageAnalysis from CaemraX now supports setOutputImageFormat(ImageAnalysis.OUTPUT_IMAGE_FORMAT_RGBA_8888) developer.android.com
Hovewer if you do not use CameraX for some reason - welcome!
Motivation:
When you're attempting to get an Image from ImageReader or ImageAnalysis.Analyzer you actually get 3 separate ByteBuffers which you can't pass to further processing. You have to merge them but that also is not easy because they are full of row and pixel strides.
Solution:
This library carefully merges 3 buffers into one with respect to all strides. As a result, you receive YUV type (NV21 or I420) and ByteBuffer to pass into OpenCV or a neural network framework.
The solution was tested with CameraX, CameraApi 2, and MediaCodec (for video files).
The whole library is a single file, you can just copy Yuv.java into your project.
Usage
private var reuseBuffer: ByteBuffer? = null
fun convert(image: ImageProxy): Pair<Bitmap, Long> {
// val converted = Yuv.toBuffer(image)
// OR pass existing DirectBuffer for reuse
val converted = Yuv.toBuffer(image, reuseBuffer)
reuseBuffer = converted.buffer
val format = when (converted.type) {
ImageFormat.YUV_420_888 -> Imgproc.COLOR_YUV2RGB_I420
ImageFormat.NV21 -> Imgproc.COLOR_YUV2RGB_NV21
else -> throw IllegalArgumentException()
}
// process converted.buffer ByteBuffer with one of converters
}About android image formats
In android documentation ImageFormat.YUV_420_888 is stated as 'generic YCbCr format', so the actual format depends on u and v order and are these planes interleaved or not (pixelStride=1 or 2).
Most of the time you will get NV21 format (just as in Camera1 api).
And you even may start wondering is there a phone with I420 format (u and v pixelStride==1) – Yes! There is one. Blackview BV6000S Android 7.0. Screenshot. It looks like it's the only one. Other models of Blackview I was able to find in local sellers used NV21.
Converters
- RenderScriptConverter.kt - built-in, no additional libraries required. Uses Bitmap for rotation.
- OpenCVRoteterter.kt - preform rotation on yuv image and then converts color. The fastest in total time.
- OpenCVConverter.kt - the fastest color conversion without rotation. Rotation of a rgb image is slightly harder.
- MNNConverter.kt - if your goal is futher processing with neural network. Conversion and rotation performed in single operation.
PS: If your goal is to get Mat you may consider this method from OpenCV.
Benchmark
Snapdragon 855 (Xiaomi Mi 9T Pro). Image resolution 480x640.
| OpenCV (YUV rotate) | OpenCV (RGB rotate) | RenderScript | MNN | |
|---|---|---|---|---|
| color | ~1ms | ~1.6ms | ~2.2ms | ~21ms |
| rotate | ~3.5ms | ~2.8ms | ~16.3ms | NA (included in color) |
Alternatives
- (For OpenCV users) Copy private method from OpenCV camera implementation: JavaCamera2View, Mat rgba().
- Capture image from camera directly to RenderScript Allocation.getSurface();
- RenderScript implementation in android/camera-samples repo.
- Switch back to CameraApi 1 (some trade-offs);
- Manipulate pixels manually as it has done in TFLite demo ImageUtils. However, even with C++ implementation it's ridiculously slow. ~50ms for image about 1280x720 on Snapdragon 855;
TODO
- make the library;
- write unit tests;
- add RenderScript example;
- add OpenCV example;
- add MNN example;
- publish to GooglePlay;
- publish to mavenCentral();
- add TFLite example.