Android Camera Api2 / CameraX YUV to ByteBuffer

OUTDATED!

7 years passed since CameraApi 2 was introduced in Android 5. So Google decided that it's time to make convenient way to get RGB images from it.

ImageAnalysis from CaemraX now supports setOutputImageFormat(ImageAnalysis.OUTPUT_IMAGE_FORMAT_RGBA_8888) developer.android.com

Hovewer if you do not use CameraX for some reason - welcome!

Motivation:

When you're attempting to get an Image from ImageReader or ImageAnalysis.Analyzer you actually get 3 separate ByteBuffers which you can't pass to further processing. You have to merge them but that also is not easy because they are full of row and pixel strides.

Solution:

This library carefully merges 3 buffers into one with respect to all strides. As a result, you receive YUV type (NV21 or I420) and ByteBuffer to pass into OpenCV or a neural network framework.

The solution was tested with CameraX, CameraApi 2, and MediaCodec (for video files).

The whole library is a single file, you can just copy Yuv.java into your project.

Usage

private var reuseBuffer: ByteBuffer? = null

fun convert(image: ImageProxy): Pair<Bitmap, Long> {

    // val converted = Yuv.toBuffer(image)
    // OR pass existing DirectBuffer for reuse
    val converted = Yuv.toBuffer(image, reuseBuffer)
    reuseBuffer = converted.buffer

    val format = when (converted.type) {
        ImageFormat.YUV_420_888 -> Imgproc.COLOR_YUV2RGB_I420
        ImageFormat.NV21 -> Imgproc.COLOR_YUV2RGB_NV21
        else -> throw IllegalArgumentException()
    }

    // process converted.buffer ByteBuffer with one of converters
}

About android image formats

In android documentation ImageFormat.YUV_420_888 is stated as 'generic YCbCr format', so the actual format depends on u and v order and are these planes interleaved or not (pixelStride=1 or 2).

Most of the time you will get NV21 format (just as in Camera1 api).

And you even may start wondering is there a phone with I420 format (u and v pixelStride==1) – Yes! There is one. Blackview BV6000S Android 7.0. Screenshot. It looks like it's the only one. Other models of Blackview I was able to find in local sellers used NV21.

Converters

1. RenderScriptConverter.kt - built-in, no additional libraries required. Uses Bitmap for rotation.
2. OpenCVRoteterter.kt - preform rotation on yuv image and then converts color. The fastest in total time.
3. OpenCVConverter.kt - the fastest color conversion without rotation. Rotation of a rgb image is slightly harder.
4. MNNConverter.kt - if your goal is futher processing with neural network. Conversion and rotation performed in single operation.

PS: If your goal is to get Mat you may consider this method from OpenCV.

Benchmark

Snapdragon 855 (Xiaomi Mi 9T Pro). Image resolution 480x640.

Alternatives

1. (For OpenCV users) Copy private method from OpenCV camera implementation: JavaCamera2View, Mat rgba().
2. Capture image from camera directly to RenderScript Allocation.getSurface();
3. RenderScript implementation in android/camera-samples repo.
4. Switch back to CameraApi 1 (some trade-offs);
5. Manipulate pixels manually as it has done in TFLite demo ImageUtils. However, even with C++ implementation it's ridiculously slow. ~50ms for image about 1280x720 on Snapdragon 855;

TODO

• make the library;
• write unit tests;
• add RenderScript example;
• add OpenCV example;
• add MNN example;
• publish to GooglePlay;
• publish to mavenCentral();
• add TFLite example.