📸 A powerful, high-performance React Native Camera library.
VisionCamera V4 is finally here! 🥳🥳🥳🥳🚀💪
The most anticipated features are:
const frameProcessor = useSkiaFrameProcessor((frame) => {
'worklet'
// draw frame itself
frame.render()
// draw red rectangle into center of frame
const centerX = frame.width / 2
const centerY = frame.height / 2
const rect = Skia.XYWHRect(centerX, centerY, 150, 150)
const paint = Skia.Paint()
paint.setColor(Skia.Color('red'))
frame.drawRect(rect, paint)
}, [])
If you followed along the VisionCamera V2 -> V3 journey, you might remember that I switched from the (at-that-time-) immature CameraX library, to the low-level almost C-like Camera2 API. The motivation behind that was that CameraX was too limiting at that time, and I wanted to gain full control over the hardware sensor and output streams. While this worked great on my test devices, it seems like Camera2 wasn't implemented the same on all devices, and countless of weird issues arised, such as Samsung phones completely crashing, photos being rotated, frame processors running really slow on Huawei, some non-null Camera properties actually being null, etc. - the list goes on. While I was fighting my way through Android's low-level Camera2 API, CameraX started gaining more and more interesting features, and just recently they built a new API: stream sharing. With stream sharing, CameraX now supports all the features that VisionCamera has, and so I decided to fully rewrite the Android codebase to CameraX. I was pretty impressed with what I built in just a few months using Camera2 APIs, but in reality this is becoming unmaintainable as countless of phone manufacturers just don't really care about Camera2 APIs, and I can't be the person monkeypatching that all the time. Compared to VisionCamera V3, VisionCamera V4 is now much more stable, and brings a ton of new features.
useSkiaFrameProcessor
hook to create a drawable Frame ProcessoruseSkiaFrameProcessor
the caller can directly draw to the frame
as it extends a SkCanvas
Skia.*
APIsframe.render(shader)
useLocationPermission
hook to get location permissionenableLocation
prop to <Camera>
$VCEnableLocation
config option to Podfile
to disable location APIs (e.g. if Apple Review is not happy with unused location APIs in your code)photoQualityBalance
prop to <Camera>
to optimize the Camera pipeline for the given prioritization (speed
, balanced
(default) or quality
)preview
prop to <Camera>
to enable or disable the preview view/stream.androidPreviewViewType
prop to <Camera>
to specify which preview implementation to use (surface-view
is faster and supports HDR, texture-view
supports transparency, masks, rotations and clipping)takeSnapshot(..)
API to take instant snapshots of the video/preview streams. This can be used for intermediate results before takePhoto(..)
completes. 📸photo
, video
, frameProcessor
, preview
and codeScanner
can all be enabled at the same time!yuv
and rgb
pixel-formats
yuv
is more efficient (uses ~60% less memory than rgb
, and does not need conversion)rgb
requires explicit conversion from yuv
and will increase latencyVisionCamera/Core
: The core library which could be used in a native iOS/Android appVisionCamera/React
: React Native bindings for VisionCamera/Core
VisionCamera/FrameProcessors
: React Native bindings for synchronous onFrame
callbacks using react-native-worklets-corequalityPrioritizaion
parameter in takePhoto(..)
in favor of new photoQualityBalance
prop on <Camera>
enablePrecapture
parameter in takePhoto(..)
because CameraX now properly runs precapture sequences with device-quirks ironed outnative
pixel-format since yuv
is nativeenableGpuBuffers
since CameraX now managed such optimizationsconfigure { ... }
abort calls and give higher priority to newer updateSharedArray
and FrameProcessor::call
)onStarted
/onStopped
sometimes being called a bit too earlyvideoCodec
if the given video-codec is not supported (e.g. h265
-> h264
)takeSnapshot
Check out the VisionCamera V4 PR (https://github.com/mrousavy/react-native-vision-camera/issues/2623) for a full list of changes.
Huge shoutout to @wcandillon for helping me get the required changes into react-native-skia! We spent a lot of time debugging weird GPU issues and optimizing the code. These are the Skia PRs needed for the VisionCamera V4 Skia Frame Processors feature:
delegate
guard (884b89d)ImageAnalysis is not supported when Extension is enabled
error (#2770) (aec14b2)Timer is already canceled
(c0f3ee0)FpsSampleCollectorDelegate
weak (e93c522)FpsSampleCollectorDelegate
(83cac76)React
, Core
and FrameProcessors
subspecs (#2764) (464ea94)onProviderDisabled
to prevent abstract method crash (b0a5585)minFps
being larger in Range
than maxFps
(#2755) (16ccfdf)ModuleProxy
) actually optional (#2750) (c3098db)enableGpuBuffers
(b751f2d)This is the first beta for Skia Frame Processors! 🥳
You can try this today in your app by using the useSkiaFrameProcessor
hook:
const frameProcessor = useSkiaFrameProcessor((frame) => {
'worklet'
// 1. create blur filter
const blurRadius = 10
const blurFilter = Skia.ImageFilter.MakeBlur(blurRadius, blurRadius, TileMode.Repeat, null)
// 2. wrap blur filter in Paint
const paint = Skia.Paint()
paint.setImageFilter(blurFilter)
// 3. render image with the blur filter paint
frame.render(paint)
}, [])
Note: Skia Frame Processors require react-native-worklets-core 1.1.1 or higher, react-native-reanimated 3.x.x or higher, and react-native-skia 1.2.1 or higher
Note: You might need to set
pixelFormat="rgb"
, as"yuv"
buffers are still work in progress (see https://github.com/Shopify/react-native-skia/pull/2357)
hardware-cost-too-high
error for iOS 16 (56a5c8d)@FastNative
flags to critical Frame Processor methods (ff2005b)RCTDeviceEventEmitter
import for RN 0.71.X (#2692) (8a95414)CREATED
as default lifecycle state (1682bb2)TimeoutException
by using STARTED
lifecycle (5283020)didSessionChangeFromOutside
(5119639)not-compatible-with-outputs
error on Android since CameraX supports StreamSharing (8ad08f3)type
to onShutter
(photo
or snapshot
) (f721d97)enableAutoStabilization
prop as this was deprecated in iOS 13 (72c8c1b).await()
(7ec53f7)Snapshot.ts
(6be93af)sendAvailableDevicesChangedEvent
before module is initialized (b4a9086)androidPreviewViewType
prop (dd56925)cancelRecording
(f51921b)onShutter
event (dd49365)takeSnapshot()
to PreviewView
(e17eb50)ImageAnalysis
) (08d86d4)VideoPipeline
(00155c9)takeSnaphot
on iOS (a638b66)ImageWriter
(f36d304)qualityBalance
(e135b3b)qualityBalance
(158de70)