React Native Camera Vision x Skia: What's New in v4

If you've ever developed a mobile application that required camera functionality, then you're likely familiar with React Native Vision Camera. Flexible and efficient, this library makes integrating photo and video functionalities into react native applications straightforward.

The v4 release of the library heralds the grand return of the useSkiaFrameProcessor hook, which had promised an integration of Skia graphical library functionalities directly with camera frames. Initially considered in v3, this hook was eventually removed due to maintainability issues. However, the determination of Marc Rousavy, the creator of React Native Vision Camera, has paid off, and this latest version welcomes the new hook.

At Premier Octet, we’ve been working on our new mobile project: Photobooth AI, a photo studio leveraging artificial intelligence to reinvent your portraits. This Expo application uses React Native Vision Camera for image capture management. Therefore, it’s an opportunity for us to explore new functionalities for our application.

Setting Up the Application on Expo

First of all, we of course need to install React Native Vision Camera

npx expo install react-native-vision-camera

(click here for detailed documentation)

To manage actions within our frame processor, we will also need React Native Worklets Core, a worklet management library created by Marc Rousavy's agency

yarn add react-native-worklets-core

Don't forget to add the plugin to your babel.config.js

module.exports = {
  plugins: [['react-native-worklets-core/plugin']],
}

And of course React Native Skia for drawing

yarn add @shopify/react-native-skia
cd ios
pod install

Drawing on Our Frames with useSkiaFrameProcessor

Since library v3, the frameProcessor functionality allows applying a JavaScript worklet to each image captured by the camera. As frameProcessors process each frame synchronously, the execution speed of their code is crucial for smooth display. Thanks to worklets, our frameProcessors operations are executed on a secondary thread, which avoids overly impacting application performance and aims for 60 FPS.

The new Skia hook adds a layer to this mechanism by directly integrating a canvas into our frames. We can draw whatever we want using the imperative API of React Native Skia on our frame:

import { PaintStyle, Skia } from "@shopify/react-native-skia"
import {
  Camera,
  useSkiaFrameProcessor,
} from "react-native-vision-camera"

[...]

// Initialize a color
const paint = Skia.Paint()
paint.setColor(Skia.Color("red"))

// Define our frameProcessor
const frameProcessor = useSkiaFrameProcessor((frame) => {
   "worklet"
   frame.render() // necessary to see camera rendering
   const centerX = frame.width / 2
   const centerY = frame.height / 2
   frame.drawCircle(centerX, centerY, 50, paint)
}, [])

[...]

// Pass the frame processor to our camera
<Camera
ref={camera}
device={device}
format={format}
orientation="portrait"
photo
frameProcessor={frameProcessor}
/>

And our shape appears on our photobooth screen:

Adding Animations with Reanimated

Let's take advantage of the new Skia hook to display a face positioning guide to our users. To make it a bit more playful, an animation would be more appealing to display the shape. Skia offers an integration with Reanimated to manage animations, let's install the library on the project:

npx expo install react-native-reanimated

To achieve a gradually appearing outline animation over time, we initialize a sharedValue and apply the withTiming hook from Reanimated. To ensure our camera is activated before the start of the animation, we'll delay its start slightly with the withDelay hook.

import { Worklets, useSharedValue as useWorkletShareValue } from 'react-native-worklets-core'
import { PaintStyle, Skia } from '@shopify/react-native-skia'
import { withTiming, useSharedValue, useAnimatedReaction, withDelay } from 'react-native-reanimated'

const paint = Skia.Paint()
paint.setColor(Skia.Color('#F1EBE1'))

const reanimatedProgress = useSharedValue(0)
useEffect(() => {
  reanimatedProgress.value = withDelay(
    500,
    withTiming(1, {
      duration: 2000,
    })
  )
}, [])

A Little Nuance for Accessing Our Progress Value in the Worklet

The value set by useSharedValue from reanimated is unfortunately not accessible from the frameProcessor. React Native Worklets Core has its own useSharedValue hook to access values within the frameProcessor. Awaiting better compatibility between the two (perhaps in v5 🤞), transitioning from one variable to the other is possible by using useAnimatedReaction to update the second sharedValue with the first one's values.

const workletProgress = useWorkletShareValue(0)
useAnimatedReaction(
  () => reanimatedProgress.value,
  (value, prevValue) => {
    workletProgress.value = value
  }
)

We then pass the drawing progress parameter to our frameProcessor to display the guide gradually:

const frameProcessor = useSkiaFrameProcessor(
  (frame) => {
    'worklet'
    frame.render()

    const width = 340
    const height = 450
    const x = frame.width / 2 - width / 2
    const y = frame.height / 2 - height / 2
    const path = Skia.Path.Make()

    path.addOval(Skia.XYWHRect(x, y, width, height))
    path.trim(0, workletProgress.value, false)

    paint.setStyle(PaintStyle.Stroke)
    paint.setStrokeWidth(4)
    frame.drawPath(path, paint)
  },
  [paint, workletProgress]
)

And the animation successfully starts when the camera does 🥳

Increased Stability on Android and Geolocation as a Bonus

Finally, this new version fixes some annoying bugs we had experienced on Android (random photo flipping and camera activation issue at launch). Originally based on the Camera2 package, v4 now uses the more reliable Android library CameraX. React Native Vision Camera thus promises more stability with this new version, which is a relief 🥲!
It's also worth noting the addition of the possibility to link location tags on photos and videos, as well as a new hook useLocationPermission to obtain related permissions.

Conclusion

With the introduction of this new Skia hook, React Native Vision Camera opens up new exciting perspectives for animations and interactions with the camera. Although documentation on React Native Skia's imperative API isn't very detailed, it is quite easy to find ways to achieve one's objectives, and we hope this hook marks the beginning of a fruitful collaboration between the two libraries.

The ability to integrate animations with Reanimated, even though the sharing of shared values remains to be improved, makes the functionality particularly appealing. We are therefore eager to discover the upcoming versions and improvements, but this v4 already heralds a promising future for React Native Vision Camera 👏👏👏!