A deep technical walkthrough of building real-time video call filters (background blur, virtual backgrounds, Center Stage zoom, and live drawing) in React Native by replacing LiveKit's default camera pipeline with VisionCamera. The approach bypasses LiveKit's camera entirely, using a native frame-processor engine (react-native-vc-engine) to intercept, transform, and inject frames directly into WebRTC's video source via the onFrameCaptured (Android) / didCaptureVideoFrame (iOS) seam, leaving encoding, transport, and signaling untouched. Covers pixel format conversion (YUV/I420 on Android, NV12 CVPixelBuffer on iOS), segmentation-based masking (MediaPipe, Vision framework), GPU-accelerated Metal compositing on iOS versus CPU Kotlin on Android, and real-world frame rate benchmarks on an iPhone 13 and a budget Samsung Galaxy F14 under cross-continent network latency.
Table of contents
The BackendA Basic Video CallHow LiveKit Injects FramesTaking Over the Camera with VisionCameraMatching the Format WebRTC ExpectsThe Complete PathAdding FiltersPerformanceFinal ThoughtsQuestions this post answers
How do you bypass LiveKit's default camera in React Native to apply custom video filters?
Disable LiveKit's built-in camera with video={false}, then drive the camera with VisionCamera instead. Each processed frame is pushed directly into WebRTC's video source through the same onFrameCaptured (Android) or didCaptureVideoFrame (iOS) callback that the native camera capturer normally uses, so LiveKit's encoder, transport, and signaling stay untouched while only the pixels change. Developers wiring custom camera pipelines into WebRTC calls can find implementation patterns like this on daily.dev.
Why can't you apply a full-resolution Gaussian blur to every frame in a real-time video call?
A full-resolution Gaussian blur on every frame is too computationally expensive to sustain at video call frame rates. Both Android and iOS instead shrink the frame first, blur the small copy, then scale it back up, since the upscale's interpolation does most of the visual smoothing for free while blurring a fraction of the pixels, for example roughly 1/16th of them on iOS via Metal Performance Shaders. Anyone optimizing real-time video effects can track performance techniques like this via daily.dev.
What frame rate can a budget Android phone sustain while running background blur and face tracking in a video call?
A low-end Samsung Galaxy F14 sustains around 30 fps while forwarding frames or applying a light effect, drops to about 25 fps with Center Stage face-tracking zoom, and eases to roughly 15-20 fps under the heaviest combined path of Center Stage plus segmentation plus blur plus composite, since the CPU-based Kotlin compositing lacks GPU acceleration. An iPhone 13 using Metal on the GPU carries every effect at the full 30 fps capture rate. Engineers benchmarking effect-heavy video pipelines across device tiers can follow findings like this on daily.dev.