1 of 6

Real-time video processing with WebCodecs and Streams��https://github.com/tidoust/media-tests/

TPAC 2023 - Technical demo

September 2023

​

François Daoust - W3C - fd@w3.org - @tidoust

2 of 6

Scenarios that require processing�video frames in real-time

Media stream analysis

  • Barcode reading
  • Face recognition
  • Gesture/Presence tracking
  • Emotion analysis
  • Speech recognition
  • Depth data stream processing

Media stream manipulation

  • Funny hats
  • Voice effects
  • Background removal or blurring
  • In-browser composition
  • Augmented reality
  • Non-linear video edition

3 of 6

Reasoning about memory copies is hard!

  • Various memory boundaries that require copies… or not
  • GPU and CPU memory are physically distinct… or not
  • Browsers use additional sandboxing to proxy GPU memory… or not
  • Browsers use hardware-based encoders/decoders… or not
  • Hardware-based encoders/decoders are GPU bound… or CPU bound
  • Efficient caching algorithms could help… but buffers cannot be reused�(they could in the future though, e.g., see w3c/webcodecs#676)
  • Memory copies also needed to transpose objects to another format… but no way to impose initial frame format

4 of 6

Example: optimization… or not

Timer

Memory

Frames

Avg.

Median

toRGBX

GPU

247

0

0

​

​

​

​

​

encode

GPU? CPU?

247

17

15

decode

GPU? CPU?

247

1

1

end2end

​

247

24

17

queued

​

247

6

0

Timer

Memory

Frames

Avg.

Median

toRGBX

GPU

256

0

0

background

CPU - WASM

256

13

12

encode

GPU? CPU?

255

8

7

decode

GPU? CPU?

255

1

1

end2end

​

255

28

21

queued

​

255

6

0

All on the GPU where processing is cheap!�Unfortunately, browser likely used a CPU-bound encoder on my machine…

Adding a slow processing step in WebAssembly. End-to-end time increase is just ~4ms as encoding times dropped. One less copy needed, probably!

5 of 6

Rough illustration of memory boundaries in browsers

CPU

GPU

I/O

​

📶

​

📷

​

🎤

​

🔈

​

📺

ML hardware / FPGA

Hardware media encoders / decoders

JavaScript�land

WASM land

WebGPU/WebGL land

Browser

Main

Worker

Worker

6 of 6

Ongoing web media standardizations and groups

Ongoing media standardizations

  • Media transport
    • MSE v2 (codec switching, support in workers, ManagedMediaSource)
    • WebTransport
  • Audio/Video frame access & processing
  • Expose device capabilities
    • Media Capabilities
  • HDR support in canvas
    • Extensions to canvas
    • HDR in PNG
    • Float16Array in JavaScript

Related groups