1 of 21

Mattias Buelens

THEO Technologies

2 of 21

Why?

Because we can control things on a lower level
Because we can now do it efficiently

WebCodecs: interface directly with hardware decoder
“Possibly to implement something like MSE or WebRTC on top with the same battery life and latency.” (https://github.com/w3c/webcodecs/blob/main/explainer.md#goals)

Because it’s a fun learning exercise

Why would you want to do this, when we already have a perfectly good built-in <video> element?

<video> element makes a lot of decisions for us: when to decode next frame, and how many frames it will decode in advance. For some use cases (low-latency streaming) might not be enough. Take control of decoding to fine-tune things.

Also, because today we can actually do this efficiently! With the WebCodecs API, we can interface directly with the hardware decoder, and decode individual frames on demand. Explainer even mentions that you should be able to implement something like Media Source Extensions using WebCodecs. To that I say: challenge accepted.

Can make this into a running theme:

2021: Collin Miller replaces ffmpeg with WebCodecs
2022: I replace <video> with WebCodecs
2023: maybe someone replaces Smell-O-Vision with WebCodecs? Submit a talk.

Finally, it’s a fun learning exercise. Some of things that we take for granted from a <video> element turn out to be quite nuanced, and you only notice when you try to build it yourself.

Do note though: this is a toy implementation, so don’t use this in production. When your kid is playing with a toy kitchen, you wouldn’t want to eat whatever they cook up in there either.

3 of 21

Hello world

<!DOCTYPE html>�<html>� <body>� <baby-video></baby-video>� <script src="./baby-video.js"></script>� </body>�</html>

class BabyVideoElement extends HTMLElement {� #canvas;� #canvasContext;�� constructor() {� super();� const shadowRoot = this.attachShadow({ mode: "open" });� this.#canvas = document.createElement("canvas");� this.#canvas.width = 300;� this.#canvas.height = 150;� shadowRoot.appendChild(this.#canvas);�� this.#canvasContext = this.#canvas.getContext("2d");� this.#canvasContext.fillStyle = "black";� this.#canvasContext.fillRect(� 0, 0, this.#canvas.width, this.#canvas.height);� }�}��customElements.define("baby-video", BabyVideoElement);

4 of 21

5 of 21

Add some buttons

Could build our own UI from scratch with HTML/CSS/JS

…but if our <baby-video> looks like a <video>

…and it quacks like a <video>

…then we can use <media-chrome>(https://github.com/muxinc/media-chrome)

6 of 21

Add some buttons

<!DOCTYPE html>�<html>� <body>�� <media-controller>� <baby-video slot="media"></baby-video>� <media-control-bar>� <media-play-button></media-play-button>� <media-time-display show-duration></media-time-display>� <media-time-range></media-time-range>� <media-fullscreen-button></media-fullscreen-button>� </media-control-bar>� </media-controller>�� <script type="module"� src="https://unpkg.com/media-chrome@0.12.0"></script>� <script src="./baby-video.js"></script>� </body>�</html>

7 of 21

Buffering with Media Source Extensions

const mediaSource = new BabyMediaSource();�video.srcObject = mediaSource;�await waitForEvent(mediaSource, "sourceopen");�mediaSource.duration = 30;�const sourceBuffer = mediaSource.addSourceBuffer('video/mp4; codecs="avc1.640028"');��const segmentURLs = ["video_init.mp4", "video_1.mp4", "video_2.mp4"];�for (const segmentURL of segmentURLs) {

� const segmentData = await (await fetch(segmentURL)).arrayBuffer();

� sourceBuffer.appendBuffer(segmentData);� await waitForEvent(sourceBuffer, "updateend");

�}

3. Append to SourceBuffer

2. Download fMP4 files

1. Create MediaSource

8 of 21

Implementing MSE

SourceBuffer.appendBuffer(data)

Parse fMP4 using MP4Box.js (https://github.com/gpac/mp4box.js)
Represent as WebCodecs configs/chunks (https://w3c.github.io/webcodecs/)

9 of 21

Playback

class BabyVideoElement extends HTMLElement {� #videoDecoder;� constructor() {� this.#videoDecoder = new VideoDecoder({� output: (frame) => this.#onVideoFrameDecoded(frame),� error: (error) => console.error("Uhoh"),� });� }

#onAnimationFrame() {� const videoTrackBuffer = getActiveVideoTrackBuffer(this.#mediaSource);� if (this.#videoDecoder.state === "unconfigured") {� this.#videoDecoder.configure(videoTrackBuffer.codecConfig);� }

const frame = videoTrackBuffer.findFrameForTime(this.currentTime);� if (frame) {� this.#videoDecoder.decode(frame);� }� }� #onVideoFrameDecoded(frame) {� this.#canvasContext.drawImage(frame, 0, 0, frame.displayWidth, frame.displayHeight);� frame.close();� }�}

3. Draw frame

2. Find and decode frame

1. Configure decoder

10 of 21

11 of 21

Playback (attempt #2)

Problem: display frame rate (e.g. 60 Hz) > video frame rate (e.g. 30 fps)

Accidentally decoding each frame twice, does not work

Solution: on every animation frame:

Find encoded video frame at current time
If same frame as last decoded frame, return
Decode frame
Render decoded frame to canvas

12 of 21

13 of 21

Seeking

…should just work, right? 🤷‍♂️

14 of 21

15 of 21

Seeking (attempt #2)

Problem:

Seek can end up at delta frame (P- or B-frame), cannot decode independently

Solution:

Decode frame and all its dependencies that we haven’t decoded yet

Starting from last decoded frame (if continuing same group-of-pictures)
Or starting from keyframe (if in different GOP)

Render frame (but skip rendering its dependencies)

Also handles case where display frame rate < video frame rate

⚠ Fewer keyframes = more frame dependencies = slower seeking

👉 Use keyframe interval of ~2 seconds

16 of 21

17 of 21

Managing buffer size

Append media when needed, but also: remove when no longer needed
SourceBuffer.remove(start, end)

Removes all frames with presentation time between start and end time
⚠ Also removes any frames that depend on previously removed frames

P

I

P

I

P

I

P

Buffer before:

P

I

P

I

P

Buffer after:

remove range

18 of 21

Managing buffer size

Proactive: remove before new append

When playing forwards: remove frames too far before current time

⚠ Don’t remove frames of currently playing GOP!
👉 Player should not remove closer than 1 keyframe interval from currentTime

When seeking backwards: remove frames too far after current time

Reactive: remove when buffer is full

SourceBuffer.appendBuffer() throws QuotaExceededError when too much data
👉 Player should reduce its buffering goal, postpone append until it can remove some data

19 of 21

Quality switching

If different codec configuration as last decoded frame, reconfigure decoder
But what if qualities have different segment durations?��

👉 Player should avoid switching qualities too close to current time

👉 Even better: align segment boundaries across qualities

Buffer before:

#1

#2

#1

480p quality:

720p quality:

#2

#3

#1

#2

#4

Buffer after:

#1

#2

#3

#4

20 of 21

Conclusion

Built our own <baby-video>�with most of the <video> API

https://mattiasbuelens.github.io/baby-video/
Uses WebCodecs, so needs Chrome 94+

Decoding is tricky to get right
Video player should be careful when cleaning buffers and switching qualities
WebCodecs is pretty neat

21 of 21

Links

WebCodecs samples (https://w3c.github.io/webcodecs/samples/)

Also has sample with audio and video

“WebRTC and Real-Time Applications Web Codecs and the Next Generation of Web Media APIs” (https://www.youtube.com/watch?v=U8T5U8sN5d4)

Bernard Aboba, Chris Cunningham and Paul Adenot at 2021 Real Time Communications Conference

Video processing with WebCodecs (https://web.dev/webcodecs/)