Preserving

Retro Game Footage for Modern Viewers with JS and WebGL

Travis (TiKevin83) McGeehan

Full Stack Developer at GFS

TASBot Administrator

TASVideos Ambassador and TAS Author

Who is TASBot and the TASBot Team?

• TASBot is like a player piano for retro game consoles, particularly the NES, SNES, N64 with help from GBI on the Game Boy
• TASBot team helps translate TASes to being played back on console
• Creates showcases at MAGFest and GDQ requiring precise video capture

A Word of Caution

• The discussion here is relevant mainly to video archival and is tuned towards animated footage, especially in retro video games
• This discussion will be less relevant to live filmed video except when trying to transmit viewable archive copies of such video over the web.

Core problems in Video Game Footage Archival

• Chroma subsampling
• 4:2:0 chroma subsampling is a technique in video compression where color data is stored at a lower resolution than the whole video to take advantage of quirks in the human eye
• Games were not typically chroma subsampled until the GameCube era
• Small Resolutions with Stretched Aspect Ratio
• Too small to show without upscaling on most modern displays, especially the most color accurate displays which are 1440p/4k
• Often alongside rectangular pixels on CRTs, impossible on modern fixed pixel displays
• Scaling for Display Presentation
• For upscaling, browsers typically use algorithms like bilinear that don’t apply well to pixel art
• Lossy compression vs Lossless
• while great for small resolution pixel art archival, lossless is not widely supported
• H264 only supports lossless in the High 4:4:4 predictive profile, not usually supported in browsers
• Art designed for CRTs and composite video signals
• Phosphor decay and composite signaling cause blending of colors between pixels, enabling dithering to create gradients with more colors than the console could otherwise generate

Scaling Algorithms on Modern Displays

Atari Dragster - Effect of mentioned artifacts stacked together

Atari Dragster - Aspect Corrected and Area scaled on the client from 4:4:4 Lossless source

Atari Dragster - With CRT Simulation

RGBScaler.com tech demo

• Uses all of the available mentioned techniques in combination to preserve crisp footage of very low resolution game boy games
• Area upscaling to aspect corrected sizes is possible in WebGL2 by copying each video frame to a canvas element
• Simulation of CRT effects possible in WebGL2 as well
• Requires the player to be on the same site as the video or for the video host to accept CORS
• Works well for games from the N64 era and earlier, especially the oldest consoles such as the Atari 2600
• Easily clonable to host your own videos with the same technology on your own sites

AV1 Video Codec

• AV1 broadens support for 4:4:4 and lossless video footage
• Compatible with Firefox and Chrome Android in addition to Chromium based desktop browsers
• Enables efficient storage and proper upscaling of pixel art footage from early consoles in the NES/SNES/N64 eras, also for footage captured from DOS and other early computers and arcade machines
• H264 more efficient that AV1 for lossless mode, AV1 shines more in lossy footage, but lossless H264 is not widely compatible

H265 Video Codec

• In srcset combination with AV1
• Necessary to complement AV1 for iOS and Safari

How to send video to a WebGL texture?

• MDN has a helpful tutorial “Animating Textures in WebGL”
• Tutorial had to be heavily adjusted for use in React

How to Interact with a Canvas in React?

• Ref attribute set to a callback defined in a custom hook
• Callback performs WebGL boilerplate setup
• Custom hook has canvas and gl state variables updated in the ref callback
• useEffects in the custom hook update things like dimensions and effect intensity

How does the WebGL Texture update for new frames?

• The video is polled for a frame using a “RequestAnimationFrame” callback
• The result from RequestAnimationFrame is an ID which we send to a state object and can then be used by CancelAnimationFrame to end the loop.
• This helps clean up the render loop when a video is paused/resumed.

Essentials of the Shader

• Texture sampled using gl.Linear to enable using the Area effect
• Area effect ported from Open Broadcaster Studio source code
• Combined with simple Scanline and Slot Mask implementations

Getting Syntax Highlighting for the Shader code

• WebGL shaders are loaded as raw strings in JS in a call to gl.shaderSource()
• rollup-plugin-string can be used to enable importing these strings from files so that the separate files can have syntax highlighting for GLSL
• .d.ts files can fix IDE warnings for this usage

Custom Controls

• Because the native HTML5 video element has to be hidden, the native video controls are also hidden
• Controls are reimplemented easily in React components

Working with DPI Scaling

• Difference between CSS and Device Pixels
• Window.devicePixelRatio

Compression Efficiency Breakdown - 2D Dragster Source using h264

• 8bit 444 lossless native res (160x224) - 119 kb/s
• 10bit 444 lossless native res - 108 kb/s
• 8bit 420 lossy (CRF 18) native res - 61 kb/s
• 8bit 420 lossy 4k (2880x2160) nearest neighbor upscale - 443 kb/s
• 8bit 444 lossless 4k upscale - 340 kb/s
• 10bit 444 lossless 4k upscale - 401 kb/s

Compression Efficiency Breakdown - 3D SM64 Source using h264

• 8bit 444 lossless native res: 10,011 kb/s
• 10bit 444 lossless native res: 5,135 kb/s
• 8bit 420 lossy native res: 916 kb/s
• 8bit 420 lossy 4k (2880x2160) upscale: 23,606 kb/s
• 8bit 444 lossless 4k upscale: 96,069 kb/s
• 10bit 444 lossless 4k upscale: 95, 101 kb/s

Compression Efficiency Takeaways

• 10bit encoding can be 10-50% more efficient than 8bit for lossless modes in x264
• Lack of hardware playback compatibility has hampered adoption due to battery drain of software decoding
• Not relevant when pre-upscaling on server side
• Filesizes are so small at native res for simple 2D sources like Atari footage that lossless is very preferable, and can even be more efficient than lossy encoding if upscaling server-side
• Native resolution lossless encoding is much less efficient on 3D sources, but still very far within typical streaming limitations at 5-10 MBit/s for N64 era and earlier footage and much more efficient than pre-upscaled lossy footage when targeting 4k
• These comparisons are only possible with clean emulator footage, noise from analog capture makes lossless encoding much less efficient