Video Upscaling & Denoising for WebGPU

Ruijun(Daniel) Zhong, Yuanqi Wang, Tong Hu

Why this project matters?

Video upscaling & denoising are important for enhanced media experience. Modern upscaling & denoising functionalities are often done per application for best performance.

WebGPU provides a way to run upscaling & denoising algorithms in near-native performance and promises cross-platform access. Since the model is running on the client side, users can have more control over the process and it is also cost-efficient.

​

Goals & Outcomes

BASIC GOALS

1. Construct pipeline for conversion from deep learning models to WGSL shaders.
2. Implementation of Anime4K using WebGPU, publish functionality as npm package.
3. Create video upscaling web demo app.
4. Denoising & line art deblurring feature.
5. Use WebAssembly for better performance.

STRETCH GOALS

1. Line art deblurring shaders
2. Including other interesting models to improve the quality of video.

Schedules of Milestones

Read related materials

Learn WebGPU and setup base code

Convert ML model to WGSL shaders

Build WebGPU pipeline

Create demos

Improve pipeline

npm package

Polish & Debug

Improve performance

Other stretch goals (optional)

upscaling

denoising

Style transfer

Performance analysis

Integrate WebAssembly

References

​

APIs, Platforms planning to use

​

1. Anime4K repo https://github.com/bloc97/Anime4K
2. StyleRF: Zero-shot 3D Style Transfer of Neural Radiance Fields https://arxiv.org/pdf/2303.10598.pdf

​

WEBGPU