Reference-based Restoration of Digitized Analog Videotapes

MICC – University of Florence, Italy

Winter Conference on Applications of Computer Vision 2024 (WACV2024)

​

Lorenzo Agnolucci

Leonardo Galteri

Analog Video Restoration

🔍 Analog magnetic tapes have been the main video data storage device for several decades, but their content show unique and severe degradation

​

❌ Standard video restoration works are designed for digital videos and do not consider the artifacts caused by media issues, while old video restoration methods only focus on structured defects such as scratches

​

Contributions

💡 We propose TAPE, an approach for restoring analog videos that exploits the time-varying nature of the artifacts by identifying the least damaged frames of each video with CLIP and employing them as references

�💡 We develop a Swin-UNet architecture that leverages reference frames through our MRSFF blocks

​

​

TAPE Overview

1. Frame classification: we use CLIP for zero-shot artifact detection to identify the cleanest frames of each video
2. Reference-based restoration: we select the references from the set of clean frames to restore the input frames with our Swin-UNet

​

Swin-UNet Architecture

• Our Swin-UNet architecture restores T frames at once by leveraging the information of both neighboring and reference frames
• The Swin Transformer allows taking advantage of the expressiveness of the attention mechanism while reducing the computational complexity

​

​

​

MRSFF Block

• Our Multi-Reference Spatial Feature Fusion (MRSFF) blocks rely on cross-attention and attention pooling to take advantage of the most useful parts of each reference frame
• Intuitively, each input frame looks at similar parts of the reference frames and exploits them to restore the lost details

​

​

Synthetic Dataset Results

[3] Agnolucci, Lorenzo et al. “Restoration of Analog Videos Using Swin-UNet“, ACM MM 2022

Synthetic Dataset Video Results

Real-world Dataset Results

[3] Agnolucci, Lorenzo et al. “Restoration of Analog Videos Using Swin-UNet“, ACM MM 2022

Real-world Dataset Video Results

Yu, Jiyang et al. “Memory-Augmented Non-Local Attention for Video Super-Resolution“, CVPR 2022

Conclusions

❌ Existing video restoration methods do not consider the artifacts typical of analog videos

💡 TAPE exploits the time-varying nature of the artifacts by identifying the cleanest frames of each video and using them as references

📈 TAPE achieves state-of-the-art results on�both synthetic and real-world videos

Leonardo Galteri

[name].[surname]@unifi.it

MICC – University of Florence, Italy