Blind Image Inpainting via Omni-dimensional Gated Attention and Wavelet Queries

Shruti S. Phutke, Ashutosh Kulkarni, Santosh Kumar Vipparthi, and Subrahmanyam Murala

Computer Vision and Pattern Recognition Lab

Indian Institute of Technology Ropar, Rupnagar, Punjab

Motivation

• Blind image inpainting is now-a-days gaining more attention due to its wide variety of applications like object removal, watermark removal, etc.

​

• Existing approaches for blind image inpainting have limited performance when different unseen corrupted regions are considered.

​

• Providing processed query features to transformer may boost its performance.

​

• The multi dimensional attention will further provide more attentive information for inpainting architecture.

The contributions of the proposed work are:

​

• An end-to-end transformer based architecture is proposed for blind image inpainting.

​

• A novel wavelet query multi-head attention mechanism is introduced in the transformer block.

​

• A omni-dimensional gated attention mechanism is proposed to forward different dimensional attentive features from encoder to respective decoder for effective reconstruction of inpainted image.

Contributions

Proposed architecture for blind image inpainting.

Proposed Method

Proposed Method

Experimental Dataset

• CelebA-HQ [1]:
• Contains 30K images of male and female faces.
• 28K images for training, 2K images for testing.

• FFHQ [2]:
• The dataset consists of 70K high-quality PNG images.

[1] Tero Karras, Timo Aila, Samuli Laine, and Jaakko Lehtinen. Progressive growing of gans for improved quality, stability, and variation. In ICLR, 2018.

[2] Tero Karras, Samuli Laine, and Timo Aila.. A style-based generator architecture for generative adversarial networks. In Proceedings of the IEEE/CVF CVPR, PP 4401–4410, 2019.

Experimental Dataset

[1] Carl Doersch, Saurabh Singh, Abhinav Gupta, Josef Sivic, and Alexei Efros. What makes paris look like paris? ACM Transactions on Graphics, 31(4), 2012.

[2] Bolei Zhou, Agata Lapedriza, Aditya Khosla, Aude Oliva, and Antonio Torralba. Places: A 10 million image database for scene recognition. IEEE TPAMI, 40(6):1452–1464, 2017

• Paris Street View [1]:
• 14K images for training, 100 images for testing.

• Places2 [2]:
• 365 different places categories with 5k images for training and 100 images for testing in each category.

Ablation study on different configurations of the proposed network on ParisSV dataset for blind image inpainting (Note: ↑- Higher is better, ↓- Lower is better).

[1] Xiefan Guo, Hongyu Yang, and Di Huang. Image inpainting via conditional texture and structure dual generation. IEEE/CVF International Conference on Computer Vision, pages 14134–14143, 2021.

Qualitative result analysis of ablation study on different configurations of the proposed network for blind image inpainting.

Ablation Study

[1] Yi Wang, Ying-Cong Chen, Xin Tao, and Jiaya Jia. Vcnet: A robust approach to blind image inpainting. In European Conference on Computer Vision, pages 752–768. Springer,2020.

[2] Haoru Zhao, Zhaorui Gu, Bing Zheng, and Haiyong Zheng. Transcnn-hae: Transformer-cnn hybrid autoencoder for blind image inpainting ACM International Conference on Multimedia, pages 6813–6821,2022.

[3] Xiefan Guo, Hongyu Yang, and Di Huang. Image inpainting via conditional texture and structure dual generation. IEEE/CVF International Conference on Computer Vision, pages 14134–14143, 2021.

Comparison of the proposed method (ours) and existing state-of-the-art methods for blind image inpainting.

Result Analysis

Qualitative results comparison of the proposed method (Ours) with existing state-of-the-art methods (VCNet [1], CTSDG [2], TransCNNHAE [3]) on Celeb (first two rows) and FFHQ (last two rows) dataset for blind image inpainting.

Qualitative results comparison of the proposed method (Ours) with existing state-of-the-art methods (VCNet [1], CTSDG [3], TransCNNHAE [2]) on ParisSV (first two rows) and Places2 (last two rows) dataset for blind image inpainting.

Qualitative results comparison of the proposed method (Ours) with existing state-of-the-art method (TransCNNHAE [2]) on unseen patterns.

[1] Yi Wang, Ying-Cong Chen, Xin Tao, and Jiaya Jia. Vcnet: A robust approach to blind image inpainting. In European Conference on Computer Vision, pages 752–768. Springer,2020.

[2] Xiefan Guo, Hongyu Yang, and Di Huang. Image inpainting via conditional texture and structure dual generation. IEEE/CVF International Conference on Computer Vision, pages 14134–14143, 2021.

[3] Haoru Zhao, Zhaorui Gu, Bing Zheng, and Haiyong Zheng. Transcnn-hae: Transformer-cnn hybrid autoencoder for blind image inpainting ACM International Conference on Multimedia, pages 6813–6821,2022.

Result Analysis

• We propose a novel approach with wavelet query multi-head attention and a omni-dimensional gated attention mechanism for blind image inpainting.

​

• The quantitative and qualitative comparison is performed with state of the art methods for blind and non-blind image inpainting.

​

• Further, the qualitative results are shown for unseen patterns.

​

• The proposed architecture has more complexity as compared to the state-of-the-art blind image inpainting method.

Summary

Thank You

If you are interested in our work, please visit the repository at: