Yingqian Wang1, Longguang Wang2, Zhengyu Liang1, Jungang Yang1, Radu Timofte3, Yulan Guo4,1
1National University of Defense Technology, 2Aviation University of Air Force, 3University of Wurzburg, 4Sun Yat-sen University
NTIRE 2023 Challenge on Light Field Image Super-Resolution: Dataset, Methods and Results
- Training Set. This challenge follows the most existing LF image SR works, and uses the EPFL, HCInew, HCIold, INRIA and STFgantry datasets for training. All the 144 LFs in the training set have an angular resolution of 9×9. The participants are required to use these LF images as HR groundtruth to train their models. External training data or models pretrained on other datasets are not allowed in this challenge.
- Validation Set. In this challenge, we develop a new LF dataset (namely, NTIRE-2023) for both validation and test, as shown in Fig. 1. The validation set contains 16 synthetic scenes rendered by the 3DS MAX software1 and 16 real-world images captured by Lytro Illum cameras. For synthetic scenes, all virtual cameras in the camera array have identical internal parameters and are co-planar with the parallel optical axes.
- Test Set. To rank the submitted models, a new test set consisting of 16 synthetic LFs (rendered in the same way as in the validation set) and 16 real-world LFs (captured by Lytro Illum cameras) are provided, as shown in Fig. 1.
Fig. 1: An illustration of the center-view images in the developed NTIRE-2023 LF dataset. Both validation and test sets contain 16 real-world and 16 synthetic LFs, respectively.
- The participants commonly performed random flipping and rotation for training data augmentation. In addition, two teams randomly sampled 5 × 5 LFs from 9 × 9 LFs to further augment the training set. Both data ensemble (a.k.a. test-time augmentation) and model ensemble were adopted in several solutions to boost the SR performance.
- There seems to be a considerable room of further performance improvement, because ensemble strategy and some advanced data augmentation approaches (such as CutBlur [89] ) have not been widely used.
- This challenge provides a PyTorch-based, open-source, and easy-to-use toolbox named BasicLFSR to facilitate participants to quickly get access to LF image SR and develop their own models.
- All the proposed methods are based on deep learning techniques. Transformers are used as the basic architecture in 6 solutions, while other models are purely based on CNNs.
- Seven teams adopted the disentangling mechanism in LF-Distg [2] to divide the 4D LFs into four 2D subspaces including spatial subspace, angular subspace, horizontal EPI subspace, and vertical EPI subspace.
- Three teams performed feature extraction and incorporation in spatial and EPI subspaces. The recently developed method EPIT [88] was used as the backbone by the OpenMeow team (winner) and the BNU-AI-TRY team.
- (1) It provides a complete pipeline to develop novel LF image SR methods. (2) It integrates a number of LF image SR methods, and retrains them on unified LF datasets. The codes and checkpoints of each model are publicly available. (3)It provides a fair and comprehensive benchmark for LF image SR. The quantitative results of each method are listed, and their super-resolved LF images are available for download.