1 of 13

Scene Understanding for Autonomous Vehicles

Week 2

1

M. Cristina Bustos Rodriguez

Lidia Garrucho Moras

Xián López Álvarez

Xènia Salinas Ventalló

​

2 of 13

Task A: TT100K Detection Dataset

Analyze the Dataset:

Traffic signs images of 64x64 pixels distributed among 221 classes.

​

​

2

RGB images

Number

Percentage

train

16527

62 %

validation

1644

6 %

test

8190

32 %

In several experiments, we observed very different results between the test and the validation sets. We argue this may be caused because the validation set is very small, and it does not accurately represent the data.

Issue on the validation set:

Since on each experiment we are saving only the weights of the best epoch (in terms of accuracy), we can only measure the accuracy and loss in the test set for this epoch. Therefore, we decided to report also these measures, for the train and validation set, in the best epoch, instead of the last one.

Methodological note:

3 of 13

VGG16 on TT100K dataset

Run the configuration file with:

  • VGG16 model.
  • TT100K Dataset and input images resized to (224x224).
  • Train the model during 50 epochs using callbacks.
  • Data pre-processing: feature-wise mean and std. deviation normalization.

​

​

3

Matrics

Accuracy

Loss

Train

97.07%

0.3093

Validation

71.71%

3.25

Test

94.32%

0.7685

Results for epoch number 50

4 of 13

4

Transfer Learning on Belgium Dataset

4

Fine-tune the last FC layers of a VGG16 for the Belgium DTS dataset using the weights obtained training the same VGG16 on TT100K dataset.

​

​

Matrics

Accuracy

Loss

Train

99.87%

0.0149

Validation

97.30%

0.2768

Test

96.98%

0.3186

Results for epoch number 40

5 of 13

VGG16 on KITTI dataset

From scratch:

Fine-tuning:

5

Set

Accuracy

Loss

Train

97.63%

0.30

Validation

96.96%

0.41

Set

Accuracy

Loss

Train

97.73%

0.27

Validation

97.03%

0.38

Best results reported for epoch 32

Best results reported for epoch 18

6 of 13

6

7 of 13

ResNet

From scratch:

Fine-tuning:

7

Set

Accuracy

Loss

Train

97.17%

0.372

Validation

61.86%

5.129

Test

93.24%

0.703

Set

Accuracy

Loss

Train

85.51%

0.779

Validation

41.55%

6.632

Test

81.45%

1.052

Best results reported for epoch 29

Best results reported for epoch 25

8 of 13

8

9 of 13

InceptionV3

From scratch:

Fine-tuning:

9

Set

Accuracy

Loss

Train

91.20%

1.352

Validation

75.00%

3.507

Test

92.25%

1.043

Set

Accuracy

Loss

Train

88.09%

1.823

Validation

34.91%

10.37

Test

79.20%

3.212

Best results reported for epoch 42

Best results reported for epoch 47

10 of 13

10

11 of 13

DenseNet

From scratch:

11

Set

Accuracy

Loss

Train

99.23%

3.413 1e-2

Validation

74.57%

1.753

Test

94.61%

0.384

Results reported for epoch 25

12 of 13

Training VGG in TT100K with Data Augmentation

12

  • With data augmentation: 66.48% validation accuracy
  • Without data augmentation: 72.32% validation accuracy

No improvement

13 of 13

13

Results on test set

  • VGG16 from scratch on TT100K: 94.32% accuracy - 0.76 loss.
  • Transfer learning of VGG16 on Belgium DTS: 96.98% accuracy - 0.31 loss.
  • KITTI on TT100K (from scratch and fine-tuning): 97% accuracy - 0.38 loss.
  • ResNet on TT100K:
    • From scratch: 93.24% accuracy - 0.70 loss.
    • Fine-tuning: 81.45% accuracy - 1.05 loss.
  • InceptionV3 on TT100K:
    • From scratch: 92.25% accuracy - 1.04 loss.
    • Fine-tuning: 79.20% accuracy - 3.21 loss.
  • DenseNet from scratch on TT100K: 94.61% accuracy - 0.38 loss.