1 of 16

Detecting sneaker males

Ketil Malde, 2024-08-26 (HIBRAIN)

2 of 16

What are sneaker males?

3 of 16

Research questions:

  • Can we automatically identify the sneaker males?
  • Which features are important?

4 of 16

The mugshot database

Annotations

5 of 16

Data set

​

Nesting males

Females

Female mimics

Total

n unique individuals

596 (4373)

774 (3264)

296

9303

n sightings

837 (5077)

1038 (3781)

396

11129

n images

1729 (10223)

2129 (7607)

808

22496

validated

unvalidated

no unvalidated specimens

6 of 16

Getting to work

Plan:

  • annotate body parts
  • extract crops of head, whole fish, and headless body
  • train Yolo to automate cropping
  • organize by known sex and sneaker status
  • train ResNet to classify crops
  • profit!

7 of 16

Training and validation sets

8 of 16

Identifying features

9 of 16

Experimental setup

Mugshots

Annotated

set

Yolo v8

training

10 of 16

Experimental setup

Mugshots

Annotated

set

Yolo v8

heads

bodies

head-

less

processing

11 of 16

Experimental setup

Mugshots

Annotated

set

Yolo v8

heads

bodies

ResNet

ResNet

head-

less

ResNet

training classifiers

12 of 16

Extracting the crops

13 of 16

ResNet classifiers

Variations:

  • ResNet 50
  • ResNet 101
  • Monochrome images
  • Image resolution
  • Early termination (or fixed 30 epochs)
  • Learning rates

14 of 16

Results

Dataset

Morph

n of images

Accuracy

Precision

Recall

F1

Head

female

142

0.97

0.93

0.99

0.96

Head

male

143

0.97

1.00

1.00

1.00

Head

female mimic

125

0.97

0.99

0.91

0.95

Full body

female

142

0.85

0.75

0.89

0.81

Full body

male

143

0.85

0.99

0.99

0.99

Full body

female mimic

125

0.85

0.84

0.66

0.74

Body only

female

142

0.88

0.81

0.90

0.85

Body only

male

143

0.88

0.97

0.99

0.98

Body only

female mimic

125

0.88

0.87

0.74

0.80

15 of 16

Comparison to experts

Two experts were given images to classify.

16 of 16

Conclusions

Surprisingly: Our deep learning algorithm is able to correctly identify sneaker males in a large fraction of the cases.

This is practically impossible for human experts.