1 of 10

AgATHON 2023

Team 22 – Labor Challenge

David Smerkous, Emily Arteaga, Maha Jinadoss, Anita Ruangrotsakun

2 of 10

Overview

created by DALL-E

Harvesting

Human labor shortage
Robotics and computer vision challenges

Our Solution

Apple identification with per apple uncertainty estimation

3 of 10

Problem: Apple Identification

Use computer vision to identify apples with uncertainty estimation to help apple picking robots decide what to grab with generalizability to other apple images.

created by DALL-E

4 of 10

Beneficiaries

Immediate beneficiaries

Apple picking robotics companies
Universities researching on robotic fruit picking

Downstream beneficiaries

Apple farmers, harvesters, and consumers

Engineering Out Loud Season 9 -- And on that farm, he had a robot

5 of 10

Data Augmentation

15% holdout set

100 images
No augmentations

Training images

Rotation
Gaussian blurring
Mirroring
Padding
Cutouts (apple-shaped)

created by us!

6 of 10

Uncertainty Based Architecture Overview

DeepLabV3+ Segmentation Model

Input Batch �of Images

Ensemble of Segmentation Networks [1]

….

6 rand init models with high acc ~ 0.82 test dice

higher variance

….

6 rand init models with low acc ~0.45 test dice

higher variance

….

Segmentation masks of potential apple

candidates

….

Apple Predictions

Weighted deep ensemble

Apple candidate masks

Mask RCNN Instance Segmentation/Detection Model

OpenCV apple instance variance

[1] Ganaie, M. A., Hu, M., Malik, A. K., Tanveer, M., & Suganthan, P. N. (2022). Ensemble deep learning: A review. Engineering Applications of Artificial Intelligence, 115, 105151.

7 of 10

Segmentations Instance Segmentations

Different confidence

scores/shapes

8 of 10

Next Steps: Calculating instance segmentation variance

High Confidence Example

Low Confidence Example

Identified high confidence RCNN

Instance/Apple #1

Network 1

Prediction

Identified medium confidence RCNN

Instance/Apple #2

Network 2

Prediction

Network 3

Prediction

Identified low confidence RCNN

Instance/Apple #3

Calculate IoR of each network segmentation, then use mean/variance of IoR to estimate confidence of apple prediction (low variance of IoR == high confidence).

A single network within an ensemble may be (over)confident for a specific apple, but others might not be.

Network 1

Prediction

Network 2

Prediction

Network 3

Prediction

IoR = (area of intersection)/(area of instance segmentation)

9 of 10

Results / Metrics

Model	Dice (lowest-highest)	AP (IoU @ 0.5)	AP (IoU @ 0.5:0.95)
DeepLab Ensemble’s	0.42 - 0.83	N/A	N/A
Mask RCNN Segmentation	N/A	0.712	0.304
Mask RCNN BBox	N/A	0.726	0.323

10 of 10

Thank you for watching!

created by DALL-E