Sauron

Jeremiah Lynn, Logan Kloft, Taylor West

Problem Statement

Project Goals

• General object detection using COCO pretrained models
• Compare Performance of Object Detection Models
• Create an application using ML models

Object Detection has a wide variety of use cases. Ranging from counting cars on a busy road or people entering a store to tracking fugitives and even supporting wildlife conservation efforts or remote operations.

Sauron Overview - Create Task

1. Select a video
2. Enter a task name
3. Select a model
4. Adjust Model Parameters
5. Create Task

Sauron Overview - Process Task(s)

On the home page, a user is able to process tasks, select a dropdown to view task information, and navigate to the task query or task upload page

Sauron Overview - Query Task(s)

• Select task(s) on the left
• Select label(s) on the right
• Click ‘view’ on the lower center panel
• Click a timestamp on the lower center panel

Solution Approach

• Models use a configuration file. Provides flexibility between tasks
• Parameters can be tuned to achieve tradeoffs between speed, accuracy, and label volume
• Inference results are output as a JSON file. Provides a standard format for processing output.
• Allows future flexibility, and easy to consume by JavaScript and Python
• Models print progress to stdout so that progress can be tracked by another process
• Used in the user interface for the progress bar
• Video player bounding boxes drawn using canvas element
• Native element with easy-to-use and well-defined interfaces
• Models used: YOLOv8, MobileNet SSD, Faster-RCNN ResNet50, EfficientDet D0

Empirical Insights

EfficientDet D0 leads the models. Faster RCNN ResNet50 and MobileNet V1 SSD are on the lower end of performance. MobileNet V1 SSD is significantly faster than other models and Faster RCNN is the slowest for inference. We expect that Precision increases as IoU increases, it does not.

Empirical Insights

Similarly, EfficientDet D0 leads the models in average recall. Faster RCNN ResNet50 and MobileNet V1 SSD are on the lower end of performance. We expect recall to decrease as IoU increases.

Application Demonstration

Application Limitations

• Did not implement ensembling due to focusing on building out application functionality
• Could implement ensembling by calculating IoU between every model for a specific frame. Then we use an IoU threshold, and if above a particular threshold, we draw the bounding box.
• Application can only process a single task at a time
• It might make sense to be able to queue tasks for processing, but not process them at once. Processing in FIFO order might be just as effective in using compute resources if the single task is sufficiently parallelized while performing inference.
• Stride affects the effectiveness of drawing bounding boxes, there is no interpolation between frames that do not contain bounding boxes
• For consecutive frame strides, calculate IoU and if find an IoU above a threshold, then use a linear interpolation function to fill in bounding boxes for missing frames. The IoU threshold is only effective depending on stride and speed of objects in the video.

Thank You

References

[1] Shao, S., Zhao, Z., Li, B., Xiao, T., Yu, G., Zhang, X., & Sun, J. (2018). CrowdHuman: A Benchmark for Detecting Human in a Crowd. arXiv Preprint arXiv:1805. 00123.

[2] Jocher, G., Chaurasia, A., & Qiu, J. (2023). Ultralytics YOLOv8 (Version 8.0.0). Retrieved from https://github.com/ultralytics/ultralytics

[3] Bradski, G. (2000). The OpenCV Library. Dr. Dobb’s Journal of Software Tools.

​

[1] Dataset used in evaluation of models

[2] Source for MobileNet, Faster-RCNN, and EfficientDet

[3] Source for YOLOv8