Distributed Wild Bird Surveillance and Recognition System

Advisor/Client: Dr. Joseph Zambreno

Claudia Athens, Ben Simon, Pierce Adajar, Francisco Arreola

Problem Statement

Two North American Birds [Cornell Lab of Ornithology]

2

  • Our client is an amateur bird watcher
  • Wants the ability to watch birds at his home at work
  • No current solutions exist on the consumer market
  • Provide an embedded realtime bird classifying system

Emphasize: What our client wants; why he cannot get it on the current market; what we’re going to do

Requirements

Functional

Non-Functional

  • Highest quality of images
  • Realtime bird classification
  • HD video stream
  • User notifications for bird detection
  • Weatherproof Case
  • Budget of $1500
  • Portable and easy to set up
  • Clean user interface

3

Use Case

4

Design Diagram

  • Major components:
    • Hardware
    • Detection & Classification
    • Data Streaming & Storage
    • User Notifications & Frontend

5

Implementation: Hardware

6

  • Feature:
    • Deployment Platform
      • Video Processing
      • Detection
      • Classification
      • Image uploading
  • Facilitation:
    • NVIDIA Jetson TX2
    • OpenCV
    • GStreamer

Implementation: Classification

7

  • Feature:
    • Image Classification
  • Facilitation:
    • Convolutional Neural Network
      • TensorFlow
      • SqueezeNet-based Architecture

Implementation: Cloud & Streaming

8

  • Features:
    • Streaming
    • Data Storage
    • Notifications
  • Facilitation:
    • Virtual NGINX Server
    • Google Firebase
      • Cloud Functions
      • Messaging
      • Hosting
      • Storage

Implementation: Web UI

9

  • Feature:
    • Web Client
      • Streaming
      • Aviary (Detected Birds)
      • Notification Service
  • Facilitation:
    • MPEG DASH
    • HTML/JS
    • Web Push

Test Plan - Holistic

10

Bird Comes into Frame

Bird detected

  • Detection algorithm detects large, moving object
  • Board saves frame
  • Frame is passed to classifier

Image classified

  • Neural Net finds most likely classification
  • Image is labeled with classification
  • Labeled image is uploaded to cloud

User notified

  • User is notified via web notification
  • Stream is available to be viewed

Image uploaded to cloud

  • Web frontend displays image
  • Bird classification is indexed by cloud service

Testing - Component Integration

  • Hardware
    • 4k picture can be taken successfully
    • Detection & Classification runs within timing constraints
    • Meets IP-55 weather-proofing constraints
  • ML
    • Images can be classified with high accuracy
    • Model runs within timing constraints
  • Web
    • Stream can be accessed and is of high quality
    • User interface is functional
  • Cloud
    • Images can be stored & retrieved
    • Notifications are facilitated

11

Component & Integration Testing

System (End-to-End)

Testing

Acceptance Testing

Non-Functional Testing

Testing - End-to-End

  • Holistic Use-Case:
  • Bird Comes into Frame
  • Bird Successfully Detected
  • Image is Classified
  • Image is Uploaded to Cloud
  • User is Notified
  • Performance Metrics Testing

12

Component & Integration Testing

System (End-to-End)

Testing

Acceptance Testing

Non-Functional Testing

Testing - Acceptance

  • Simulated Environment Testing
    • Client Feedback
  • Field Testing
    • Data Collection

13

Component & Integration Testing

System (End-to-End)

Testing

Acceptance Testing

Non-Functional Testing

Testing - Non-Functional

  • Setup / Teardown
    • Demonstration
    • Feedback
  • User Interface / User Experience Testing

14

Component & Integration Testing

System (End-to-End)

Testing

Acceptance Testing

Non-Functional Testing

Major Project Milestones

  • Jetson TX2 flashed with software
  • Neural network implemented
  • Object detection system
  • Cloud image storage
  • Enclosure purchased and modified

15

  • Fleshed out web interface
  • User notifications
  • Communication between all components
  • System Testing

Results: Project Deliverables

  • Field-Deployable Monitoring Platform
    • 4k Camera
    • Onboard Detection & Classification
    • Touch Screen
    • IP55-Compliant Enclosure

16

Results: Project Deliverables

  • Cloud
    • Storage of all captured images
    • Storage of identification data
    • Web Push Notifications
  • Web UI
    • 1080p Video Streaming
    • Card Notifications
    • Full 4k Images

17

Results: Challenges

Challenges

  • Design & Implementation
    • Limited experience
    • Underestimating work
  • Testing
    • Large number of sub-components
  • Team
    • Distributed team members

Lessons Learned

  • Understanding our limits
  • Sandwich approach to development
  • Establish clear communication guidelines

18

Future Work

  • Improve Classification Accuracy
    • Refining architecture
  • Website Board Controls
  • Notification System
    • Notification filtering
  • Improving Web Interface
  • Distributed Cameras

19

Questions?

20

Appendices

21

First Semester Gantt Chart

22

Second Semester Gantt Chart

23

Jetson TX1 vs. Jetson TX2

Jetson TX1/TX2 Specs [NVIDIA]

24

e-CAM131_CUTX2 Specifications

25

Camera Options

26

Neural Network Architecture

27

Market Survey

Radar Bird Detection System [Unmanned Systems Technology]

Qualifying Wild Animals

[Norouzzadeh et al.]

28

International Protection (IP) Code

Solid Particle Protection [IP Code]

29

International Protection (IP) Code Part II

Liquid Ingress Protection [IP Code]

30

Resource/Cost Estimate

31

Total Budget

$1500

Jetson TX2

$600

Camera

$270

Cloud Storage

$100

Enclosure

$100

Remaining

$430

Training Results: Train Accuracy

32

Training Results: Train Loss

33

Training Results: Distribution Histograms

34

Training Results: Distribution Histograms

35

Training Results: Distribution Histograms

36

Training Results: Distribution Histograms

37

Training Results: Distribution Histograms

38

Training Results: Distribution Histograms

39

Training Results: Distribution Histograms

40

CPR E 492 Final Presentation - Google Slides