ADAPT: Drone Payload for Data Collection and Real-Time AI Processing in the Field
Ocean Sciences Meeting 2022
Website: https://kitware.github.io/adapt/
PI: Matt Brown, PhD, Kitware (matt.brown@kitware)
Co-PI: Peter Webley, PhD, U Alaska-Fairbanks
1
Supported by Federal funds under award NA21OAR0210112 from the National Oceanic and Atmospheric Administration, U.S. Department of Commerce
ADAPT Multi-Mission Payload
2
2
Image from https://www.weather.gov/otx/Our_Office
Oil Spill
An open source platform for deploying state of the art deep-neural-network computer vision in real time on small unmanned aircraft systems (sUAS).
ADAPT Multi-Mission Payload
The Problem: 2013 Galena Alaska flooding
3
River 2km away
Ice blocks and debris the size of minivans, flowing down mainstreet
Develop UAV Payload to Optimize Data Collection and Exploitation
4
Process to acquire products from 2017 sUAS flights
Optimized Data Collection and Exploitation Life Cycle
5
5
VIAME
Imaging
Sensors FOV
INS
Embedded
Computer
Data Storage
Post-Flight Data Exfiltration
Data-Collection Payload
Upload New Deep Network Models
Train Deep Network Models
Real-Time Detections
Real-Time, In-Flight Segmentation Model
6
Compression of Data Product
7
NVIDIA
Xaiver
Open Source Software and Documentation
8
Available on our Website�https://kitware.github.io/adapt/parts/
We are looking to build an open community of users
Completed System Build
9
Camera �adjustable from nadir to forward-looking
Inertial Navigation System (INS)
NVIDIA Xavier Computer
WiFi Downlink
GPS Antenna
Inertial Navigation System (INS)
System in the Field
10
Sept. 8th 2021, Test Flights Fairbanks, AK
at the Tanana and Chena Rivers
The System in Action
11
System about 2,500 ft away along Tanana River
Ground
Sky
Trees
to test live segmentation chain
The System in Action
12
Frozen Water
Background
The System in Action
13
System about 2,500 ft away along Tanana River
Ground
Sky
Trees
to test live segmentation chain
The System in Action
System compatible with beyond line of site operation
14
Nearly Beyond Line of Sight
The System in Action
Real-time, remote image inspection for quality control
15
System about 2,700 ft away
Highly Synchronized Imagery and Navigation Data
16
Comparison between camera orientation change (10 Hz) as measured from imagery via structure-from-motion (red) and INS (blue)
Analysis Over Time with Pixel-Level Registration
17
2017-05-05
2017-05-08
Significant ice breakup between observations
ADAPT Multi-Mission Payload
18
18
Image from https://www.weather.gov/otx/Our_Office
Oil Spill
An open source platform for deploying state of the art deep-neural-network computer vision in real time on small unmanned aircraft systems (sUAS).
ADAPT Multi-Mission Payload
Differentiating ADAPT: Annotation
19
Complex Topology
Novel Annotation Workflow
Sparse Human Annotations
New Training Image
Model-based annotation interpolation
Human Annotator�Corrections
20
Novel Annotation Workflow
Extant segmentation model bootstraps annotations requiring only minor corrections to rapidly achieve high-quality ground truth
21
Annotation Workflow
22
Raw Image Data
GT-Generation Model Training
Ground Truth Segmentation Annotations
Sparse Annotation
Correction
Sparse Annotation
Model Output Human Adjudication
GT-Generation Model Deployment
Thank You!
23
Please visit our website for more information�https://kitware.github.io/adapt/
This presentation was prepared by Kitware Inc. using Federal funds under award NA20OAR0210083 from the National Oceanic and Atmospheric Administration, U.S. Department of Commerce. The statements, findings, conclusions, and recommendations are those of the author(s) and do not necessarily reflect the views of the National Oceanic and Atmospheric Administration or the U.S. Department of Commerce.
Ice Segmentation Results
24
WATER
ICE
SNOW
GROUND
Legend
Potential high-risk ice breakaway!
Results using BiSeNetV2 running real-time on a Xavier AGX
Retrain Model with Corrected GT
Original Hard Negative vs Retrained
25
AI-Assisted Annotation Workflow
Human inspection of model output on unseen examples allows
26
ADAPT Multi-Mission Payload: Oil Spills
27
27
Existing Approaches
ADAPT Approach
Pipeline Inspection / Spill Detection
Active Spill Tracking
Miles-Long Undersea Pipeline
Pilot
Spotter
Remotely Piloted UAV
Relay
Raw Video
Limited Range
Inefficient Voice Comms to HQ
Oil Spill
Manually �Surveyed Coordinates
Reduced Costs, Fewer Personnel, More-Timely Knowledge
Autonomous Operation
Low-Bandwidth
Long-Range
Visible– Thermal Imagery
Rapid Survey of Georegistered Boundary of Spill Sent Directly to HQ
ADAPT
Payload
Thickness Est.
Mapping
Classification
Detection
Inefficient and
Costly
Low-Bandwidth
Long-Range
Georegistered ALERTS
Inefficient and
Costly
Manned Boat
HQ
Inefficient Voice Comms to HQ
ADAPT
Payload
Thickness Est.
Mapping
Classification
Detection
Autonomous Operation
Active Spill
Potential Spill
Miles-Long Undersea Pipeline
HQ
ADAPT payload for optimized oil spill response operations�• Working with Alaska Clean Seas to define improved spill remediation approaches