1 of 41

Team B

EmberEye

Progress Review #10

2 of 41

The Team

Kavin

Gangadhar

Jaskaran

Ishir

Shashwat

Andrew Jong�PhD, AirLab [POC]

Dr. Sebastian Scherer�PI, AirLab

3 of 41

Cool Testing Videos

4 of 41

Table of contents

01

03

02

04

Tasks planned looking forward to PR #9

Where we stand

Roadblocks & Bottlenecks

Teamwork

Progress

Challenges

Future Work

Work Distribution

5 of 41

Progress

01

Moving towards FVD

6 of 41

PR 10 Goals

  • Perception: Test thermal fusion pipeline in-flight

​

  • State Estimation: In-flight test, correct TF transforms�
  • Navigation: End-to-End testing in simulation. Deploy on NUC�
  • Hardware: Complete new platform, Flight tests (endurance)�

7 of 41

Hardware

8 of 41

Hardware | New UAS Platform

  • Airframe fully integrated and flight tested
  • Preliminary and endurance flight tests completed
  • Tested Flight Time: 16 Minutes
  • Newly discovered issues

​

9 of 41

Hardware | Modifications & Shielding

  • ​

​

10 of 41

Perception

11 of 41

Perception: Multi-hotspot localization

​

RealSense

Point Cloud

Project into thermal camera frame

Reproject pixels for 3D location

Extract ROI from fire segmentation

Realsense-

Thermal Camera

Point Cloud

Thermal Camera Feed

Segmented ROI

Hotspots Mapped

Rate on NUC: 30 Hz

12 of 41

Perception: Fire Mapping

Dataset for evaluation:

  • Distances: 1-5 meters
  • Heat Sources: heater and thermal checkerboard
  • Location: Heat sources present at all the locations of the image (to evaluate performance at corners - distortion)
    • Left -> right
    • Bottom -> top

Checkerboard: bottom-left

Checkerboard: bottom-centre

Heater: 3 meters centre

Checkerboard: 5 meter - centre

13 of 41

Perception: Fire Perception Validation and Testing

RI SQH Evaluation (Indoor)

NREC H-H Evaluation (Outdoor)

In-Flight Testing

Predicted depth: 4.917602 | Actual: 5m

Stress Testing

14 of 41

State Estimation

15 of 41

State Estimation: Setup TF tree linking RS and Base Link

  • VINS odometry comes b/w map and imu_flu�
  • Used inverse static tf to get camera_link��
  • Used inverse static tf to get base_link

For Depth Data

For Navigation

16 of 41

State Estimation: VINS in-flight evaluation

Run 2 Metrics

Run 1 (Hover) Metrics

Hotspot

Run 2

Run 1

Launch/Land

17 of 41

State Estimation: VINS with RS Infrared Cams + IMU

Find IMU �Gyroscope and Accelerometer Errors

Use TF and dt values to run VINS with RealSense IR Cameras + IMU, then evaluate

Calibrate RealSense IR Cameras + IMU to get TF + Time-Offset

closed loop trajectory >110m

18 of 41

Autonomy

19 of 41

Autonomy Manager

Autonomy Manager

Exploration Interface

Autopilot API

P2P Interface

FSM Mission Logic

Exploration Node

P2P Node

FCU

20 of 41

Autonomy: Mission Flow

Logic plan for single manager:

​

  • First is P2P nav : Select goal in rviz�
  • Check status in Manager: Goal published to P2P node�
  • Once goal is reached: Switch to wait mode�
  • Check status in Manager: Start exploration�
  • Once exploration done: Set goal as home base�
  • Check status in Manager: Goal published to P2P node�
  • Once home base reached: Finish mission and Land��

21 of 41

Autonomy: Integration

Steps:

​

  • End-to-End testing in Simulation : Done�
  • Integrate P2P and Exploration in Autonomy Manager : Done�
  • Decoupled controllers, and use Mavros API : Done�
  • Sequential testing on Hardware : In Progress�
  • Integrate fail-safes for failure cases : To Do�
  • Refine final mission and demo logic : To Do��

22 of 41

Autonomy: Integration

  • Testing planner with on-platform Realsense inside Airlab��

23 of 41

Project Management

Reinforced the deliverables and timeline.

​

More thorough PM tracking.

24 of 41

Challenges

02

Challenges, Issues and Risks

25 of 41

Issue Review

In Progress: Evaluating performance of IMU propagated VIO

Challenges

26 of 41

Issue Review

Resolved: Delinking arming and control from Planner

Challenges

27 of 41

Issue Review

In Progress: Communication and USB 3.0

Challenges

28 of 41

Issue Review

In Progress: DC-DC Regulator for NUC

Challenges

29 of 41

Risk Logs

Challenges

30 of 41

Risk Management

Challenges

31 of 41

Risk Management

Challenges

32 of 41

Future Work

03

Current State & Plans for PR#10

33 of 41

Future Work

Thermal depth estimation

Autonomy Stack

Repeatability and Generality

Deploy on Platform

In-flight Validation and Verification

Performance optimization

Failure Testing

State-Estimation

FCU odometry

fusion

Stress testing

in-flight

Hardware

Robust

Communication

Spare parts

34 of 41

Future Work:

Perception

  • Integrate all transforms with central TF tree
  • Performance optimization
  • In-flight verification and validation

35 of 41

Future Work:

State Estimation

  • Stress test VINS-Fusion on-board and in-flight
  • Fuse VINS estimates on FCU with altimeter and PX4 Odometry
  • Desirable: Loop closure testing

36 of 41

Future Work:

Hardware

  • Flight tests with new voltage regulator
  • Resolve communication issues
  • Secure spare parts

37 of 41

Future Work:

Autonomy

  • Sequential testing on hardware
  • Plan out deliberate failure cases
  • Repeatability tests
  • Robust failsafes and final intricacies for mission

​

~ this week

38 of 41

Future Work:

Logistics

  • EMI shielding material
  • Secure obstacles to simulate trees
  • Finalize demonstration setup at NREC cage

​

​

39 of 41

Work Distribution

04

Teamwork

40 of 41

Wrapping Up,

Redistributing Work

FVD

SVD

In Progress

41 of 41

Stay Safe.

Team EmberEye