1 of 58

Project FireFly

Preliminary Design Review I March 14 2022

1

Team:

Arjun Chauhan, Kevin Gmelin, Sabrina Shen, Manuj Trehan, Akshay Venkatesh

Main Stakeholders:

Sebastian Scherer, Andrew Jong

2 of 58

Purpose

In the US, in 2020 alone:

There were 58,950 wildfires compared with 50,477 in 2019
About 10.1 million acres were burned in 2020, compared with 4.7 million acres in 2019
Six of the top 20 largest California wildfires fires occurred in 2020
17,904 Structures torched (54% residences)

2

3 of 58

Outline of the Presentation

3

1

Project Description

Use Case

2

System-level Requirements

3

Functional Architecture

4

Cyberphysical Architecture

5

Subsystem Descriptions

6

Current System Status

7

Project Management

8

4 of 58

Project description

4

1

Project Description

Use Case

2

System-level Requirements

3

Functional Architecture

4

Cyberphysical Architecture

5

Subsystem Descriptions

6

Current System Status

7

Project Management

8

5 of 58

User Needs

Firefighters need continuous information about the environment to plan their approach e.g.

They need to know exit routes are unavailable
A system which offers continuous monitoring

5

Mapping of prescribed fire perimeter in open-field environment using a thermal and RGB camera
Autonomous flight between takeoff and landing
Stream out fire information and diagnostics
Basic visualization of fire data received

Scope

6 of 58

Use case

6

2

1

Project Description

Use Case

2

System-level Requirements

3

Functional Architecture

4

Cyberphysical Architecture

5

Subsystem Descriptions

6

Current System Status

7

Project Management

8

7 of 58

Primary Use Case:

A firefighter inputs a GPS estimate of a wildfire location into the drone system. The drone autonomously takes off, and flies to the approximate GPS location. On reaching the location, the drone begins detecting the wildfire by segmenting the fire and the perimeter of the fire using a thermal camera. This information is sent back to the ground station for visualization. After 10 minutes, the drone heads back and lands at its original takeoff position.

7

“

8 of 58

System-level Requirements

8

3

1

Project Description

Use Case

2

System-level Requirements

3

Functional Architecture

4

Cyberphysical Architecture

5

Subsystem Descriptions

6

Current System Status

7

Project Management

8

9 of 58

Requirements

9

Performance

Detect fire grids with a minimum resolution of 2.5m x 2.5m
Update fire map for user at least every 10 sec
Operating range greater than 200 m
Detect fires with temperature of at least 315ºC

Non-functional

Meet FAA regulations for a sUAS
Withstands conditions near wildfire
Easy to use
User friendly ground-station interface

10 of 58

Performance Requirements

10

Update fire map for user at least every 10 sec

Detect fire grids with a minimum resolution of 2.5m x 2.5m

M.P.1 Provide fire map updates of at least 25 m x 25 m at a spatial resolution of 0.5m x 0.5m for users at least once every 10 sec

Operating range greater than 200 m

M.P.2 Have an operating range greater than 250 m euclidean from ground station

�

Detect fires with temperature of at least 315ºC

M.P.3 Localize 5 fire hotspots with absolute accuracy of ± 5 m in a 10,000 m² region within 15 mins with less than 20% false positive rate and less than a 20% false negative rate

�

From Conceptual Design Review

Updated

Since then, we have made some modifications to our requirements. The updated performance requirements are showed on the right hand side of the table. For the first requirement we have ___, this is a combination of previous requirements that we had for minimum resolution and update rate. Nest we have ___, which is a slight change from the previous requirement of 200m which was implemented due to conversations with our stakeholders. Finally we have entirely dismissed our requirement for ___ and instead have put in place a new requirements which is that the system should ___. This change is due to the fact that the previous requirement was formulated with the combustion point of wood in mind, however we have since realized that many fires would be burning fuel that has a lower range of variable combustion values we have decided to make a more flexible requirement. Furthermore, as we developed our spring validation demonstration we were able to come up with more detailed performance requirements for our system.

M.P.3 Provide fire map updates of at least 25 m x 25 m at a spatial resolution of 0.5m x 0.5m for users at least once every 10 sec

M.P.1 Have an operating range greater than 250 meters (Change due to stakeholders)

(Combustion point was )

M.P.2 Classify fire pixels from thermal images with accuracy of 70%

M.P.4 Localize 5 fire hotspots with absolute accuracy of ± 5 meters and relative accuracy of ± 2.5 meters in a 10,000 square meter region within 15 minutes with 0 false positives and 0 false negatives

(due to SVD definition)

11 of 58

Non-Functional Requirements

11

Withstands conditions near wildfire

Meet FAA regulations for a sUAS

M.N.1 Meet FAA regulations for a sUAS

Easy to use

M.N.2 Utilize angled cameras so that wildfire can be monitored without having to fly the sUAS directly above the fire in the smoke

�

User friendly ground-station interface

M.N.3 Easy to use

M.N.4 User friendly ground-station interface

�

From Conceptual Design Review

Updated

12 of 58

Functional Architecture

12

4

1

Project Description

Use Case

2

System-level Requirements

3

Functional Architecture

4

Cyberphysical Architecture

5

Subsystem Descriptions

6

Current System Status

7

Project Management

8

13 of 58

Functional Architecture

13

14 of 58

Cyberphysical architecture

14

5

1

Project Description

Use Case

2

System-level Requirements

3

Functional Architecture

4

Cyberphysical Architecture

5

Subsystem Descriptions

6

Current System Status

7

Project Management

8

15 of 58

Cyberphysical Architecture

15

16 of 58

System and Subsystem Descriptions

16

6

1

Project Description

Use Case

2

System-level Requirements

3

Functional Architecture

4

Cyberphysical Architecture

5

Subsystem Descriptions

6

Current System Status

7

Project Management

8

17 of 58

Overall System Representation

17

DJI M600 Pro
FLIR Boson uncooled thermal camera
Nvidia Jetson Xavier
RFD 900
Ground station User Interface

18 of 58

Overall System Graphical Representation

18

19 of 58

Hardware Subsystem

19

Drone

Optics

Onboard Computer

20 of 58

Hardware Subsystem

20

System Electrical Architecture

21 of 58

Current Status: Drone

21

Drone platform

Switched to DJI M600 Pro from DJI M100

22 of 58

Current Status: Drone

22

Incorporated optics and compute platforms onto drone

23 of 58

Current Status: Optics

23

First pass:

FLIR Boson
Relative uncooled thermal camera
Incorporated into system
Poor test results in daylight; probably due to IR crossover phenomenon

24 of 58

Current Status: Optics

24

25 of 58

Current Status: Optics

25

Current efforts:

Move to Seek radiometric thermal camera
Reports absolute temperature per pixel
Current testing has given us reason to believe this will be better
Currently in process of migrating to Seek camera

26 of 58

Current Status: Optics

26

27 of 58

Software Subsystems

Fire Perception

Mapping

Telemetry and Ground Station

Planning and Control

27

28 of 58

Fire Perception

28

Binary classification of images

Image semantic segmentation

Data collection and labelling

29 of 58

Fire Perception : Current Status

29

Labelling

Completed labelling of 1078 out of 1955 images collected during test flights

Image semantic segmentation

UNet based segmentation

Signal Processing

30 of 58

Fire Perception : Current Status

30

UNet processing based segmentation

Semantic segmentation on limited dataset
Trained on 620 images, validation set of 177 images, and test set of 89 images
Hyperparameter tuning ongoing

31 of 58

Fire Perception : Current Status

31

Signal processing based segmentation

Find histogram of processed image
Detect relative peaks in the histogram
Determine which peak is caused by the fire
Identify pixels associated with the peak and designate the pixels as fire

32 of 58

Fire Perception : Current Status

32

May have to discard all of this if we move to radiometric camera as we will directly get temperature values

33 of 58

Software Subsystem: Mapping

Camera Calibration

Establish coordinate transform (extrinsic camera matrix) relating the camera frame to drone IMU

Coordinate Frame Management

Takes in current world frame location of drone to generate transforms from known reference location in world frame

Projection and Bayes Filter

Uses camera matrices and coordinate frame transforms to project each pixel into a bin in pre-generated grid on region of interest
Bin values are updated using a bayes rule update to generate fire occupancy grid

33

34 of 58

Software Subsystem: Mapping (Current Status)

Camera Calibration

Performed image gathering sequence in front of checkerboard and used Kalibr to establish the transform

TO DO: Use openCV for intrinsic calibration

34

35 of 58

Software Subsystem: Mapping (Current Status)

Projection and Bayes filtering

Pixels have been projected onto locations on the ground

TO DO: Publish grid updates for transmission to ground station

35

36 of 58

Telemetry and Ground Station

36

Communication over MAVLink radio link

Data

Take sample (for SVD min-viable demonstration)
Grid cell map origin reset
Grid cell indices to be toggled
Diagnostic information

Ground station UI to provide graphical visualization of grid cells marked with presence of fire over a map

37 of 58

Telemetry and Ground Station: Current Status

37

Communication over MAVLink radio link established during test flights over a range of ~1500 ft

Ground station UI and integration with MAVLink to be done

38 of 58

Planning and Control

38

Generate an autonomous flight path in phases:

Take-off and landing
Locate fire after take-off
Follow fire-front after locating fire
Implement a state machine to toggle between different behaviour modes to enable sampling of multiple fire locations

Current status: Architecting this and identifying the algorithms to be used is a task we have set aside for majority of the fall semester

39 of 58

Software Subsystem: Architecture

39

40 of 58

Current System Status

40

7

1

Project Description

Use Case

2

System-level Requirements

3

Functional Architecture

4

Cyberphysical Architecture

5

Subsystem Descriptions

6

Current System Status

7

Project Management

8

41 of 58

Path to Desired Final State

Hardware:

Incorporate radiometric camera

Software:

Fire Perception:

Replace existing approaches with simple thresholding using radiometric camera input
Generate estimates for false positive and false negative rates for Bayes rule update

Mapping:

Estimate camera intrinsic matrix
Complete fire occupancy grid publisher for transmission to ground station
Perform camera-IMU calibration

Telemetry and Ground Station:

Develop ground station UI
Synthesize radio communications packet format

Planning and Control:

Design and develop

41

42 of 58

Project management

42

8

1

Project Description

Use Case

2

System-level Requirements

3

Functional Architecture

4

Cyberphysical Architecture

5

Subsystem Descriptions

6

Current System Status

7

Project Management

8

Project management (6)

a. Work Breakdown Structure (1) . Present a high-level summary of the three-level Work

Breakdown Structure you developed in the Systems Engineering class. Save the WBS details

for the Critical Design Review Report (document) due at the end of the semester.

b. Schedule (0.8). Don’t present anything that is too detailed to see or talk about, such as a detailed

(as opposed to a summary and readable) Gantt Chart. Answer these key questions:

i. What are the major system development milestones in the remaining schedule?

ii. Are you behind, ahead of, or on schedule? If behind, how will you catch up?

c. High-level test plan (2)

i. Use a table to present your test plan as a set of capability milestones for the three

(3) remaining Progress Reviews (PR) in the spring semester (PR 3, 4, and 5-6, since 5-

6 are the Spring Validation Demonstration and its encore) and for roughly the ends

of each of the four (4) months (Aug-Nov) in the fall semester. Indicate briefly how

these capabilities will be tested.

ii. Spring and Fall Validation Demonstrations (SVD/FVD). These are the final two (2)

Progress Reviews in each semester, to be presented in lab demonstrations lasting

no more than 30 minutes per team. Include them in summary form in the table

above, but also describe them separately in somewhat greater detail than the other

capability milestones, including these essential elements:

A. The test location.

B. The sequence of events.

C. The quantitative performance metrics that your system will be measured

against.

D. Graphical depiction(s) to illustrate your SVD/FVD.

The SVD/FVD you present in the PDR should be as concise, clear, and graphical as

possible. You will provide a more detailed description in the SVD/FVD one-pagers

you are also required to submit as part of the PDR assignment.

d. Budget. Answer these key questions (0.7):

i. What is your total budget?

ii. What are the big-ticket items that comprise the majority of your budget?

iii. How much/what percentage have you spent to date?

e. Risk management (1.5)

i. Using techniques learned in the Systems Engineering class, present the following (an

example of each is given below):

A. A Risk Management table (Figure 1) with Risk ID, Risk, Requirement, Type,

Likelihood, Consequence, Mitigation.

B. A Risk Likelihood-Consequence Table (Figure 2).

43 of 58

Work Breakdown Structure

43

44 of 58

Schedule Overview

44

45 of 58

High-Level Test Plan

45

ID/Date	Capability Milestone	Test / Demonstration
PR3, 3/23	Fire Segmentation	Segmentation accuracy of at least 70%
PR3, 3/23	Occupancy Grid Mapping	Euclidean distance of a projected ground control point is less than 2 meters away from the point’s ground truth position
PR4, 4/6	Telemetry and Ground Station Visualization	Successful transmission and visualization of fire map updates over radio to the ground station of a 25m x 25m area at 0.5m x 0.5m resolution at 0.1Hz
SVD, 4/20	Full Teleoperated System	Successful mapping of 5 fire hotspots under teleoperated control
August	Waypoint Following	Drone autonomously moves to gps setpoints
September	Coverage Planner	Drone samples entire user-specified area
September	Fire-Front Following Planner	Drone follows fire-front in simulation
October	Autonomous Takeoff and Landing	Drone autonomously takes off and lands
November	Fully Autonomous System	Successful mapping of 5 fire hotspots without human intervention

46 of 58

Spring Validation Demonstration

46

Test location	Hawkins parking lot
Procedure	Spread out and place the fire pits in the parking lot of Hawkins Measure and record the location of the fire pits using the GQ7 RTK receiver Set up and light fires in the fire pits Fly drone to sweep entire hawkins parking lot with DJI M600 Pro cameras within 15 minutes Analyze accuracy of hotspots displayed in UI compared to RTK measured ground truth Put out fires using water jug and clean up ash into ash bucket
Verification Criteria	At least 80% of hotspots have a positive detection grid cell within 5 meters of their location At least 80% of positive detections are within 5 meters of a hotspot

47 of 58

Fall Validation Demonstration

47

Test location	Hawkins parking lot
Procedure	Spread out and place the fire pits in the parking lot of Hawkins Measure and record the location of the fire pits using the GQ7 RTK receiver Set up and light fires in the fire pits Drone flies autonomously to sweep entire hawkins parking lot with DJI M600 Pro cameras within 15 minutes Analyze accuracy of hotspots displayed in UI compared to RTK measured ground truth Put out fires using water jug and clean up ash into ash bucket
Verification Criteria	At least 80% of hotspots have a positive detection grid cell within 5 meters of their location At least 80% of positive detections are within 5 meters of a hotspot

48 of 58

Budget

Total Budget: $5000

Big Ticket Purchases:

Mechanical Hardware: $321
SSDs: $220
Fire Setup (Fire pit, Firewood, etc.): $172

Percentage Spent: 14%

48

49 of 58

Budget

A lot of components retrieved from MRSD inventory and AirLab inventory
Big ticket items that we did not have to buy:

49

Item	Approximate Cost	Retrieved from
DJI M600 Pro	$7000	MRSD Inventory
FLIR Boson Thermal Cameras	$1500 per camera	AirLab Inventory
NVIDIA Jetson Xavier NX	$1700	AirLab Inventory
SEEK Thermal Camera	$700	MRSD Inventory

50 of 58

Risk Management Table

50

Risk #	Risk	Reqt.	Type	Description	LH	CQ	Risk Reduction Plan
1	Fire-segmentation does not work accurately	M.P.1, M.P.3	Software	The fire-segmentation pipeline does not work accurately, or fails to detect fires	3	4	Use multiple thermal cameras Switch to a radiometric camera Try different segmentation approaches
2	Loss of GPS signal, position information	M.N.3, M.P.1, M.P.3	Hardware,Software	Loss of GPS signal System not able to gather useful data System detects something else as a fire Fire not visible to the drone on reaching the set GPS location	3	3	Add a tele-op override capability if a fire is not detected, or position information is lost We legally are not allowed to lose line of sight

LH: Likelihood

CQ: Consequence

1. Fire-segmentation does not work accurately

During flight, the fire-segmentation pipeline does not work accurately, or fails to detect fires. This could be due to various reasons, including bad thermal camera output

Use multiple thermal cameras to check which gives the best output

Switch to a radiometric camera which will give absolute temperature values and make segmentation easier

Try different segmentation approaches. We have tried a U-Net based segmentation model, and a signal processing approach, to compare the output and see what performs better

2. Loss of GPS signal/ Loss of position information

Loss of GPS signal due to reasons such as obstructions, GPS module failure. The system is not able to gather useful data mid-flight, to detect the fire. The system detects something else as a fire. Fire is not visible to the drone when it reaches the entered GPS location.

Add a teleop override capability if a fire is not detected, or position information is lost

We legally are not allowed to lose line of sight

51 of 58

Risk Management Table

51

Risk #	Risk	Reqt.	Type	Description	LH	CQ	Risk Reduction Plan
3	High speed winds/Bad weather during SVD	M.N.3, M.P.3	Situational	Not able to demo our system due to bad weather, high speed winds, etc.	3	3	Record multiple successful demo runs and ROS bags before the SVD as backup
4	Mapping pipeline does not project accurately	M.P.1, M.P.3	Software	The pixel to ground projections obtained from the mapping pipeline are not accurate	3	4	Attach an RTK unit to the drone Improve camera calibration

LH: Likelihood

CQ: Consequence

3. High speed winds/Bad weather during SVD

Not able to demo our system on the day of the SVD due to bad weather, high speed winds, sensor failure etc.

Record multiple successful demo runs before the SVD as backup

4. Mapping pipeline does not project accurately

The pixel to ground projections obtained from the mapping pipeline are not accurate. This could be due to a build up of errors from all the measurements and calculations taking place before this step. E.g inaccuracies in GPS data, segmentation model not accurate

If we face this issue, we plan to attach an RTK unit to the drone. This will greatly reduce the GPS errors, and will help in reducing the pixel projection errors as well.

Improve camera calibration. There could be errors due to changes in the position of the camera, since we currently have an adjustable mount. We can fix that and improve calibration.

52 of 58

Risk Management Table

52

Risk #	Risk	Reqt.	Type	Description	LH	CQ	Risk Reduction Plan
5	Camera system not working	M.N.2, M.P.3	Hardware, Software	Degraded thermal camera performance due to hardware damage and wear and tear of existing hardware Thermal crossover leading to bad output Hazy RGB video feed due to vibrations	2	3	Long term mitigation : Get a new thermal camera, or switch to a radiometric camera Current implemented mitigation : Disable non-uniformity correction to prevent ghosting event scene Add dampers to RGB camera mount

LH: Likelihood

CQ: Consequence

53 of 58

Risks

Fire-segmentation does not work accurately

Loss of GPS signal/Loss of position information

High speed winds/Bad weather during SVD

Mapping pipeline does not project accurately

Camera system not working

53

Consequence

Likelihood

1,4

2,3

5

54 of 58

Progress Review 3

54

Capability Milestone	Test / Demonstration
Fire Segmentation	Create a ROS node which publishes fire pixel values with a segmentation accuracy of at least 70%
Occupancy grid mapping with calibrated intrinsic and extrinsic parameters of thermal camera	Euclidean distance of a projected ground control point is less than 2 meters away from the point’s ground truth position

55 of 58

Progress Review 4

55

Capability Milestone	Test / Demonstration
Telemetry	Onboard computer can transmit fire map updates over radio to the ground station of a 25m x 25m area at 0.5m x 0.5m resolution at 0.1Hz
Ground Station Visualization	Ground station can display the latest updates to the fire map

56 of 58

Fall Progress Reviews (Projected)

56

Month	Capability Milestone	Test / Demonstration
August	Waypoint following Algorithm determined for fire-front following planner	Drone autonomously moves to gps setpoint
September	Coverage planner Fire-front following planner	Drone samples entire specified area Drone follows fire-front in simulation
October	Autonomous Takeoff/Landing Full System Integration	Drone autonomously takes off and lands
November	Fully autonomous fire-mapping	System successfully completes FVD

57 of 58

57

58 of 58

Thank you!

ANY QUESTIONS?

58