1 of 43

1

2022

Developing Object

Detection for Autonomous

Obstacle Avoidance

Paul Turek | David Rovner | Marc Goulart

Anjanee Nikhila | Mingxing He | Aidan Corral

Mentors: Navid Sabbaghi | SMC Business Analytics

Brian Boogaard, Nelson.Brown |NASA

2 of 43

2

Paul Turek | David Rovner | Marc Goulart

Anjanee Nikhila | Mingxing He | Aidan Corral

Navid Sabbaghi, PhD | Program Director, SMC Business Analytics

Brian Boogaard | TTP Coordinator, NASA

Nelson Brown | Project Chief Engineer, NASA

Aerial LIDAR Applications In UAS Conflict Detection

3 of 43

LIDAR BASED OBSTACLE

DETECTION IN

UAS SYSTEMS

Saint Mary’s College of California MSBA Practicum | NASA

4 of 43

4

Saint Mary’s College MSBA Practicum | NASA

COHORT 12 | 2022

5 of 43

5

Meet the Team

Marc Goulart

Paul Turek

Ming xing He/Melanie

Currently enrolled in the Business Analytics Master’s Program at Saint Mary’s College of California. 2021 Saint Mary’s undergraduate graduate, earning a B.S. in Business Administration.

Graduated from SDSU with a B.A. in Economics. He works at SVB as an account manager and in his free time likes to play golf and go snowboarding at Lake Tahoe.

Graduated from Xiangtan university with a B. A.in Art design. Current ,Master program in Business Analytics at Saint Mary’s College of California. I’m a dog person,hiking often.

6 of 43

6

Anjanee Nikhila

Currently pursuing Master’s in Business Analytics at Saint Mary’s College of California. Earned Bachelor’s degree in Business Management & Accountancy at University of Mumbai. Free time dedicated to singing and art.

David Rovner

Aidan Corral

Current MSBA student and recent 2021 undergrad graduate in mathematics and computer science both at Saint Mary’s College of California. Currently works as a market research analyst at Deep North. Likes to workout, mix martial arts, and play basketball.

I am a Federal Law Enforcement Officer with 15 years of federal service. I am pursuing my MS in Business Analytics to change careers. I am a U.S. Army Veteran. I graduated from Cal Poly San Luis Obispo with a B.S. in Industrial Technology. In my free time, I enjoy traveling, going to concerts, hiking, going to the beach, going on random adventures, and riding my Harley .

​

Meet the Team

7 of 43

7

Table of contents

1

Introduction

2

Tools and Libraries

​

3

Data Exploration

4

Approaches

5

Results

6

Recommendations

8 of 43

8

Introduction

Background

Problem Statement

Project Goals

01.

9 of 43

9

LIDAR: Light Detection and Ranging

Background: What is LIDAR?

Light pulses are sent out, reflected off objects, and received for interpretation, mainly to measure distance.

Mapping professionals to examine both natural and manmade environments with accuracy, precision, and flexibility.

10 of 43

10

LIDAR: Light Detection and Ranging

  • LIDAR datasets are abundant and updated regularly

​

​

Background: LIDAR Datasets

Visualization of U.S. topography surveyed by aerial LIDAR (USGS).

Point clouds are sets of points that describe an object or surface. Each point contains an ample amount of data that can be integrated with other data sources or used to create 3D models.

11 of 43

11

Why Integrate LIDAR in Drone technology?

Enable drones to use as much information as possible to avoid collisions with static and dynamic obstacles

​

​

Background: Drone Autonomy

Collision avoidance systems are crucial for enabling autonomous operations for unmanned vehicles of all kinds. These systems take in data from various onboard sensors, as well as data from external sources, and calculate the best manoeuvres for the vehicle to make in order to avoid hitting an obstacle or hazard.

More advanced systems may use artificial intelligence and computer vision to perform detection and classification of objects picked up by the sensors.

12 of 43

12

Task

  • Use available datasets captured from to increase the capabilities of object detection and avoidance systems
  • Develop an algorithm that can be used to identify obstacles in areas that have been surveyed by aerial LIDAR.

​

​

Background: NASA TTP Goals

Why?

  • Commercial drones are becoming increasingly more popular
  • These drones are relatively small, and the hardware that is onboard the craft must be sparse to minimize weight

​

​

How:

  • Using publicly available LIDAR datasets, create a geodatabase of tall obstacles in urban environments that are hazardous to drones and other aerial vehicles flying at low altitudes.
  • Minimize the size of the obstacle database so that it can be stored onboard drones, depending on the flight path.

​

​

13 of 43

13

What’s happening now

Amazon’s new delivery drone scheduced start deliveries in Lockeford,California and College Station, Texas by the end of 2022.

​

The last few years have been seen a dramatic change in the drone industry. “COVID absolutely gave this industry a huge push.

14 of 43

14

https://www.digitaltrends.com/news/drone-delivery-crash-knocks-out-power-for-thousands/

https://dronedj.com/2022/03/25/amazon-delivery-drone-crash-oregon/

Current challenge

Google sister company Wing

When a Wing delivery drone on its way to deliver a food order to a customer’s home in Australia, it crashed into an 11,000-volt power line.

Amazon delivery drone

Motor failure and crashed to the ground.

15 of 43

15

What is object-detection?

A software that identifies and locates objects within an image or video.

Object-Detection Background

Uses?

Image annotation, vehicle counting, activity recognition, face detection, face recognition, video object co-segmentation.

16 of 43

16

​

​

  • A point cloud is an extremely large bundle of points that are specifically put together to take a geographical area, terrain, building or feature and plot them in to a 3d space.

​

What is a Point-Cloud?

17 of 43

17

The Intense Details of a Point Cloud

03

04

18 of 43

18

DEM (Digital Elevation Model)

A  representation of the bare earth topographic surface of the Earth  excluding trees, buildings, and any other surface objects. 

https://www.usgs.gov/faqs/what-digital-elevation-model-dem

Using a DEM raster, we can predict the elevation of areas of non-ground points.

19 of 43

19

Tools and Libraries

02.

20 of 43

20

Tools

21 of 43

21

ArcGIS Pro

A full-featured professional desktop GIS application from Esri.

Uses spatial data.

Explore, visualize, and analyze data

Create 2D maps and 3D scenes.

22 of 43

22

01

03

05

Open3D

Supports rapid development of software that deals with 3D data.

laspy

Python library for reading, modifying and creating LAS LiDAR files

Matplotlib

Static, animated, and interactive visualizations in Python

02

06

CSF

An airborne LiDAR filtering method which is based on cloth simulation.

GDAL

translator library for raster and vector geospatial data formats

Libraries

04

PDAL

C++ library for translating and manipulating point cloud data.

07

Others

numpy, pandas.

23 of 43

23

Data Exploration

03.

Object-Detection Project Data

Object-Detection Project Journey

​

24 of 43

24

Object-Detection Project Data

USGS Aerial LIDAR

DALES Semantic

Segmentation Dataset

25 of 43

25

  • This is free public data accessible to anyone.
  • Captures a large-scale of land, obstacles, roofs and tall vegetation.
  • Aircraft emits and captures light pulses.
  • Point Cloud surfaces colored by elevation.
  • First return is the top surface. Last is the bare earth.

​

USGS Aerial LIDAR

26 of 43

26

DALES Semantic Segmentation Dataset

  • Source: University of Dayton

​

  • Initial data set: 10 km2 area.

Final data set: 40 tiles @ 0.5 km2. each

​

  • Scene Types: Urban ,Suburban, Rural, and Commercial.

​

  • Object Categories : Ground (blue), Vegetation (dark green), Cars (pink), Trucks (yellow), Power Lines (light green), Fences (light blue) , Poles (orange), and Buildings (red).

​

27 of 43

27

Source: USGS Lidar Point Cloud (LPC) ARRA-CA_GoldenGate_2010_001048 2014-08-27 LAS

Cloud Compare

Visualization of an aerial LIDAR dataset of the San Francisco skyline and Bay Bridge

11M+ data points in this .LAS file

28 of 43

28

​

  • We found that current object detection algorithms on 3D point clouds was too time consuming
  • Tested object detection deep learning algorithms on jpeg images and high density LIDAR point clouds.

​

Object-Detection Project Journey

29 of 43

29

Next

We then took the LIDAR data and extracted into a 3D point-cloud.

Object-Detection Project Journey

Through downsampling, we then voxelized the data structure which is a store of geometric information in a continuous domain into a rasterized image.

30 of 43

30

Approaches

04.

31 of 43

31

Lidar Data Filtering: Previous Algorithms

Many ground filtering algorithms have been proposed, but are mathematically complex or separated ground and non-ground measurements by removing non-ground points from LiDAR datasets

Problems:

  1. The performance of these algorithms changes according to the topographic features of the area
  2. The filtering results are usually unreliable in complex cityscapes and very steep areas.
  3. Models oftentimes fail to effectively model terrain with steep slopes and large variability because they are based on the assumption that the terrain is a smooth surface

32 of 43

32

Lidar Data Filtering: Cloth Simulation

Cloth simulation filters the ground points by simulating a physical process that an virtual cloth drops down to an inverted (upside-down) point cloud.

Advantages:

  • Few parameters are used in the proposed algorithm, and these parameters are easy to understand and set

​

  • The proposed algorithm can be applied to various landscapes without determining elaborate filtering parameters

​

  • This method works on raw LiDAR data

33 of 43

33

Lidar Data Filtering in r

LiDAR technology has brought the possibility to separate the vegetation from the ground return (when it is visible) even under the canopy.

34 of 43

34

Lidar Data Filtering: Classification in r

Once we can differentiate ground points from vegetation and buildings, we can measure the height along the z-axis by subtracting the height of the ground level from the height of the obstacle.

Alternatively, we can view a cross section of just the ground level:

35 of 43

35

Results

05.

36 of 43

36

Store obstacles in an accessible database

Save results to a .csv file.

Data Preprocessing

CSF pipeline to determine ground points & base elevation.

Point Segmentation/K Means Clustering

Automate finding the optimal amount of clusters within a .las dataset.

​

Find the radius of each cluster using the max distance from each cluster centroid.

37 of 43

37

Lidar Data Filtering: Cloth Simulation

Experimental results yield an average total error of 4.58%, which is comparable with most of the state-of-the-art filtering algorithms

38 of 43

38

Lidar Data Filtering: Creating a Threshold

Filtering out points in the point cloud that are < 30 feet.

39 of 43

39

Lidar Data Filtering: K Means Clustering

Finding an optimal cluster value

40 of 43

40

Lidar Data Filtering: Exporting Results

Saving the output to a .csv

41 of 43

41

Future Research

Recommendations

​

06.

42 of 43

42

Future Research & Recommendations

Automate the process of creating accurate classification models with the cloth method, and using thresholds to create clusters of tall objects and exporting the data into a SQL database.

For irregular shapes, apply more complex methods to fit the shapes of clusters into polygons, and find the vertices of the objects (maybe CNN - pytorch/ CUDA/tensorflow).

Efficiently downsample the point cloud without loss for small/thin objects like power lines.

Use voxelization to efficiently loop through point clouds with CNNs, take voxel heights and use for step 2.

Algorithms to scrape LIDAR data and apply findings to add to obstacle database.

43 of 43

43

Questions?

Thanks for Listening!