1 of 73

1

Self-Driving Car

SUPERVISED BY:

Dr\ IBRAHIM L. ABDALLA

Dr\ AZHAR A. HAMDY

Team Members:

Ahmed Emad Mohamed Ahmed

Ayman Ezz El-Din Mohamed

Ayman Yousef Esmaiel El-Gammal

Ahmed Mohamed Abo-Yehia

Hassan Mohamed Hassan

Salma Fahmy El-Shabrawy Fahmy

Norhan Mohamed Mohamed

Sara Abd-Elnabi Mohamed

Ahmed Maher Abdel-Fattah

Omar Soliman Elsayed

2 of 73

Outline

  • Introduction
  • Project main idea
  • Project main parts
  • Conclusion
  • Future work
  • Results

2

AI part

Embedded part

Cloud part

3 of 73

Introduction

3

4 of 73

Introduction

  • Self-driving cars are autonomous vehicles that can drive through traffic all by themselves. As unbelievable or far-fetched as it may sound, it may just be the greatest technological revolution of the near future.
  • It takes most of the responsibilities of the driver, providing a more manageable control over it.
  • Imagine a life with no traffic accidents, no road delays, no traffic jam. This life can be provided with self driving cars.
  • Simply, self-driving car is a computer-controlled car that drives itself.
  • Applying the AI field and embedded systems field to the smart cars will produce a new generation of cars that are able to drive itself.

​

​

4

5 of 73

Introduction

  • SAE Levels of Automation:

​

​

5

Source: https://www.nhtsa.gov/technology-innovation/automated-vehicles-safety

6 of 73

Introduction

Basic Physical idea of an Autonomous Vehicle:

6

  • Global Positioning System (GPS)
  • Light Detection and Ranging (LIDAR)
  • Radar Sensors
  • Cameras (Video)
  • Ultrasonic Sensors
  • Central Computer

7 of 73

Project main idea

7

8 of 73

Project main idea

  • Designing a self-driving car with the following features:
    • Detect the road lanes.
    • Take a decision of steering angle based on the lane detection results.
    • Detect objects (stop sign, traffic light, pump signs).
    • Avoid obstacles in front of it.
    • Collect data about the road anomalies using MPU6050 Gyroscope sensor.
    • Store the anomalies coordinates on cloud. (future feature)

​

8

9 of 73

Project main parts

9

10 of 73

Project main parts

  • As you may notice the project consists of the fusion of three different fields:
      • Embedded System field.
      • AI field (deep learning).
      • Cloud computing field.

10

11 of 73

Project main parts

  • Overall diagram:

11

Part 1

Part 5

Part 2

Part 3

Part 4

12 of 73

Project main parts

  • Part 1 of the Project: ( Computer Vision Part)
    • Hardware component:
      1. Raspberry pi 4.
      2. Camera raspberry pi module V2 .
      3. SD Card 64 GB
    • Software component:
        • Raspbian.
        • Open CV
        • TF
        • Pandas

​

​

12

13 of 73

Computer Vision part:

13

Image

Collect data

Labelling

Prepare data

Pre-trained model

Transfer learning

Test model

Object detection

Prepare data

(Image Processing)

Lane detection

Calculating the steering angle

Steering Angle

Collect data

Data visualization

Build model

Train model

Test model

Predict steering angle

Behavioral

cloning

14 of 73

Computer Vision part:

1- Lane Detection: Build an algorithm to detect the lane lines in the track in the front of the car.

      • Edge detection:- Identifying sharp changes in intensity in adjacent pixels.

- convert to gray scale: convert from 3 channel image to one channel. (less data)

​

​

14

15 of 73

Computer Vision part:

1- Lane Detection:

      • Edge detection:-

- Reduce noise by applying Gaussian blur filter.

15

Gaussian Blur

16 of 73

Computer Vision part:

1- Lane Detection:

      • Edge detection:-

- Identify edges (Canny function) input :gray , output : edges .

important parameters (high_threshold & low_threshold)

16

canny

17 of 73

Computer Vision part:

1- Lane Detection:

      • Edge detection:-

- region of interest: focus on a specific region in the image to not confuse the algorithm.

​

​

​

​

​

bitwise_and =

17

18 of 73

Computer Vision part:

1- Lane Detection:

2. Find lanes:-

​

- Haugh transform:

​

​

​

A quick summary of Haugh Transform

​

18

Image Space

Haugh Space

19 of 73

Computer Vision part:

1- Lane Detection:

2. Find lanes:-

​

- Haugh transform:

​

​

​

Haugh Transform follows Vote Technique

19

20 of 73

Computer Vision part:

1- Lane Detection:

2. Find lanes:-

      • Haugh transform:

But the form Y=mX+b cannot represent vertical lines in Haugh Transform So we're going to deal with the polar coordinates.

​

​

​

20

21 of 73

Computer Vision part:

1- Lane Detection:

2. Find lanes:-

      • Haugh transform:

Lets look for an example

​

​

21

22 of 73

Computer Vision part:

1 - Lane Detection:

2. Find lanes:-

      • Haugh transform:

Our result

​

​

22

23 of 73

Computer Vision part:

1- Lane Detection:

2. Find lanes:-

      • Haugh transform:

Optimization : We will calculate the average slope of the set of these lines and draw the best fit line.

By the slope of the lanes . Left_fit when slope < 0 & right_fit when slope > 0 then we take the average

​

​

​

Finally we have got

the two lanes

​

23

24 of 73

Computer Vision part:

2- Steering angle calculation:

​

  • Two ways :-

​

1- joystick

2- from the two lane lanes slope

24

25 of 73

Computer Vision part:

2- Steering angle calculation:

Calculate steering angle of the two lanes lines.

line = [x1,y1,x2,y2] , The equation to represent

the line is y = mx+b.

where: m : slope & b intercept.

​

25

θ

(X2,Y2)

(X1,Y1)

tan(θ) =

Opposite

Adjacent

Opposite = X1 – X2

Adjacent = H/2

θ = atan

Opposite

Adjacent

26 of 73

Computer Vision part:

2- Steering angle calculation:

Calculate steering angle of the two lanes lines.

problem: what if the car sees just one lane!?

the solution: we will take the same slope

like the lane line For the steering angle.

26

27 of 73

Computer Vision part:

2- Steering angle calculation:

Calculate steering angle of the two lane lines.

27

  • We consider the straight forward as 90 degree.
  • Angles <90 as turn left.
  • Angles > 90 as turn right.

Make car more stable for sudden change in angles.(ML)

28 of 73

Computer Vision part:

behavioral cloning : is a method by which human sub cognitive skills can be captured and reproduced in a computer program. (AI)

Artificial intelligence (AI) : refers to the simulation of human intelligence in machines that are programmed to think like humans and mimic their actions.

28

29 of 73

Computer Vision part:

Machine learning classifications :

​

1- supervised learning:

Classification & regression problems.

29

30 of 73

Computer Vision part:

Machine learning classifications :

​

1- unsupervised learning:

Clustering problems.

30

31 of 73

Computer Vision part:

Machine learning :

- Main problems solved By ML

31

32 of 73

Computer Vision part:

Machine learning :

  • In our project the predicted output is a numeric value “the steering angle”

(regression problem) & we will use supervised machine learning

32

The input

Expected output

100

33 of 73

Computer Vision part:

CNN(convolutional neural network ): is a class of deep neural network, most commonly applied to analyze visual images.

  • Use multiple kernels to extract more features.

​

​

33

34 of 73

NVIDIA Model

34

Input

Extract

35 of 73

Save This Data: (Steering Angle) : To Build Model.

Processing:

​

- Move The car on the track.

- Read frames from Video.

- Calculate Steering angle in each frame.

- Store this steering in file.

- We will train our model on 5k images

​

​

​

​

​

​

35

36 of 73

Save This Data: (Steering Angle) : To Build Model.

Processing:

* data visualization

* Split into training and validation sets.(4k,1k)

* Image augmentation.

​

​

�

​

​

​

​

​

36

37 of 73

Save This Data: (Steering Angle) : To Build Model.

Processing:

image preprocess: we also need to change our images into the color space and size that the NVIDIA model accept.

​

* Crop the top half of the image.

* Convert to YUV color space

* Resize the image to the required size (66,200)

​

​

​

�

​

​

​

​

​

37

38 of 73

Build the Model: (Steering Angle)

Processing:

Training the model using Google colab software.

​

​

​

​

​

​

​

�

​

​

​

​

​

38

39 of 73

Object Detection

The signs to be detected and then making decisions depends on the movement of the car.

39

Stop Sign Detection

Traffic Lights Detection

Bump Sign Detection

40 of 73

  • The tool that we used for object detection is Cascade Trainer GUI
  1. Stop Sign Detection

1.Capture positive & negative samples

40

Positive Samples

Negative Samples

41 of 73

2.Convert images to grayscale and crop positive samples

41

42 of 73

3.Modify the Parameters �

1. Pre-calculation buffer size

2. Number of Threads

3. Acceptance Ratio break value

4. Sample width & height

5.Feature Type

​

​

42

43 of 73

4.Wait for the training to complete

​

​

​

​

​

​

​

​

​

​

​

​

​

​

  • After training “classifier” folder contains XML files that are created during different stages of training. There is a “cascade.xml” is the actual cascade classifier. This file will use for testing our model.

43

44 of 73

5.Test the model

​

​

​

​

​

​

​

​

​

  • Select the “Cascade.XML” file , positive samples and the directory which new images will be saved in.

44

45 of 73

5.Result

​

​

​

​

​

​

​

​

​

​

​

2. Traffic Lights Detection

  • We will do the same steps as for the stop sign, from collecting pictures and training the model, to finally get to detect the traffic light and make a decision based on the color of the LED.

​

45

46 of 73

​

​

​

​

​

​

​

3.Anomaly Detection

  • We will do the same steps as for the stop sign, from collecting pictures and training the model, to finally get to detect the Anomaly Sign and make a decision based on reducing the speed of the car.

​

46

47 of 73

Project main parts (Embedded part)

  • Part 2 of the Project:
    • Hardware component:
      1. ATmega32a microcontroller.
      2. L298 motor driver.
      3. The car and its motors.
      4. USBasp burner to burn the code.

​

    • Software component:
        • Microchip Studio IDE.
        • AVRDUDESS.

47

48 of 73

Project main parts (Embedded part)

  • Part 2 of the Project:

48

ATmega32a MCU

L298 motor driver.

The car

USBasp

49 of 73

Project main parts (Embedded part)

  • Processing limitations:
    • The Atmega32a microcontroller offers only 16MHz, but the Tiva™ C Series architecture offers 80 MHz Cortex-M processors.
    • the Tiva™ C Series microcontrollers are the leading choice in high-performance 32-bit applications.
    • For example, The TM4C123GH6PM controller supports both asynchronous and synchronous serial communications with:
      • Two CAN 2.0 A/B controllers.
      • USB 2.0.
      • Eight UARTs.
      • Four I2C modules with four transmission speeds including high-speed mode.
      • Four Synchronous Serial Interface modules (SSI).

49

Tiva C is better choice

50 of 73

Project main parts (Embedded part)

  • Motor torque limitations:
    • If we decreased the speed of the car or decelerate its velocity the motor torque will not be able to move the car.
    • Pulse width modulation (PWM) range is from 0 to 255 but we can only use the range from 190 to 255, if the PWM is smaller than that range the car will not move.
    • Therefore, our car will not be able to move in very curved roads.
    • To overcome this problem, we should get bigger car with higher motor torque.
    • Of course, this problem will not occur in real case.

50

51 of 73

Project main parts (Embedded part)

  • Part 2 of the Project:

​

    • Code and Software simulation:

​

51

DC motor driver

52 of 73

Project main parts (Embedded part)

  • Part 3 of the Project:
    • Hardware component:
      1. MG995 servo motor.
      2. ATmega32a microcontroller.
      3. USBasp burner to burn the code.

​

    • Software component:
        • Microchip Studio IDE.
        • AVRDUDESS.

52

53 of 73

Project main parts (Embedded part)

  • Part 3 of the Project:

​

    • Code and Software simulation:

​

53

Servo motor driver

Equation of PWM frequency

ICR1 Register = TOP

54 of 73

Project main parts (Embedded part)

  • Part 4 of the Project:
    • Hardware component:
      1. Ultrasonic sensor.
      2. ATmega32a microcontroller.
      3. USBasp burner to burn the code.

​

    • Software component:
        • Microchip Studio IDE.
        • AVRDUDESS.

54

55 of 73

Project main parts (Embedded part)

  • Ultrasonic range limitations:
    • The ultrasonic maximum range is 4 meters, and we can not depend on it in real case as it has limited accuracy.
    • To overcome this problem, light detection and ranging (LIDAR) can be used.
    • Lidar uses light pulses to measure the depth of the objects in the three dimensions, so it is much faster than the ultrasonic which uses sound waves pulses.
    • For example, The maximum detection distance of TFmini-S is 12 meters. and it can work in outdoor light; the overall weight is 4.7g.

55

56 of 73

Project main parts (Embedded part)

  • Part 4 of the Project:

​

    • Code and Software simulation:

​

56

Ultrasonic driver

57 of 73

Project main parts (Embedded part)

  • Part 5 of the Project:
    • Hardware component:
      1. MPU6050 gyroscope & acceleration sensor.
      2. GPS module.
      3. Arduino microcontroller.
      4. Arduino cable.

​

    • Software component:
        • Arduino IDE.

​

57

MPU6050

GPS module

58 of 73

Project main parts (Embedded part)

  • Part 5 of the Project:
    • MPU6050 software simulation:

58

MPU6050 pinout

59 of 73

Project main parts (Embedded part)

  • Overall simulation Project:

​

59

60 of 73

Project main parts (Embedded part)

​

  • Project Code:
  • The ATmega32a code is written in C programming language.
  • Our code consists of three layers: -
    1. Microcontroller Abstraction Layer (MCAL).
    2. Hardware Abstraction Layer (HAL).
    3. Application Layer.

​

60

Application

HAL

MCAL

Visibility

https://github.com/aymanyou15/Self_Driving_Car_final

61 of 73

Project main parts (communication)

  • Raspberry pi and ATmega32a talk to each other through the Serial Peripheral Interface (SPI).

​

61

SPI driver

SPI in main.c

62 of 73

Project main parts (Cloud part)

Introduction:

  • the cloud computing platform provides easy access to the organization’s storage infrastructure and high-performance computing.
  • there are certain services and models working behind the scene making the cloud computing feasible and accessible to end users:
    1. Deployment models (public-private-hybrid).
    2. Service models (IaaS-Paas-SaaS).

62

63 of 73

Project main parts (Cloud part)

63

Cloud VS Dedicated Servers:

  1. Availability
  2. Scalability of resources
  3. Safety and security
  4. Cost efficiency
  5. level of control

​

64 of 73

Project main parts (Cloud part)

Implementation:

  • We need to make connection between the microcontroller (Raspberry pi 4) and cloud
  • (Azure)
    1. create account in Azure platform
    2. create IOT hub in azure
    3. Register a device for (Raspberry pi) in our IOT hub�

​

64

65 of 73

Project main parts (Cloud part)

Implementation:

  • Set up raspberry pi to connect Azure:

# Install below packages: -

    • sudo pip3 install azure-iot-device
    • sudo pip3 install azure-iot-hub
    • sudo pip3 install azure-iothub-service-client�
  • Configure Azure to connect Raspberry pi:
    • az extension add --name azure-cli-iot-ext�

​

65

66 of 73

Project main parts (Cloud part)

  • Implementation:
  • Configure the application (Code).
  • Create storage space to store data on it.
  • Link the storage space with our device.
  • Choose the storage you created on dashboard, click Blobs, choose you Blob service to monitor the stored data. �
  • Problems in real case:
  • Cloud require a fixed internet.
  • All roads must be covered by the Internet.

​

​

66

67 of 73

Conclusion

67

68 of 73

Conclusion

By the help of Allah and our supervisors, we can say that we are one step closer to the next generation of smart cars.

By designing this car and solving the problems we faced, we have increased our knowledge and our greed for more experiences.

As a conclusion of our project, we designed a Self Driving Car that can drive itself in certain roads (level 4).

68

69 of 73

Results

69

70 of 73

Results

  • We successfully manage to steer according to the lane detection model.
  • We successfully manage to stop the car according to the existence of stop sign.
  • We successfully manage to stop the car according to the existence of pump sign.
  • we successfully manage to stop the car according to the existence of obstacles in front of it.

70

Click on any sentence to display its video

71 of 73

Future Work

71

72 of 73

Future Work

  • Implementing the can bus module between ATmega32a and Raspberry pi.
  • Use real time operating system (RTOS).
  • Implement RPLIDAR module.
  • Implementing GPS module.
  • Use ultrasonic sensors for self parking system.
  • displaying the output on a web page.
  • making application on azure cloud and send command to our car.
  • using live online Azure Maps webinar to detect the location of our car.

72

73 of 73

73

THANK YOU