1 of 40

Real-time Driver Monitoring System for Drowsiness Detection and Accident Prevention

2 of 40

Presentation Outline

Motivation
Dataset
Model Architecture
Implementation
Results
Conclusion

3 of 40

Motivation

Drowsy Driving Accident Statistics

Num of Injuries

Num of accidents

Num of Fatalities

Drowsy Driving Accident Statistics in Korea (2022)

Visualization: Data processed and visualized using Matplotlib library in Python.

4 of 40

Dataset Structure

Data is structured in JSON

inside BoundingBox
�Cigar, Leye, Face, Phone, Reye, Mouth class exist

In each classes
Position data representing coordinate value

5 of 40

Dataset

Cigarettes, phones�dataset is not used

Eye State

("Is Opened"): The primary label for classification.

6 of 40

Dataset Structure (YOLO Input Format)

[YOLO labeling format]�

The image data used for training and its corresponding label

label format .txt

Each line represents�CLASS X Y WIDTH HEIGHT

X Y WIDTH HEIGHT is essential to normalize

CLASS

X

Y

WIDTH

HEIGHT

7 of 40

Dataset Structure �(YOLO Input Format)

X: The X-coordinate of the center of object�
Y: The Y-coordinate of the center of object�
WIDTH: The total width of the bounding box, �
HEIGHT: The total height of the bounding box,

Y

X

IMAGE

object being detected

width

center of X

center of Y

height

8 of 40

Dataset Structure

Data visualization

Class
Face
Leye	Closed	Open
Reye	Closed	Open
mouth

9 of 40

train

0: Face
1: Leye open
2: Leye closed
3: Reye open
4: Reye closed
5: mouth

10 of 40

Dataset

TRAIN DATA

77952

VALIDATION

4932

TEST

4932

11 of 40

Model Architecture

Train result

12 of 40

Model Architecture

decreasing mAP50 were eliminated.

Train result

13 of 40

Model Architecture

Chosen Architectures: YOLOv8m and YOLOv8n

Train result

14 of 40

Model Architecture

YOLOv8n�(with optimizer Adam)

0.41

0.38

epoch 80

15 of 40

Model Architecture

epoch 26

Early stopping

0.44

0.39

YOLOv8m �(with optimizer Adam)

16 of 40

Model Architecture

YOLOv8n (optimizer Adam)

purpose: Face bounding box

Input shape

1 x x x

Model�(nano)

640

3

640

17 of 40

Model Architecture

YOLOv8m (optimizer Adam)

In Face detection add margin and crop

Input shape

1 x 3 x 320 x 320

Model�medium

Output class
Face
Eye open
Eye closed
Eye open
Eye closed
mouth

18 of 40

Implementation

CUDA Not Available: Typical in-vehicle environments

Solution: Local Inference

19 of 40

Implementation

<environment & goal>�intel i5 13420H | Intel UHD graphics (internal graphic)

30FPS in given environment

Intel UHD graphics -> Rtx4060(laptop) 15x

20 of 40

Implementation

Cuda is available

YOLOv8n (face detect) [cuda]

YOLOv8m (drowsy detect) [tensorRT]

Cuda not available

YOLOv8n (face detect) [CPU]

YOLOv8m (drowsy detect) [OpenVINO]

21 of 40

Implementation

Initial Run:

Upon first execution, the model will be exported to the most suitable format for the user's hardware environment.�

Subsequent Runs:

The system will first search for an already exported model file.
If no exported model is found, it will dynamically export the model to the appropriate format based on the available hardware.
This ensures efficient model loading and execution on different systems.

22 of 40

main.py

shared memory

Multiprocess(4)

COSTANTS

GUI

Manager

Image show

Wait show_Event

show result of model thru openCV

eye state

detector

Check Awake

Every 0.5sec

Check Drowsy

Every 2.0 sec

Time

Calculator

Wait FPS_Event�

Calc fps�
Copy eyes state

(timeline by frame)

cv Camera read

Model inference

�call FPS_Event

Resize

Call Show_Event

Sound manage

Nano model�(every 2sec)

openVINO

(from v8m)

Device info

Start

Exit

23 of 40

main.py

shared memory

Multiprocess(4)

COSTANTS

GUI

Manager

Image show

Wait show_Event

show result of model thru openCV

eye state

detector

Check Awake

Every 0.5sec

Check Drowsy

Every 2.0 sec

Time

Calculator

Wait FPS_Event�

Calc fps�
Copy eyes state

(timeline by frame)

Sound manage

cv Camera read

Model inference

�call FPS_Event

Resize

Call Show_Event

Nano model�(every 2sec)

openVINO

(from v8m)

Device info

Start

Exit

24 of 40

result

Deactivated for 2.0 second

Boundbox result

Show event

25 of 40

Deactivated for 2.0 second

Boundbox result

Show event

result

loop

Time

Calculator

Wait FPS_Event�

Calc fps�
Copy eyes state

(timeline by frame)

Result Analyzed

Eyes state per frame

26 of 40

Deactivated for 2.0 second

Boundbox result

eye state

detector

Check Awake

Every 0.5sec

Check Drowsy

Every 2.0 sec

Time

Calculator

Wait FPS_Event�

Calc fps�
Copy eyes state

(timeline by frame)

Sound manage

Eyes state per frame

loop

27 of 40

Implementation

First-Time Program Execution

1.Check if processor is running

2.Init and run multiprocessor

(exception if failed)

Set Running value true

1.Check alarm running and stop

2.Check if processor is running and collect

3. sys.exit(0)

28 of 40

Implementation

Multiprocess(4)

Image show

Wait show_Event

show result of model thru openCV

eye state

detector

Check Awake

Every 0.5sec

Check Drowsy

Every 2.0 sec

Time

Calculator

Wait FPS_Event�

Calc fps�
Copy eyes state

(timeline by frame)

Sound manage

cv Camera read

Model inference

�call FPS_Event

Resize

Call Show_Event

Nano model�(every 2sec)

openVINO

(from YOLOv8m)

29 of 40

Implementation

cv Camera read

Model inference

�call FPS_Event

Resize

Call Show_Event

Nano model�(every 2sec)

openVINO

(from v8m)

Check runtime environment and load model

30 of 40

Implementation

cv Camera read

Model inference

�call FPS_Event

Resize

Call Show_Event

Nano model�(every 2sec)

openVINO

(from v8m)

Choose device (CPU, GPU, NPU) auto using openVINO toolkit & Load model

31 of 40

Implementation

cv Camera read

Model inference

�call FPS_Event

Resize

Call Show_Event

Nano model�(every 2sec)

openVINO

(from v8m)

Complie model optimized for runtime device environment

32 of 40

cv Camera read

Model inference

�call FPS_Event

Resize

Call Show_Event

Nano model�(every 2sec)

openVINO

(from v8m)

Define crop box�regarding nano model result

33 of 40

cv Camera read

Model inference

�call FPS_Event

Resize

Call Show_Event

Nano model�(every 2sec)

openVINO

(from v8m)

Store result in shared memory & call event

34 of 40

Implementation

Multiprocess(4)

Image show

Wait show_Event

show result of model thru openCV

eye state

detector

Check Awake

Every 0.5sec

Check Drowsy

Every 2.0 sec

Time

Calculator

Wait FPS_Event�

Calc fps�
Copy eyes state

(timeline by frame)

Sound manage

cv Camera read

Model inference

�call FPS_Event

Resize

Call Show_Event

Nano model�(every 2sec)

openVINO

(from v8m)

35 of 40

eye state

detector

Check Awake

Every 0.5sec

Check Drowsy

Every 2.0 sec

Sound manage

36 of 40

eye state

detector

Check Awake

Every 0.5sec

Check Drowsy

Every 2.0 sec

Sound manage

37 of 40

Implementation

eye state

detector

Check Awake

Every 0.5sec

Check Drowsy

Every 2.0 sec

Sound manage

Overlap prevention

Audio file duration => 35sec

Margin 1sec -> do not play any additional audio, until 36sec pass

38 of 40

Result

순서	Model	FPS
1 :	Yolov8m model	3~4
2 :	Onnx model	4~6
3 :	Onnx + with multiprocess	8~10
4 :	OpenVINO exported model	23
5 :	Post-processing with multiprocess	26
6 :	Yolov8n(crop) + OpenVINO	30~33

<environment>�intel i5 13420H | Intel UHD 620 (internal)

39 of 40

Result

Full code

https://github.com/hwkim-dev/Driver-drowsiness-detection

40 of 40

Thank you