1 of 52

A Markerless Augmented Reality Platform for User Support: 'ARabeitak' Car Maintenance App as a Case Study

1

Table Of Contents

1.

Introduction

& Problem Definition

2.

Objectives

3.

System Overview

4.

Features &

Methodology

5.

Diagrams

6.

Dataset

7.

UI Prototype

8.

Demo

2 of 52

2

Table Of Contents

1.

Introduction

& Problem Definition

2.

Objectives

3.

System Overview

4.

Features &

Methodology

5.

Diagrams

6.

Dataset

7.

UI Prototype

8.

Demo

3 of 52

3

1.1 Introduction

AR in the automotive industry has flourished in the past few years mainly targeting assembly, driving assistance, and training.

An uprising need has emerged for the automotive AR maintenance field. Thus, many companies adopted the idea of integrating AR in the automotive field.

4 of 52

4

1.2 Problem Definition

Lack of available modern and user-friendly resources for car owners.

Skimming through text-based manuals is no longer conceivable

Limited independence and ease of access to information as people:

tend to pay a visit to an expert even for the simplest maintenance issues
are not familiar with their car's full potential

5 of 52

5

2. Objectives

To develop a user-friendly application that is accessible through mobile devices and tablets to assist average-skilled car owners to easily follow the instructions for their vehicle maintenance.

To allow the detection and recognition of the different car parts, whether under-hood components or car dashboard triggers.

To overlay detailed instructions over the car parts, in the real scene, to perform a task. Instructions can be textual, pictorial, or animated 3D models.

To offer a tele-assistance feature for the car owners to seek assistance from a remote expert.

To detect dynamic gestures and trigger a warning if an incorrect gesture was detected.

To automate maintenance instructions retrieval.

6 of 52

6

3. System Overview

7 of 52

7

4.

Features &

Methodology

All your comments and concerns were taken into consideration, and we set out to address each of them.

We addressed them in 3 main tracks, that we believe added a lot of value to the product.

8 of 52

8

3.

Features &

Methodology

AR Remote Assistance

9 of 52

9

3.

Features &

Methodology

Hand Tracking & Error Detection

AR Remote Assistance

10 of 52

10

3.

Features &

Methodology

AR Remote Assistance

Hand Tracking & Error Detection

Instructions

Authoring

Automation

With GPT

11 of 52

11

3.

Features &

Methodology

AR Remote Assistance

12 of 52

12

4.1 Remote Assistance - Overview

Video Streaming

Adding AR elements

Rendering AR elements

Car owner

Technical expert

13 of 52

13

4.1 Remote Assistance - Overview

Car owner

Technical expert

Video Streaming

Adding AR elements

Rendering AR elements

14 of 52

14

4.1 Remote Assistance - Overview

Video Streaming

Adding AR elements

Rendering AR elements

Car owner

Technical expert

WebRTC Experiments

WebView API

Used a Firebase Hosting Emulator for Signaling
Used a Firestore database for token generation
Issues

AR integration in the WebView

+

15 of 52

15

4.1 Remote Assistance - Overview

Video Streaming

Adding AR elements

Rendering AR elements

Car owner

Technical expert

WebRTC Experiments

WebView
Native WebRTC

Solves the AR related issues
Issues

.NET experience
Handling the audio and video call
Signaling and hosting overhead

+

16 of 52

16

4.1 Remote Assistance - Overview

Video Streaming

Adding AR elements

Rendering AR elements

Car owner

Technical expert

WebRTC Experiments

+

Agora Choice

Streaming overhead reduced
Time to focus on AR development

17 of 52

17

Video Streaming

Adding AR elements

Rendering AR elements

Car owner

Technical expert

Challenges:

Drawing on the caller’s stream
Add elements on surfaces (hit test)

4.1 Remote Assistance - Overview

18 of 52

18

Video Streaming

Rendering AR elements

Car owner

Technical expert

Adding AR elements

Challenges:

Drawing on the caller’s stream
Add elements on surfaces (hit test)

4.1 Remote Assistance - Overview

19 of 52

19

Video Streaming

Rendering AR elements

Car owner

Technical expert

Adding AR elements

Challenges:

Sending and receiving data elements and their details through a data channel

4.1 Remote Assistance - Overview

20 of 52

20

4.1 Remote Assistance

Stream audio and video

Channel setting

ARCore integration

Add advanced drawings

Deploy a token generating server

Integrate with app + user authentication

Done

To do

21 of 52

21

3.

Features &

Methodology

AR Remote Assistance

Hand Tracking & Error Detection

Instructions

Authoring

Automation

22 of 52

22

Hand Tracking & Error Detection

23 of 52

23

4.2 Hand Tracking –

Extending Wikitude via C++ Plugins

Native Java

Wikitude SDK

JS WebView

Wikitude doesn’t offer any kind of CV/ML.

It must be extended in a way that allows us to access the inner flow of the camera feed.

Start Architect Plugin

Default Wikitude Camera

24 of 52

24

Native Java

Wikitude SDK

4.2 Hand Tracking –

Extending Wikitude via C++ Plugins

JS WebView

Wikitude doesn’t offer any kind of CV/ML.

It must be extended in a way that allows us to access the inner flow of the camera feed.

Start Architect Plugin

Default Wikitude Camera

To extend Wikitude functionality…

25 of 52

25

JNI Bridge

OpenGL Texture Buffer

YUV420_888 Image

JNI Bridge

WebView JavaScript, JNI Bridge

YUV420_888 Image

Native Java Camera2 API

Processing, and rendering

Wikitude SDK

C++ Plugins

A plugin is a class, or rather a set of classes, written in C++ that allows extending the functionality and alter the inputs and outputs of WikitudeSDK.

4.2 Hand Tracking –

Extending Wikitude via C++ Plugins

26 of 52

26

JNI Bridge

OpenGL Texture Buffer

1- A JNI Bridge is used to call a C++ function from Java that sets up a buffer.

2- When the camera frame is captured by Camera2 API, it is passed to the buffer.

3- Processing and Rendering done by Wikitude core and can be customized as we wish in C++.

YUV420_888 Image

JNI Bridge

WebView JavaScript, JNI Bridge

YUV420_888 Image

Native Java Camera2 API

Processing, and rendering

Wikitude SDK

C++ Plugins

4.2 Hand Tracking –

Extending Wikitude via C++ Plugins

27 of 52

27

JNI Bridge

OpenGL Texture Buffer

YUV420_888 Image

JNI Bridge

WebView JavaScript, JNI Bridge

YUV420_888 Image

Native Java Camera2 API

4.2 Hand Tracking –

Extending Wikitude via C++ Plugins

Processing, and rendering

Wikitude SDK

Now to integrate Hand Tracking in this workflow…

C++ Plugins

28 of 52

28

4.2 Hand Tracking -

Extending Wikitude &

Integrating MediaPipe

TextureView

OpenGL ES

TensorFlow Lite Models

Bitmap

YUV420_888 image

Native Java Code

OpenGL Texture Buffer

JNI Bridge

Processing through graphs and calculators.

CameraX API

Rendered

FrameTexture

custom camera

triggered

.SO built Library

Wikitude SDK

Processing, and rendering

JS WebView

Start

Wikitude

C++ Plugins

29 of 52

29

TextureView

OpenGL ES

TensorFlow Lite Models

Bitmap

YUV420_888 image

Native Java Code

OpenGL Texture Buffer

JNI Bridge

Processing through graphs and calculators.

CameraX API

Rendered

FrameTexture

custom camera

triggered

.SO built Library

Wikitude SDK

Processing, and rendering

JS WebView

Start

Wikitude

Remarks:

All C++ extensions are built and compiled into a .SO file (Shared Object) for arm64-v8a architecture.

This process is automated through a Gradle task that triggers a CMake File that generates all the required .SO files and places them in the Project’s directory as appropriate.

MediaPipe is excluded from this process as it won’t require constant rebuilding. Its .SO File is generated once and added as a dependency.

4.2 Hand Tracking -

Extending Wikitude &

Integrating MediaPipe

30 of 52

30

4.2 Hand Tracking

Extending Wikitude via C++ Plugins

Wrapping MediaPipe Library in an Android App with TFLite Models.

Having Wikitude and MediaPipe under the same package.

Get the connection between MediaPipe and Wikitude working.

Train TFLite Model on certain gestures.

Possibly expand to dynamic gestures.

Done

To do

31 of 52

31

3.

Features &

Methodology

AR Remote Assistance

Hand Tracking & Error Detection

Instructions

Authoring

Automation

32 of 52

32

3.

Features &

Methodology

Instructions

Authoring

Automation

33 of 52

33

4.3 Instructions Automation – �Approaches Considered

34 of 52

34

Ada

(fastest)

Babbage

Curie

Davinci

(most powerful)

4.3 Instructions Automation – OpenAI's Language Base Models

35 of 52

35

4.3 Instructions Automation – �Approaches Comparison

User Manual	GPT-3 (text-davinci-003)

36 of 52

36

4.3 Instructions Automation �

Compare gpt-3 responses with the manual instructions

Decide on approach

Integrate API into code

Process & structure responses

Done

To do

Find approach for generating visual aids

37 of 52

37

4.3 Instructions Automation – �Sequence Diagram

38 of 52

38

5. Diagrams –

Architectural Diagram

(Layered Architecture)

39 of 52

39

5. Diagrams – Class Diagram

40 of 52

40

5. Diagrams –

Design Patterns – Chain of Responsibility

41 of 52

41

5. Diagrams – Class Diagram

42 of 52

42

5. Diagrams –

Design Patterns – Decorator

43 of 52

43

5. Diagrams – Class Diagram

44 of 52

44

5. Diagrams –

Sequence Diagram

45 of 52

45

5. Diagrams –

Database Design

46 of 52

46

6. Dataset

461 images
Classes:

Coolant
Engine
Oil reservoir
Windshield fluid
Power

47 of 52

47

7. UI Prototype

48 of 52

48

7. UI Prototype

49 of 52

49

7. UI Prototype

50 of 52

50

7. UI Prototype

51 of 52

51

7. UI Prototype

52 of 52

52

8. Demo