1 of 41

Using Zephyr to Build Smaller Driver Domains for Virtualized Vehicle Systems

TOKITA Hiroshi

Zephyr Maintainer

2 of 41

Who am I?

I’ｍ TOKITA Hiroshi.

I contribute to Zephyr since 2018.

I taken on maintainer of the Raspberry Pi Pico and Renesas RA.

Today I speak as an individual.

License: CC-BY-4.0

3 of 41

Agenda

What is Zephyr and SDV?

Zephyr in SDV

What’s need to run Zephyr on Xen as VirtIO backend?

Running Zephyr based VirtIO Backend

Real World/Hardware Problem

License: CC-BY-4.0

4 of 41

Before beginning session…

As you hear now, Bear with my accent…

Everything is in the PDF. →

My English is not great, but my Mandarin is even worse.
Last year, I spoke I only master Mandarin term is 廁所在哪？
I learn one more term 我要這個.
This should make the trip perfect! :-)

License: CC-BY-4.0

5 of 41

What is Zephyr and SDV?

6 of 41

What is Zephyr?

Open-source real-time OS
Lightweight (> 2kb mem)
Strong in IoT, Bluetooth and Matter
Support over 1000 boards
POSIX APIs (Most of PSE53)

But In this session, I will talk about usage that is not on the MCUs.

License: CC-BY-4.0

7 of 41

What is SDV?

SDV is a ‘buzz word’ in automotive domain.�So, not exists formal definition.
It's often described as cars becoming like smartphones, but the definition seems to vary from person to person.
However, there is a shared understanding that the importance of software in automobiles is increasing.
And that server-based technologies such as cloud computing are being incorporated.

License: CC-BY-4.0

8 of 41

Automotive Grade Linux

Automotive Grade Linux(AGL) is Open Source Project for building Linux systems for Automotive.

Project is since 2012, Many car OEM companies joined.

Last year, we launched a platform named “SoDeV” for realize SDV concept.

License: CC-BY-4.0

9 of 41

SoDeV platform by Automotive Grade Linux

License: CC-BY-4.0

10 of 41

Our booth: SoDeV Runs AGL and AAOS on Xen

Today, we are demonstrating the SoDeV system in our booth.

AGL SoDeV reference implementation with Retronix Sparrow-Hawk board (Renesas R-Car V4H)
AAOS (Android Automotive OS) on the SoDeV system with RPi5

License: CC-BY-4.0

11 of 41

Zephyr in SoDeV

12 of 41

Virtualization

We think “Virtualization” is key technology of SDV.

AAOS “Trout” is a VirtIO-based virtualized reference, officially supported by QNX Hypervisor �Reference platform | Android Open Source Project
AGL already provide VirtIO based full virtualized system image based on KVM(Kernel Virtual Machine). SoDeV platform is proceeding this concept one more step.�How to build AGL Basesystem for VirtIO - IVI Production Readiness - Confluence

License: CC-BY-4.0

13 of 41

Xen and VirtIO

Xen is an open-source hypervisor that debuted in 2003.
Recent years, A lot of effort paid for improve using in embedded systems and functional safety.
VirtIO is Paravirtualized devices/buses specification standardized by OASIS since 2014.
Xen has mechanism for support VirtIO.

License: CC-BY-4.0

14 of 41

Zephyr in SoDeV platform

Zephyr appears in SDV system with following contexts …

Safe Islands
Guest VM
A components of Xen �hypervisor based �system.

License: CC-BY-4.0

15 of 41

Xen Domain concept

Xen

Hardware

Dom0

DomD

VirtIO Backend

Driver

DomU

VM Management & Monitoring

VirtIO Frontend

Application

Xen and Dom0, and DomU should not have any device depend implementation.

DomU uses only VirtIO devices.

DomD handles VirtIO requests with delegating to real devices.

Passthrough

License: CC-BY-4.0

16 of 41

Linux DomD is huge.

Can we split and reduce it?

Xen

Hardware

DomD

Linux

VirtIO BE

Driver

DomU

App

Driver

VirtIO BE

VirtIO FE

DomU

App

VirtIO FE

DomU

App

VirtIO FE

License: CC-BY-4.0

17 of 41

SoDeV Platform Architecture (My proposal)

Type 1 Hypervisor (e.g. Xen)

Automotive SoC or Virtual Environment

Dom 0

Linux/Zephyr

Control Domain�(Optional)

Dom D #0

Zephyr

VirtIO Backend

Zephyr Driver�(CAN, Sound, SPI,...)

VM Management & Monitoring

Dom0 Kernel

Guest & Function Domains

Dom U

Cluster

Dom U

ADAS/AD

VirtIO Frontend

AGL

Zephyr

Dom U

IVM

VirtIO Frontend

Android

Unified HMI

Dom D #1

Zephyr

VirtIO Backend

Zephyr Driver�(Ether, GPU,

block, ...)

Adding Zephyr's VirtIO BE Framework.�(My patch)

DomD

Linux

Driver Domain �(HW Access)

VM Management & Monitoring

Dom0 Kernel

Replacing Linux Dom-D with multiple Zephyr Dom-D.

This helps target function safety authorization for the driver domain

License: CC-BY-4.0

18 of 41

What’s need to run Zephyr on Xen

as VirtIO backend?

19 of 41

Process outline

Setup virtual MMIO register

Launch ioreq server
Map the address to ioreq server

Setup event

bind interdomain event channel
enable event

Query configuration

Fetch XenStore information

License: CC-BY-4.0

20 of 41

XenStore

XenStore is a mechanism for sharing information with the hypervisor.�It stores virtio configuration information.

/local/domain/1/backend/virtio/2/0 = ""

/local/domain/1/backend/virtio/2/0/frontend = "/local/domain/2/device/virtio/0"

/local/domain/1/backend/virtio/2/0/frontend-id = "2"

/local/domain/1/backend/virtio/2/0/online = "1"

/local/domain/1/backend/virtio/2/0/state = "1"

/local/domain/1/backend/virtio/2/0/irq = "33"

/local/domain/1/backend/virtio/2/0/base = "0x2000000"

/local/domain/1/backend/virtio/2/0/type = "virtio,device2"

/local/domain/1/backend/virtio/2/0/transport = "mmio"

/local/domain/1/backend/virtio/2/0/grant_usage = "1"

License: CC-BY-4.0

21 of 41

(Virtual) MMIO Register / Interrupt

DomD

VirtIO Backend

DomU

VirtIO Frontend

(Virtual)

MMIO Register

When read/write accesses to (virtual) mmio register are occurred, �call internal callbacks.

Configure (Virtual) interrupt to FE refer the Xen configuration.

License: CC-BY-4.0

22 of 41

Query Virtqueue Address

The (virtual) MMIO register configuration is defined in VirtIO spec.

We can found Virtqueue Address from register layout.

License: CC-BY-4.0

23 of 41

Access Virtqueue

DomD

VirtIO BE

DomU

VirtIO FE

(Virtual)

MMIO Register

BE read and write virtqueue with given pointer.

Response QueueDesc request with �Virtqueue Address.

Virtqueue

FE allocate virtqueue and tell to BE via MMIO.

Virtqueue Ref

License: CC-BY-4.0

24 of 41

Two rings: arteries and veins

Descriptor

Data

Frontend (Guest)

Backend (Host)

Used

Avail

Ring entity points to a descriptor by index

Descriptor points to a memory region (address, length)

Frontend picks free descriptor and writes a request, and post to the avail ring.

Backend reads�from the avail ring, processes, then returns it to used ring.

License: CC-BY-4.0

This is a bit heavy, but let's take a look at how VirtIO works.

While the content of messages in VirtIO differs depending on the device it implements, at its core, it uses a single data structure.

These are the two ring buffers shown in the diagram. In VirtIO, these are called Available and Used. They correspond to Request and Response.

Why are two rings needed? Because we want to send requests without waiting for a response. Because it is asynchronous, requests can be sent at high speed, and the receiving side can return responses in the order in which processing is completed.

A similar concept is used in the Linux kernel's io_uring.

The actual data is stored in memory, and each data is associated with a descriptor.

The ring buffer references these descriptors by index,

and the actual memory is accessed via the descriptor.

25 of 41

Running Zephyr based VirtIO Backend

26 of 41

Configure VM to enable VirtIO

Xen Virtual Machine is launched with “xl create” command.
Give a config file for launch.
Add following line declare to use VirtIO feature.

virtio = [ "type=virtio,device2,transport=mmio,backend=domd,grant_usage=1" ]

This is DomU/FE side configuration.

DomD/BE should behave according to FE configurations.

License: CC-BY-4.0

27 of 41

Launch VMs in Pause state

Launch DomD/BE in pause state.�(FE configuration declare as depends to BE, need to be launched preceding to FE.)

Launch DomU/FE in pause state.

(BE is not ready because that is in pause state.)

Unpause DomD/BE.

(XenStore configuration is ready.)

Unpause DomU/FE.

(All Green.)

License: CC-BY-4.0

28 of 41

Demo: virtio-blk on QEMU

Ubuntu on WSL2

QEMU

Xen

Zephyr (DomD)

Linux (Dom0)

Linux (DomU)

Virtual

Disk (mem)

R/W

Launch

/dev/pts/X

/dev/vport0p1

socat

License: CC-BY-4.0

29 of 41

How much reduced?

Compare the image size.

Linux-based DomD: 832,907,264 bytes
Zephyr-base DomD: 87,640 bytes

While the comparison conditions aren't strictly aligned, it's clear that a large infrastructure is necessary for Linux.

License: CC-BY-4.0

30 of 41

Current Status

31 of 41

Progress in underwater

And last week, many new VirtIO frontend drivers PRs are published.

License: CC-BY-4.0

32 of 41

Real World/Hardware Problem

33 of 41

Retronix SparrowHawk R-Car V4H SBC

Renesas RCar V4H SoC

Cortex-A76/R52

License: CC-BY-4.0

34 of 41

Peripheral address and pages

Xen can assign memory to a VM by page (4KB aligned).�The same applies to registers; if plural peripheral registers are in same page, it raises complexity in design.��Solution Ideas:

Carefully design each GPIO controller so that it can be realized by the DomD responsible for its purpose.
All are treated as virtio-gpio.

E6050000: gpio0�(PinFuncCtrl part)

E6050180: gpio0�(GPIO part)

E6050800: gpio1�(PinFuncCtrl part)

E6050980: gpio1�(GPIO part)

1page�(4KB)

This case, we need to delegate gpio0 and 1 for single DomD

License: CC-BY-4.0

35 of 41

PinControl and ClockControl drivers

Typically, Zephyr driver set pinctrl and clock control at initializing.

I couldn't find a simple solution.�This is obstacle.

Timer0-2

DMA0,1

SD

Serial0,1,3,4

static int pwm_rcar_init(const struct device *dev)

{

/* Configure dt provided device signals when available */

ret = pinctrl_apply_state(config->pcfg, PINCTRL_STATE_DEFAULT);

…

ret = clock_control_on(config->clock_dev, (clock_control_subsys_t)&config->mod_clk);

…

For example, the clock register has bits set for each function.

License: CC-BY-4.0

36 of 41

Hardware Security Module

The parts related to ARM TrustZone etc. are SoC-specific.

Likely to be related to proprietary specifications.

Sometimes a random number generator

It is tied to Hardware Security Module(HSM), so it may not be possible to use it simply.

SoC

OS

ROM IPL

OS verification

Complex interactions to use SoC-specific security features

RNG, SHA256�Accelerator driver, etc

License: CC-BY-4.0

37 of 41

Configuring Virtio for Zephyr Virtio FE

DeviceTree fragment injection is not work with Zephyr because Zephyr not handle DTB.

Solution idea:�Most of the information can also be obtained from XenStore. But not all.

�

💻

Linux Kernel

DTB

Launch

Xen HV

virtio@2000000 {

compatible = "virtio,mmio";

reg = <...>;

dma-coherent;

xen,dev-domid = <1>;

};

Xen inject devicetree fragment to existing DTB for configuring VirtIO component.

License: CC-BY-4.0

38 of 41

Next: Supporting virtio-blk on SD-Card

I already raise PR to SD card support on Renesas R-Car V4H board
But this is bit complex. PinMux, Clock, GPIO for detection, …
And there are obstacle: How we design boot sequence?

Who provide disk access?
Is it became a single point of failure?

License: CC-BY-4.0

39 of 41

Conclusion

40 of 41

Conclusion

Yes, Zephyr can be a VirtIO backend DomD

It works, and the image is ~4 orders smaller.

But the designing domain division are still complex task.

Shared registers, MMIO page, pinctrl, clock, HSM are all the same problem — resources that multiple peripherals share cannot be assigned to different domains.

Surprisingly, development on both Xen and virtio has picked up considerably in the past week.
The real work is just began.
Let’s Join Us.

License: CC-BY-4.0

41 of 41

Thank you for your attention.

Anyone have questions?