1 of 52

LVGL in WebAssembly: Building NuttX Touchscreen Apps with Zig and testing them in the Web Browser

Lup Yuen LEE

github.com/lupyuen

lupyuen.github.io/nuttx

2 of 52

Our Story Today

LVGL Graphics Library +�Zig Programming Language (WebAssembly) +�Apache NuttX Real-Time Operating System (RTOS)
Let’s create an LVGL App,�Preview it in a Web Browser
By compiling LVGL Library to WebAssembly with Zig Compiler (based on Clang)
Browser-based alternative to NuttX Simulator�(Maybe someday NuttX will run in WebAssembly?)
Demo Video

2

lupyuen.github.io/nuttx

LVGL is a popular Graphics Library for Embedded Devices

Zig is a newish programming language that runs in the Web Browser with WebAssembly

Apache NuttX is Real-Time Operating System that many of us use for running LVGL Applications on Embedded Devices

What if we could run LVGL Applications directly in a Web Browser?

That might be an interesting new way to learn LVGL Programming

And experiment with LVGL Apps

It’s a little like running LVGL Apps in the NuttX Simulator

Except that all we need is a Web Browser!

But the LVGL Library needs to run in a Web Browser

Can we compile the LVGL Library from C to WebAssembly?

There is a fascinating new way to compile C programs into WebAssembly

We can use the Zig Compiler to do this!

It works because Zig Compiler bundles a Clang Compiler inside

So Zig Compiler will happily compile the LVGL Library from C to WebAssembly

OK since we are using the Zig Compiler

Why don’t we code the LVGL Apps directly in Zig?

This LVGL Application was created in Zig

It runs on NuttX Devices

And it runs in the Web Browser too!

In our presentation today

We explain how we made all this happen

LVGL Applications in Zig

Running in a Web Browser and on NuttX Devices

3 of 52

Why LVGL with WebAssembly?

LVGL Apps can get painfully tedious to create:�Recompile, Reflash and Reboot�Again and again and again!
Can we preview LVGL Apps in a Web Browser?�Then run the exact same code on a NuttX Gadget?�(Live Reload)
LVGL already compiles to WebAssembly with Emscripten and SDL, but let’s explore Zig�(Safer, simpler LVGL Apps)
Maybe someday we can Drag-and-Drop to create LVGL Apps in the Web Browser?

3

lupyuen.github.io/nuttx

Today we write LVGL Applications in C

Compile them into NuttX

Flash the NuttX Firmware to our device

Boot the firmware and test our LVGL App

And we repeat this process

Even for the tiniest tweaks to our LVGL App

Yes we have a NuttX Simulator today

That will run LVGL Apps on any computer that supports X11

Can we do better, go lighter?

What if we could run LVGL Apps directly inside a Web Browser?

Like this LVGL App here?

Maybe someday we can even do Live Reload

When we tweak the Source Code for our LVGL App

The LVGL App will automatically recompile

And reload in the Web Browser!

LVGL already compiles to WebAssembly with Emscripten and SDL

So we know that LVGL will probably run OK in WebAssembly

But today we shall explore Zig for running LVGL in WebAssembly

And how we might create simpler, safer LVGL Apps

By coding them in Zig

4 of 52

“NuttX NextGen”?

Upsizing from 32-bit MCUs with tiny displays
To 64-bit Phones and SBCs with huge displays:�MIPI DSI and HDMI�(720 x 1440 and 1920 x 1080)
Large screens need more Design and Collaboration�(Perfect for Education)
Open source drivers (non-GPL) are rare�Only GPL Drivers contributed by vendor as BSP�(Missing docs)
We seek your contribution!

4

lupyuen.github.io/nuttx

Why are we doing all this now?

Why are we giving special attention to LVGL Touchscreen Apps?

Well something fascinating has happened over the past year

Let us call it “NuttX NextGen”

Traditionally, NuttX runs on 32-bit Microcontrollers with tiny displays

But over the past year, NuttX has evolved to run on 64-bit Smartphones and Single-Board Computers

Like PinePhone, PinePhone Pro and Star64 JH7110

These modern gadgets have enormous displays

NuttX finally goes High Definition!

But

With enormous displays

Comes great responsibility

To provide a simpler, easier way to collaborate and create LVGL Apps

For these modern giant screens

Right now we are busy building NuttX Drivers for these Next Generation Devices

And we welcome your contribution!

Hopefully everything that we discuss in this presentation

Will make it easier to create LVGL Apps for the Next Generation of NuttX Devices

5 of 52

WebAssembly with Zig

WebAssembly

LVGL

Display

Input

UI

lupyuen.github.io/nuttx

6 of 52

Why Zig

Works well with C�
Runtime Safety�(But not Memory Safe)�
Type Inference�
Auto-Converts C to Zig

6

lupyuen.github.io/nuttx

7 of 52

Compare C and Zig

7

lupyuen.github.io/nuttx

8 of 52

WebAssembly with Zig

Zig compiles to WebAssembly Bytecode�Runs in a Web Browser
Zig works both ways with WebAssembly:�JavaScript calling Zig�Zig calling JavaScript
Let’s render the Mandelbrot Set in WebAssembly:�Zig computes the Mandelbrot Set�JavaScript calls Zig for every pixel�JavaScript renders pixel to HTML Canvas
Source Code

8

lupyuen.github.io/nuttx

9 of 52

9

lupyuen.github.io/nuttx

10 of 52

JavaScript calls Zig WebAssembly

10

lupyuen.github.io/nuttx

11 of 52

JavaScript renders to HTML Canvas

11

lupyuen.github.io/nuttx

12 of 52

Zig Calls JavaScript

Now we call out from Zig to JavaScript�
We import the JavaScript Function into Zig�
Runs slower, but it works!�
Source Code�Read the Article

12

lupyuen.github.io/nuttx

13 of 52

LVGL in WebAssembly

WebAssembly

LVGL

Display

Input

UI

lupyuen.github.io/nuttx

14 of 52

LVGL App in Zig

LVGL already works with Zig on PinePhone NuttX�
Zig Compiler compiles LVGL Apps from Zig to Arm64 Machine Code�
“Simpler Safer Touchscreen Apps with Zig”

14

lupyuen.github.io/nuttx

15 of 52

LVGL App in WebAssembly

Can the same LVGL App run in a Web Browser? With WebAssembly?�
Zig Compiler compiles LVGL Library�from C to WebAssembly�(Via bundled Clang Compiler)�
JavaScript will do rendering and mouse input�And call our WebAssembly

15

lupyuen.github.io/nuttx

16 of 52

Compile LVGL to WebAssembly with Zig Compiler

16

lupyuen.github.io/nuttx

17 of 52

LVGL Porting Layer for WebAssembly

LVGL expects us to provide Platform-Specific Functions�
For Timers, Logging and Assertions�
We implement these in Zig

17

lupyuen.github.io/nuttx

18 of 52

WebAssembly Logger for LVGL

LVGL Logs sent by Zig to JavaScript, printed in the JavaScript Console�
Works OK because Zig can call JavaScript�
We’ll see this Common Pattern:�LVGL ↔ Zig ↔ JavaScript�Source Code�Read the Article

18

lupyuen.github.io/nuttx

19 of 52

LVGL Display

WebAssembly

LVGL

Display

Input

UI

lupyuen.github.io/nuttx

20 of 52

20

lupyuen.github.io/nuttx

21 of 52

21

lupyuen.github.io/nuttx

22 of 52

Render LVGL Display in Zig

LVGL expects a Callback Function to flush our display�
We implement it in Zig and call out to JavaScript to render the HTML Canvas�
Zig notifies LVGL that display has been flushed

22

lupyuen.github.io/nuttx

23 of 52

23

lupyuen.github.io/nuttx

24 of 52

Render LVGL Display in JavaScript

Zig calls JavaScript to render the display to HTML Canvas�
Our JavaScript copies the Canvas Pixels from WebAssembly Memory to HTML Canvas

24

lupyuen.github.io/nuttx

25 of 52

25

lupyuen.github.io/nuttx

26 of 52

Handle LVGL Tasks

But LVGL won’t flush our display unless we trigger the LVGL Background Task�
Here’s the simple Zig way to trigger the LVGL Background Task�
But it won’t work for Mouse Input�
Let’s trigger this from a JavaScript Timer…

26

lupyuen.github.io/nuttx

27 of 52

Handle LVGL Tasks: JavaScript Timer calls Zig

27

lupyuen.github.io/nuttx

28 of 52

Initialise LVGL Display

Zig Memory Allocator for malloc()
Set Custom Logger for LVGL
Set LVGL Display Buffer and Display Driver
Register the LVGL Display Driver
Render the LVGL Widgets
Handle the LVGL Tasks
S ource Code
Read the A rticle

28

lupyuen.github.io/nuttx

29 of 52

LVGL Input

WebAssembly

LVGL

Display

Input

UI

lupyuen.github.io/nuttx

30 of 52

30

lupyuen.github.io/nuttx

Once again, another messy diagram

But let’s walk through the implementation of LVGL Input

At left is our JavaScript Code that runs in the Web Browser

At centre is our Zig WebAssembly Code

At right is the LVGL Library compiled to WebAssembly

We capture the Mouse and Touch Events in the Web Browser

Mouse Down, Mouse Up

Touch Start and Touch End

When the Web Browser is clicked or touched

Our JavaScript sends the X-Y Coordinates to our Zig Function

Zig remembers the Coordinates

How will LVGL Library receive the Touch Coordinates?

Earlier we saw the Event Loop that triggers LVGL Tasks

One of the LVGL Tasks will poll for Touch Input periodically

This calls our Zig Function to fetch the Touch Coordinates

This might trigger a display refresh, like when we click a button

LVGL Library will redraw the screen and flush the display

Which we have walked through earlier

This is how we handle Mouse Input and Touch Input for LVGL

Spanning JavaScript, Zig and WebAssembly

Let us look at the code

31 of 52

31

lupyuen.github.io/nuttx

32 of 52

Handle LVGL Input

Capture JavaScript Events:�Mouse Down, Mouse Up,�Touch Down, Touch Up�
Pass the Touch Point to Zig�
Zig will remember it and pass to LVGL later�
LVGL will poll our Zig Input Driver for Mouse and Touch Input

32

lupyuen.github.io/nuttx

33 of 52

LVGL Input Device

Implement the LVGL Input Device in Zig�
Plus some C code because Zig can’t access LVGL Structs with Bit Fields�
Source Code�Read the Article

33

lupyuen.github.io/nuttx

34 of 52

Feature Phone UI

WebAssembly

LVGL

Display

Input

UI

lupyuen.github.io/nuttx

35 of 52

Feature Phone with LVGL, Zig and NuttX

Feature Phone UI as an LVGL Touchscreen App
That runs in the Web Browser with WebAssembly
And on an actual phone: PINE64 PinePhone
NuttX RTOS will support Outgoing Calls and Outgoing SMS on PinePhone
Though PCM Audio, Incoming Calls and Incoming SMS are still missing
Demo Video

35

lupyuen.github.io/nuttx

36 of 52

36

lupyuen.github.io/nuttx

37 of 52

LVGL Containers, Labels and Buttons

LVGL Containers for the Display Digits, Call Buttons, Digit Buttons�
Inside the LVGL Containers are�LVGL Labels�LVGL Buttons�
We handle presses of the LVGL Buttons�and update the LVGL Labels

37

lupyuen.github.io/nuttx

38 of 52

Display Container with Digit Label

38

lupyuen.github.io/nuttx

39 of 52

Call Container with LVGL Buttons

39

lupyuen.github.io/nuttx

40 of 52

40

lupyuen.github.io/nuttx

41 of 52

LVGL Buttons in Zig

We create the entire LVGL App in Zig�
Create the LVGL Buttons�
Set the Event Callback for the Button Press

41

lupyuen.github.io/nuttx

42 of 52

Handle LVGL Buttons

Zig handles the Event Callback�
Get the Event Code�
Fetch the Button Text�
Convert the text from C String to Zig Slice

42

lupyuen.github.io/nuttx

43 of 52

Handle Call Button and Digit Buttons

Call Button pressed:�Call the Phone Number that �was dialed�
Digit Button pressed:�Append the Digit to the Phone Number�
But Call Button will work differently:�WebAssembly vs PinePhone NuttX

43

lupyuen.github.io/nuttx

44 of 52

Works on WebAssembly AND PinePhone!

Zig allows importing of Target-Specific Functions:�WebAssembly vs NuttX�
So our Call Button will work �differently on WebAssembly �vs NuttX�
Source Code�Read the Article

44

lupyuen.github.io/nuttx

45 of 52

Feature Phone App in WebAssembly

We have created an LVGL App that runs in a Web Browser with WebAssembly�Demo Video �
And the exact same LVGL App runs on PinePhone NuttX�Demo Video �
Maybe someday we can Drag-and-Drop to create LVGL Apps in the Web Browser?

45

lupyuen.github.io/nuttx

46 of 52

Phone Calls and SMS over 4G

PinePhone has an LTE Modem inside:�Quectel EG25-G�
Connected over UART and USB�
Outgoing Calls and Outgoing SMS are OK�
UART Interface is ready for Voice Call and SMS Commands

46

lupyuen.github.io/nuttx

47 of 52

47

lupyuen.github.io/nuttx

48 of 52

“NuttX NextGen”?

48

PinePhone

Allwinner A64

PinePhone Pro

Rockchip RK3399

i.MX 8QuadMax MEK�NXP i.MX 8

Star64�StarFive JH7110

lupyuen.github.io/nuttx

49 of 52

49

PinePhone

Allwinner A64

PinePhone Pro

Rockchip RK3399

i.MX 8QuadMax MEK�NXP i.MX 8

Star64�StarFive JH7110

Banana Pi M64 /

Olimex A64 /

Pine64 LTS

Retro Pixel Pocket

Pinebook Pro /

Chromebooks

VisionFive2 /

Milk-V Mars

Librem 5

Phone

Mars CM

Orange Pi /

Rock Pi 4 /

RockPro64

PineTab2�(RK3566)

PineTab-V

Powkiddy RGB30 (RK3566)

SOQuartz�(RK3566)

Olimex TERES-I /

Pinebook

Arm64

RISC-V

lupyuen.github.io/nuttx

50 of 52

“NuttX NextGen”?

50

Arm / RISC-V SoC	Allwinner A64	Rockchip RK3399	NXP i.MX 8	StarFive JH7110
Smartphone	PinePhone	PinePhone Pro	Librem 5
SBC	Banana Pi M64 / Olimex A64 / Pine64 LTS	Orange Pi / Rock Pi 4 / RockPro64	i.MX 8QuadMax MEK	Star64 / VisionFive2 / Milk-V Mars
Tablet		PineTab2*		PineTab-V
Laptop	Olimex TERES-I / Pinebook	Pinebook Pro / Chromebooks
CM4 SoM		SOQuartz*		Mars CM
Gaming Handheld	Retro Pixel Pocket	Powkiddy RGB30*

*RK3566

lupyuen.github.io/nuttx

51 of 52

Articles

51

lupyuen.github.io/nuttx

52 of 52

Thank You

Thanks to the NuttX, LVGL and Zig Communities�
Let’s experiment and discover new ways of working with LVGL�
From Microcontrollers to Smartphones to Single-Board Computers�
And create many more cool, useful and sophisticated LVGL Applications!

52

lupyuen.github.io/nuttx