Firmware Development

For Non-Firmware Engineers

Agenda

• What is Firmware?
• How do I build it?
• How do I debug it?

Firmware Design

• Proprietary OS of our own design
• Development started in 2012
• Mostly C, some inline asm
• 220k~ LOC
• Scheduler & concurrency primitives provided by FreeRTOS

Firmware Design Goals

• Very low resource usage
• 128 KB RAM, 512 KB flash
• For comparison, Apple Watch: 512 MB RAM, 8 GB Storage
• A lot of decisions are made to keep code size footprint low
• Modular
• Be able to slice and dice the firmware up into different builds
• Allows us to redesign and rewrite things frequently as new requirements come up
• Allows us to unit test individual components
• YAGNI
• We don’t build things we don’t need.
• “The Dumbest Thing That Could Possibly Work”
• Very microcontroller focused, depends heavily on vendor libraries from ST

​

Firmware Concepts

• Apps
• Workers
• Services
• Applib
• Modals
• Tasks
• Resources
• Exported Symbols
• Syscalls
• Boards
• Shells

Firmware Concepts - App

• Pretty much all the UI you interact with on the watch is an app
• The launcher is an app
• Timeline is an app
• Settings is an app
• Every watchface is an app
• Apps can be comprised of multiple windows
• There is only one app running at a time
• Apps are either built into the firmware (system apps) or installed (3rd party apps)
• 3rd party apps are run in “unprivileged” mode
• Only allowed to access their own data and what we expose through syscalls

Firmware Concepts - Worker

• Allows developers to run functionality in the background
• For example: Misfit, Jawbone, Battery+
• Just like an app, but smaller and runs independently
• There are no system workers, only 3rd party workers

Firmware Concepts - Modals

• UI that the system can pop up without interrupting the running app
• Notifications
• Alarms
• Runs in privileged mode from inside the system

​

Firmware Concepts - Services

• Software components that are always present
• Respond and generate their own events
• Examples of services:
• Activity
• Timeline
• PutBytes
• Animations
• Features are frequently divided into service and app portions
• For example: Health
• Activity service is always running, counting steps
• Health app queries data from the activity service

Firmware Concepts - Applib

• Graphics, UI, and app runtime code
• Used by both apps and modals
• Can be cross compiled to other backends
• For example, RockyJS!

Firmware Concepts - Exported Symbols

• The functions that we export to 3rd party apps
• Defined by tools/generate_native_sdk/exported_symbols.json
• Results in an autogenerated pebble.h that we ship with our SDK
• These symbols are not allowed to change!
• Changes to these symbols or their behaviour must go through a WTF review

}, {� "type": "function",� "name": "graphics_draw_text",� "addedRevision": "4"�}, {�

​

Firmware Concepts - Syscalls

• 3rd party apps run in an unprivileged context and must use syscalls to access system data
• Looks like an ordinary function, but when called will raise and lower privilege
• Care must be taken to not allow an app to crash the system or access secured data

DEFINE_SYSCALL(void, sys_get_app_uuid, Uuid *uuid) {� if (PRIVILEGE_WAS_ELEVATED) {� syscall_assert_userspace_buffer(uuid, sizeof(*uuid));� }�� *uuid = app_manager_get_current_app_md()->uuid;�}�

​

​

Services

Applib

3rd Party

Apps

System

Apps

Exported

Symbols

Syscalls

Firmware Concepts - Tasks

• Firmware code executes in a fixed number of threads that are created at startup
• FreeRTOS calls threads “tasks”.
• There are 8 tasks
• App, Worker, Kernel Main, Kernel Background…
• Tasks typically communicate by sending each other events
• See src/fw/kernel/events.h

Firmware Concepts - Resources

• A firmware release is made of two parts, the firmware image itself and a resource pack (aka the pbpack)
• Resource pack includes fonts, images, animations and other raw data
• The resource pack is stored in the external flash, as opposed to in the microprocessor like the firmware image itself
• The image and the resource pack are built at the same time and distributed together inside a .pbz
• Without a matching resource pack the firmware image will refuse to boot
• Will force boot PRF on a real watch
• Will leave you in a “panic” screen on a big board
• Apps are also distributed with their own resources packs
• Apps use a combination of system resources and their own app resources

Firmware Concepts - Build Configuration

Firmware Concepts - Board

• A “board” is the target piece of hardware we want to build for
• snowy_dvt is a board
• snowy_evt is a board
• ev2_4 is a board
• silk_bb is a board
• The “board” specifies both which features to build in as well as which drivers to include
• For example, Health is not included in pre-Time firmwares

Firmware Concepts - Shells

• A build-time configuration that describes what type of firmware is generated
• For example:
• Normal
• PRF
• sdkshell
• 3.X Migration

​

• Determines which parts of the codebase are built when it comes to services and system apps, and how the UI moves between them
• See src/fw/shells

Building Firmware

Building Firmware

• Contained in the “tintin” repo on github
• Please don’t call the firmware “tintin”, it’s a misnomer
• Built from the command line using python and the “waf” build system
• Curious about waf? https://www.youtube.com/watch?v=sjVnhEP94vM
• Make sure you have your dependencies installed and up to date
• `pip install -r requirements-osx.txt`
• Some dependencies are hosted on our internal server, make sure you’re in the office or your VPN is connected and the server is configured.
• https://pebbletechnology.atlassian.net/wiki/display/DEV/pypi
• These change from time to time, if your build fails try this again

Building Firmware - Bigboard

​

Building Firmware - Bigboard

• `./waf configure --board=snowy_bb2`
• Configures your build for a snowy big board.
• `./waf build`
• Builds the firmware from top for the current configuration.
• `./waf flash`
• Installs the currently built firmware onto a board connected to your computer over USB
• `./waf image_resources`
• Installs a new resource pack onto your board once it’s booted the new firmware
• This is only required if the resource pack is out of date
• If your bigboard isn’t booting, check the logs to see if this is the issue

Building Firmware - QEMU

• We have an emulator that you can run on your computer that runs a modified firmware image
• Use this when possible, it’s easier and faster to work with in general

Building Firmware - QEMU

• `./waf configure --board=snowy_bb2 --qemu`
• `./waf build`
• `./waf qemu_image_spi`
• Build a flash image that includes the resource pack based in
• Replaces the “image_resources” step for non-qemu
• `./waf qemu`
• Launch the emulator with the currently built firmware and flash image

Debugging Firmware

• Logging
• Core dumps
• GDB
• Unit Tests
• Automated Testing

Debugging Firmware - Logging

• The firmware frequently emits logs out its serial port while it’s running
• Adding a log to the firmware is as easy as…
• PBL_LOG(LOG_LEVEL_DEBUG, "Starting app <%s>", app_name);
• You can view logs from a running device using…
• `./waf console` for a big board
• `./waf qemu_console` for QEMU
• Note that only one thing can read from the serial port at a time
• Can’t image_resources and use the console at the same time
• Can’t open the console twice

Debugging Firmware - Logging

​

* M 15:14:52.777 main.c:114> __TINTIN__� * M 15:14:52.781 main.c:116> v3.11-beta1-13-g1f2d9c4� * M 15:14:52.785 main.c:117> (c) 2013 Pebble� * M 15:14:52.785 main.c:118>� I M 15:14:52.785 clocksource.c:29> LSE oscillator already running� D M 15:14:52.789 rtc.c:50> Current time is <Wed Mar 16 15:14:52 2016>� D M 15:14:52.792 new_timer.c:398> NT: Initializing� D M 15:14:52.796 accessory.c:292> Changing accessory connector baud rate to 115200�

​

​

Debugging Firmware - GDB

• `./waf gdb` or `./waf qemu_gdb` will give you a gdb connection to your device
• If you’re debugging a crash, consider using the --nowatchdog configure option to waf
• If you crash, the board will halt, allowing you to connect GDB
• Use PBL_ASSERT liberally!

Distributing Firmware

• `./waf bundle` will make a .pbz out of your firmware build you can send to others to sideload
• Once your software is running on a sealed watch, you don’t have serial logs or direct GDB access anymore
• Need to rely on “support emails” and our debugging tools

Debugging Firmware - Flash Logging

​

• Log lines that are level INFO and above (warning, error, etc) are also logged to flash for later retrieval
• Stored in “generations”, which represent a unique boot
• Generation 0 is the log from the boot that was currently running
• Generation 1 is the previous boot
• Generation 2 is the boot before that
• Can be retrieved using the “contact support” button in your mobile app

Debugging Firmware

* M 15:14:52.777 main.c:114> __TINTIN__� * M 15:14:52.781 main.c:116> v3.11-beta1-13-g1f2d9c4� * M 15:14:52.785 main.c:117> (c) 2013 Pebble� * M 15:14:52.785 main.c:118>� I M 15:14:52.785 clocksource.c:29> LSE oscillator already running� D M 15:14:52.789 rtc.c:50> Current time is <Wed Mar 16 15:14:52 2016>� D M 15:14:52.792 new_timer.c:398> NT: Initializing� D M 15:14:52.796 accessory.c:292> Changing accessory connector baud rate to 115200�

​

​

Debugging Firmware - Log Hashing

* M 15:15:19.226 :0> NL:b1c3�* M 15:15:19.226 :0> NL:b1ea `v3.11-beta1-13-g1f2d9c4`�* M 15:15:19.230 :0> NL:b209�* M 15:15:19.230 :0> NL:b235�I M 15:15:19.234 :0> NL:cf75�D M 15:15:19.234 :0> NL:cf34 `Wed Mar 16 15:15:19 2016`�D M 15:15:19.238 :0> NL:13fa0�D M 15:15:19.242 :0> NL:b3fd 1c200�

​

​

​

Debugging Firmware - Log Hashing

• Think of it as log compression
• PBL_LOG() macros get compressed down at build time
• To decompress, you need a loghash_dict.json
• build/src/fw/loghash_dict.json
• ./waf console and friends will dehash for you automatically
• Matches are made between firmware versions and loghash dictionaries using the build ID, and cannot be regenerated later

I M 15:14:52.824 debug.c:185> BUILD ID: 89ffde64bee1ed31bae11eeee0226caf31e82f07�

​

​

Distributing Firmware - Hagen Daas

• Set of web services maintained by the firmware team
• Started out to solve the problem of dealing with log hashing
• Daas = Desym as a service
• Files.tools.getpebble.com
• Stores loghash_dict.json files from previous builds for later retreival
• Bamboo builds are automatically uploaded using `./waf upload_symbols`
• You can run this locally as well to upload local information
• Can also use the web interface
• Also stores other items for us like cores, symbols…
• Desym.tools.getpebble.com
• Paste or upload a log file using the web interface, get a dehashed and desym’d version back

Distributing Firmware - Hagen Daas

Distributing Firmware - Core Dumps and pbldebug

• Whenever we crash, a core dump is created and saved in flash
• Contains a complete snapshot of the firmware state at the time of the crash
• Can be retrieved using the “contact support” option in your mobile app
• Analyze it with pbldebug
• pip install pbldebug
• Make sure you have our internal pypi server available
• https://github.com/pebble/pbldebug
• `pbldebug PBL-12345` will triage a bug in JIRA
• Dehash any logs attached to the ticket
• Open any cores attached to that ticket in GDB interactively
• `pbldebug --core_dump crash.data` will debug a core file locally
• Does the lookup for the symbol file for that build in files.tools.getpebble.com
• Some cores are encrypted, look in the support email for an account ID and pass that as the --account option

Debugging Firmware - Automated Tests

Debugging Firmware - Unit Tests

• `./waf test` will run our full test suite locally on your computer
• Firmware code is rebuilt for your desktop and run in small units using a framework named “clar”
• Python tests exist as well for our tools
• Tests live in the tests/ folder
• Useful stubs and fakes can be found in tests/stubs/ and tests/fakes/
• See the Firmware Unittest wiki page for details
• https://pebbletechnology.atlassian.net/wiki/display/DEV/Firmware+Unittest

Debugging Firmware - Integration Tests

• Integration tests are key to our firmware stability
• Apps are loaded, buttons are pressed, screenshots are verified
• There’s a closet full of bigboards that are constantly running automated tests
• There’s a cloud full of QEMU instances that are also constantly running tests
• There’s a few test plans you need to be aware of
• Phabricator Diffs run on QEMU every time you upload changes to Phabricator
• Master Checkins run on real boards every time a change is landed on master
• Nightly Build runs on real boards every night
• Each test plan is larger and more rigorous than the last
• http://orchestrator.hq.getpebble.com/ is the home for all our automated testing

​

Debugging Firmware - Phabricator

​

• We use Phabricator for our code reviews
• The only way to get your code shipped to our master branch is through Phabricator
• Do your work on a feature branch, then `arc diff origin/master`
• Get 2 or more reviewers to sign off on it
• Wait for the automated tests to pass
• `arc land <branch>` to land your review
• Keep an eye on #firmware to make sure the Master-Checkins test plan goes green

You now know…

• The general structure of firmware
• How to modify and build your own firmware
• How to debug that firmware

​

So go make some changes to firmware already!