1 of 16

How Mozilla uses Python to Build and Ship Firefox

2 of 16

How do we get from this….

To this

3 of 16

Who am I?

Chris AtLee

catlee@mozilla.com

4 of 16

some numbers

4 GB in version control (hg) for Firefox; 494k commits

Over 200 commits per day; around 7,000 per month

1,750 build & test tasks per commit

approximately 15 cpu days of automation per commit

20 releases per month. Each release can have over 7,000 individual tasks and generates 50-100GB of artifacts!

All powered by Python!

5 of 16

Goal of this talk:

understanding pipelines & python

6 of 16

treeherder

7 of 16

taskcluster

mostly not python, sorry :(

https://docs.taskcluster.net/docs

Cluster of services that provides a CI and release platform to Mozilla.

The queue maintains a queue of tasks. Tasks are represented by json objects specifying which commands to run, what dependencies they have, etc.

Provisioners create new cloud resources to handle increases in load (e.g. in AWS)

Workers claim tasks from the queue, and run on the AWS instances, or on fixed hardware (e.g. mac minis, moonshots). They execute the tasks and then and upload resulting logs and artifacts, for example the builds, or screenshots from test failures or crash stacks.

For our hg repos, we have a service called mozilla-taskcluster which watches for new changes to the repos. It then triggers “decision tasks”, which run `mach taskgraph`

Photo: https://commons.wikimedia.org/wiki/File:Galileo_supercomputer_rack.jpg

8 of 16

mach taskgraph

it’s complicated

This is a graph (a DAG) of tasks for a recent Firefox release.

There are approximately 5k discrete tasks here. Graphviz….struggles a bit to make sense of this.

Tl;dr there are a lot of moving pieces here. For each platform we’re releasing, we have this pattern that we repeat of building Firefox itself; creating localized versions of it (we now have an en-CA locale!); creating complete and partial updates for each version; signing all the binaries; and finally publishing. For desktop, we have 5 platforms, 100 locales, and we create partial updates going back 3 or 4 versions. That means we create about 500 individual installers, and 2000 partial updates per release.

mach is a frontend CLI to the build system written in python.

Among other things, it wraps the build tool (mozbuild, which is written in python), as well as providing other common utilities, like taskgraph generation.

Taskgraph generation reads in yaml configuration files from the repository and process them with a series of python functions (that we call transforms), and finally generates the concrete set of tasks representing what to build/test/sign/upload. Like .travis.yml file, but hyped up on way too much halloween candy.

mozbuild: https://developer.mozilla.org/en-US/docs/Mozilla/Developer_guide/Build_Instructions/How_Mozilla_s_build_system_works

9 of 16

workers

Photo by Randy Fath on Unsplash

Interact with the Taskcluster Queue to claim work

Many different kinds of workers. Builds are done on docker or bare metal workers. Some workers are implemented in nodejs, some in Go, and others in Python.

Tasks definitions for these types of workers allow you to specify arbitrary commands and environment variables to run, as well as what docker containers to use. Great for developer ergonomics! Bad for security! We need to protect sensitive security tokens for publishing to S3 for example, and allowing people to run arbitrary code on the workers puts those tokens at risk.

We also have workers that operate in a restricted environment. They operate only on data passed in the task payload. We call them “scriptworkers” because the code or script that is run for the task is baked into the worker, and can’t be overridden.

We use use asyncio and aiohttp to do things in parallel when possible. So we can do things like download artifacts from our upstream tasks in parallel.

We use these types of workers for security sensitive work, like getting binaries signed, or pushing them to S3 or the Google Play Store for publishing Android apps.

The output from our release taskgraph is a set of signed binary artifacts ready to be published. The scriptworkers are also responsible for publishing the updates to our S3 buckets, and to our update server.

Chain of trust

https://github.com/mozilla-releng/scriptworker