1 of 63

Building secure applications with Qubes OS

Qubes OS Summit 2022

Michael Z, Senior Software Engineer

2 of 63

Overview

About FPF
SecureDrop

What it is
Who uses it
“Classic” architecture…
… plus SecureDrop Workstation

SecureDrop Workstation

Current architecture
Lessons learned…
Possible futures?

https://securedrop.org

3 of 63

Freedom of the Press Foundation

https://freedom.press

https://securedrop.org

4 of 63

About Freedom of the Press Foundation

Advocacy

Mostly focused on US press freedom issues

US Press Freedom Tracker

documents and categorizes press freedom violations across the US, involving national, state and local authorities.

Digital Security Training,

Training journalists and activists, organizational audits (internationally)

Open Source Software Development

SecureDrop, SecureDrop Workstation, Dangerzone…

https://securedrop.org

5 of 63

SecureDrop is an online whistleblowing platform, hosted on-premises by news organizations. It uses Tor onion services for source anonymity and GPG for encryption. The code is free software, licensed under the AGPL.

6 of 63

Some of the organizations that use SecureDrop

7 of 63

Current SecureDrop Team

Kevin

Engineering

Allie

Engineering

Cory

Engineering

Rowen

Engineering

Erik

Engineering

Harlo

Digital Security

Olivia

Digital Security

David

Digital Security

Nathan

Support Engineering

Kunal

Engineering

Gonzalo

Engineering

Alex

Security Engineering

Abigail

Support

Giulio

Security Engineering

Michael�Engineering

100% of time on SecureDrop

>= 25% of time on SecureDrop

>= 50% of time on SecureDrop

Tina

UX

https://securedrop.org

8 of 63

Quick intro: SecureDrop “Classic”

9 of 63

Application server: Runs two Python web applications (one for sources, one for journalists) exposed via Tor Onion Services.

Source Interface: Public v3 Onion URL, accessible by anyone in Tor Browser

Journalist Interface: Authenticated v3 Onion URL. Requires key-based auth to resolve. Only accessible to journalists.

10 of 63

What the source sees

11 of 63

What the journalist sees

12 of 63

Private key to decrypt documents only in the air-gap environment.

13 of 63

Downsides

Existing workflows are slow (~1 hour round-trip)
It’s hard to patch an air-gapped system
To do their jobs, journalists need more tools than just viewing/exporting/printing

https://securedrop.org

There are downsides, and a couple are admittedly quite glaring. �
The time spent on merely inspecting submissions, which can be all spam mind you, is significant. Copying files to transfer devices, going to a different room, booting a new machine, entering multiple passphrases etc. and then seeing nothing useful… is a massive time sink

The air-gapped machine runs Tails, and keeping it patched requires that users actively keeping track of updates - spoiler alert: sometimes they don’t.

It only provides decryption and basic view/print/export options, and doesn’t address how submissions are stored or shared, so there’s plenty of opportunity for security mistakes for journalists to make elsewhere in their workflow, which yes, is more than just viewing printing and sharing with their team.��In summary, while SecureDrop “Classic” gets the job done, the usability of the system is quite lacking on a number of fronts [pause]

14 of 63

Introducing �SecureDrop Workstation

15 of 63

Classic air-gap architecture

16 of 63

New consolidated architecture

17 of 63

Demo Time!

18 of 63

Demos are cool - this one can seem a bit trivial, but diagrams do not! This one shows what’s actually been going on in the background.

First off, the only networked VM in is sd-proxy, which only communicates with the application server, and does so via a dedicated Whonix gateway sd-whonix. All other VMs have no network provider
The client application runs in the sd-app VM. It receives submissions via the proxy by employing a custom qrexec RPC call. It never opens submissions directly, instead, when submissions are exported or printed they are copied to a disposable VM called sd-devices, when they are viewed they’re opened in an unnamed DispVM using sd-viewer as its template. This provides a measure of protection against malicious submissions.
The sd-gpg VM is a vault VM holding the private key used to decrypt submissions, and only sd-app has access to it by using Qubes’ split-GPG tools.

Overall, this provides a virtual air-gap, preserving most of the benefits of the physical air-gap but reducing the manual overhead.

And as you’ve seen: users don’t interact with these VMs directly - their main point of contact is the client - this makes SecureDrop Workstation accessible to folks who might not otherwise be able to handle Qubes OS workflows manually. [pause]

19 of 63

What we’ve learned

(or: pain points)

20 of 63

Changes to dom0 are mandatory

It’s where templates get installed and VMs set up!
SDW needs qrexec policies set up to allow VMs to communicate without user intervention

https://securedrop.org

Yes, we’re starting off with something that can be a little bit of a taboo around these parts, but, we wouldn’t be able to do what we do without making quite a few changes in dom0

dom0 is the only place to install templates - ok, that is not controversial at all, those templates we need to create a specific set of pre-configured virtual machines, and at least in our current setup, the order of operation matters too, at least if you don’t want to flood users with RPC error notifications�
Leveraging Qubes’ power while ensuring it’s usable by journalists means we need to add our own RPC policies to let some inter-VM communication happen without user intervention, while deny most things because our goal is to isolate our VMs as best as possible, keeping users from accidentally doing something they really should not

21 of 63

Changes to dom0 are mandatory

It’s where templates get installed and VMs set up!
SDW needs qrexec policies set up to allow VMs to communicate without user intervention
Installing anything into dom0 requires 1337 skillz

Our installation procedure relies heavily on manual terminal input in multiple VMs
No easy-access to copy-paste from a browser = pain

https://securedrop.org

22 of 63

23 of 63

Keeping Qubes OS fully updated is non-trivial

We enforce an update policy that requires users to apply all OS updates before starting the client (once every 8h)

We reboot sys-* VMs when necessary
If dom0 is updated, a reboot is required

https://securedrop.org

24 of 63

Keeping Qubes OS fully updated is non-trivial

We enforce an update policy that requires users to apply all OS updates before starting the client (once every 8h)

We reboot sys-* VMs when necessary
If dom0 is updated, a reboot is required

The more templates an application like ours depends on, the more error prone and slow updating can become

We slimmed down from 6 (or 8) Template VMs to 2 (or 4)

https://securedrop.org

25 of 63

Unusual UX challenges

The devices widget is risky for non-technical users…

SDW auto-attaches USB drives and printers to a dedicated non-networked named DispVM (sd-devices)

https://securedrop.org

That Qubes OS itself presents an unusual UX challenge is not news to anyone here, but building usable yet secure applications for people who don’t particularly care about the depths of their OS … complicates the matter further

As an example, non-Qubes users expect to be able to attach a device and have it ready for use. You know, like a regular computer. Obviously that is at odds with Qubes’ design, for good reason, but there is a very real chance of error when using the devices widget. (If you use the XFCE frontend and never mess up, hats off to you!)

So to get around this, we break a bit with Qubes’ idioms add udev rules and scripts in the template that runs sys-usb to automatically attach USB sticks and printers to a dedicated VM called sd-devices. This has its downsides - for example if a user wants to attach a device elsewhere they have detach and reattach it - but it does reduce the chance of user error.

26 of 63

Unusual UX challenges

The devices widget is risky for non-technical users…

SDW auto-attaches USB drives and printers to a dedicated non-networked named DispVM (sd-devices)

VM separation creates unexpected behaviours

Example: printing is entirely separate from viewing
Documents are opened in a regular unnamed DispVM
… but printer is only available in sd-devices

https://securedrop.org

27 of 63

SaltStack

Difficult to debug

When states fail, the info one looks for is often hidden

Not all that idempotent as one would hope as soon as cmd.run sneaks in
Enforcing an order of operation with multiple VMs depending on other VMs isn’t supported natively, requires OOB scripting
Relying on it too much can bite you

https://securedrop.org

28 of 63

29 of 63

30 of 63

Possible futures

(or: things we’re still discussing)

31 of 63

Maintain our own RPC calls

Main motivation: VM pipelines!

qvm-open etc is good for direct VM communication, but for workflows involving multiple VMs, it gets complex to manage…

https://securedrop.org

32 of 63

33 of 63

Maintain our own RPC calls

Main motivation: VM pipelines!

qvm-open etc is good for direct VM communication, but for workflows involving multiple VMs, it gets complex to manage…
e.g.: sanitize/redact/archive

https://securedrop.org

34 of 63

Maintain our own RPC calls

Main motivation: VM pipelines!

qvm-open etc is good for direct VM communication, but for workflows involving multiple VMs, it gets complex to manage…
e.g.: sanitize/redact/archive

Also: substituting technologies (Sequoia based RPC replacing split-GPG) or printing from DispVMs?

https://securedrop.org

35 of 63

Pluggable architecture

Motivation: journalists still need more tools than just viewing! (e.g.: sanitize/redact/archive)

https://securedrop.org

36 of 63

Pluggable architecture

Motivation: journalists still need more tools than just viewing! (e.g.: sanitize/redact/archive)
Asking users/orgs to just change our templates?

https://securedrop.org

37 of 63

Pluggable architecture

Motivation: journalists still need more tools than just viewing! (e.g.: sanitize/redact/archive)
Asking users/orgs to just change our templates?

Update policy + downstream template mods = ouch
dom0’s best not to touch it rule may apply to our templates as well

https://securedrop.org

38 of 63

Pluggable architecture

Motivation: journalists still need more tools than just viewing! (e.g.: sanitize/redact/archive)
Asking users/orgs to just change our templates?

Update policy + downstream template mods = ouch
dom0’s best not to touch it rule may apply to our templates as well

How to go about extending the application then?

https://securedrop.org

39 of 63

Pluggable architecture

We’re looking for a persistent way for our client to discover…

what’s possible to do …
and what to do that with …
while leaving templates untouched.

⇒ RPCs!

⇒ other VMs!

⇒ dom0 based “registry”?

https://securedrop.org

40 of 63

Pluggable architecture

We’re looking for a persistent way for our client to discover…

what’s possible to do …
and what to do that with …
while leaving templates untouched.

Potential target: build a key-value store with qubes.features�similar to qubes.ext.services:ServicesExtension

⇒ RPCs!

⇒ other VMs!

⇒ dom0 based “registry”?

https://securedrop.org

One of the potential avenues we’ve identified to build something like this is by utilising qubes.features. It’s a generic key value store, but by default not readable from inside VMs

However, it’s extensible and another system that uses it is the direct inspiration for this: services. You may be more familiar with the tab for VM settings in Qube Manager, but its primary use is acting as a very lightweight way of telling a VM “turn this systemd service on”. And we believe the same approach could be used to provide a VM-readable key-value store that allows more complex information to be shared persistently across reboots… like which target VMs supports which RPC call, and what localised human readable name that action has, and which RPC calls can be chained in a meaningful way.

41 of 63

Pluggable architecture

We’re looking for a persistent way for our client to discover…

what’s possible to do …
and what to do that with …
while leaving templates untouched.

Potential target: build a key-value store with qubes.features�similar to qubes.ext.services:ServicesExtension

Get configurability without Salt, use more DispVMs?

⇒ RPCs!

⇒ other VMs!

⇒ dom0 based “registry”?

https://securedrop.org

42 of 63

Conclusions

43 of 63

Qubes OS already is a powerful platform for multi-VM applications

qrexec and its policy system are rad as hell

They alone allow for building “native” multi-VM apps

Installing a multi-VM app is hard, but a solvable problem
A lot of the times, solutions to troubles we’re facing may be “discovered” in existing frameworks

https://securedrop.org

44 of 63

45 of 63

46 of 63

Questions?

47 of 63

Get involved!

Twitter: @SecureDrop, @FreedomOfPress
Code: https://github.com/freedomofpress/securedrop-workstation
Our Gitter channel: https://gitter.im/freedomofpress/securedrop
Want help installing SecureDrop? https://securedrop.org/help/
Want to donate? https://securedrop.org/donate

https://securedrop.org

48 of 63

Fini

https://securedrop.org

49 of 63

This layout is not for section titles but for hot takes! 🔥

50 of 63

Useful timeline graphic from Jen that is probably useful for most SD presentations

Jan 2019

Dec 2019

Aug 2019

Apr 2019

SecureDrop 0.11.1

released

January 24, 2019

Emergency release for upstream vulnerability in apt package manager

SecureDrop 0.12.0

released

February 26, 2019

First support for Ubuntu Xenial, updated kernel, workaround for upstream NoScript bug

SecureDrop 0.12.1

released

March 20, 2019

Improved upgrade experience for administrators moving to Ubuntu Xenial

SecureDrop 0.12.2

released

April 25, 2019

Provide broader hardware support for SecureDrop servers

SecureDrop 0.13.0

released

May 29, 2019

Remove support for Ubuntu Trusty (end of life)

SecureDrop 0.13.1

released

June 18, 2019

Bugfix release for Python 3 compatibility issue

SecureDrop 0.14.0

released

July 10, 2019

Mitigations for ongoing attacks against GnuPG keyserver ecosystem

SecureDrop 1.0.0

scheduled

September 17, 2019

V3 onion service support, migration of servers to Python 3 only

SecureDrop 1.2.0

scheduled

December 3, 2019

SecureDrop 1.1.0

released

October 21, 2019

Tails 4 support release

https://securedrop.org

51 of 63

This title heading can handle two lines, so feel free to make these longer

Keep main bullet points succinct

Each level below can go into more detail, but this is more for things for people to follow up on when they ask “can I get a copy of the slides pls?”

Another very important point

More on that in our guide here: https://www.omfgdogs.com/

BONUS: Everyone gets a free onion address!

Offer limited to residents of CA, NY, MI, OR and MA

If you have more points, create/duplicate a new slide for them, do not make text smaller to squeeze it all in!

https://securedrop.org

52 of 63

🌏 SecureDrop is now in 20 languages! 🌏

Arabic
Catalan
Czech
English
German
Greek
Spanish
French

Hindi
Icelandic
Italian
Norwegian
Dutch
Portuguese, Brasil
Romanian
Russian

https://securedrop.org

53 of 63

🌏 SecureDrop is now in 20 languages! 🌏

Slovak
Swedish
Turkish
Chinese, Traditional

https://securedrop.org

54 of 63

Use this caption area to cite source of third-party content: https://www.cyberscoop.com/securedrop-dhs-vulnerability-disclosure-def-con/

https://securedrop.org

55 of 63

Think of this spot as an “alt text” field. Also, link link link: https://twitter.com/EricTrump/status/1170065011634581504

https://securedrop.org

56 of 63

This heading should be < one line, keep it short and let the image shine

https://securedrop.org

57 of 63

Think of this spot as an “alt text” field. Also, link link link: https://twitter.com/EricTrump/status/1170065011634581504

https://securedrop.org

58 of 63

“Sometimes it makes sense to use or remove existing titles/body fields for non-standard stuff like quotes. Do what you gotta do but make it fashion.”

- Ron Wyden, US Senator (OR)

59 of 63

Questions?

60 of 63

Get involved!

Twitter: @SecureDrop, @FreedomOfPress
GitHub: https://github.com/freedomofpress/securedrop
Join our Gitter channel to discuss how to get involved with code, translations, or docs: https://gitter.im/freedomofpress/securedrop
Want help installing SecureDrop at your organization? https://securedrop.org/help/
Want to donate? https://securedrop.org/donate

https://securedrop.org

61 of 63

SDW relies on:

VM isolation
Network isolation and chaining
Templates and OS update infrastructure
Disposable VMs
qrexec and RPC policies

https://securedrop.org

62 of 63

UX challenge focus: printing

TODO: Screenshots here? xpp vs eog printing UI (entire slide is WIP)

SD client print → copy doc → conversion → xpp
eog print → GNOME printing UI → cups/IPP

Completely disparate pipelines with different expectations at different stages

https://securedrop.org

63 of 63

Allow printing from DispVMs

Main motivation: Usability (and aesthetics)
Modern printing infrastructure works by auto-detecting hardware and doesn’t need drivers to print

… and is basically synonymous with, IPP

RPC service to proxy IPP, provide a PDF printer in the source VM, ship result to the target VM with the printer attached to it for actual printing?

https://securedrop.org

To stay with printing for a moment… (and maybe also to emphasise how unusual the problems become if the goal is to make Qubes OS pretend its a normal OS while retaining strong separation). It’s not as straight-forward as one might hope

Currently, we only support 2 printers for which we generate the cups drivers for in every session that attempts to print documents, if the correct printers are attached. That’s a ways away from modern printing infrastructure, where driverless printing has become the norm.�
“Modern printing” is somewhat synonymous with the Internet Printing Protocol. IPP uses a whole bunch of technologies, among them IPP and DNS-Service-Discovery - and even though I haven’t looked at the spec in depth, I very much doubt that we’d be comfortable with just tunneling some ports between VMs to make this happen.

So what are the alternatives? The most obvious one would be to do something similar to what we do with our securedrop.Proxy RPC service, that is, to introduce an architectural bottleneck of sorts, that only allows a subset of interactions with a target VM, instead of _all_ the HTTP. Whether that is viable for a much larger more complicated protocol, more research is certainly needed.

But even if something like this were to become reality, with whatever approach turns out to be the most viable, that would still leave the question of how we handle printing directly from the client, at least if we continue to support this mode of interaction