Bringing Rust Home�to Meet the Parents

Jeremy Fitzhardinge�Facebook

jsgf@fb.com

About me

• 30 years in industry, lots of it doing kernel/hypervisor/systems work in pure C
• C: the tool I had, rather than the tool I loved
• Rust the first new language that could replace C in every role�
• 3 years at FB
• First large company I’ve deliberately joined - had to learn to navigate org at scale
• Lots of latent interest in Rust, but nobody doing anything about it
• Why not?

Bringing Rust to Facebook

• Introducing a new language is not just a technical matter
• New languages are intrinsically risky: they’re a long-term bet which might not pay off
• Most people are more interested in getting their job done than playing with a new language
• Therefore Rust needs to look like its worth their time - the best tool for the job�
• Making the case: it doesn’t just need to be better
• It needs to be much better at something
• And at least on par for everything else

Rust’s 10x advantage

Rough rule of thumb: A new language needs to be ~10 times better at something than any of the existing languages to make it worth the effort of introducing and supporting it.�

Rust detects large classes of serious bugs at compile time.��The cost of a bug at compile time is orders of magnitude less�than in production.

Case study: Mononoke

• From scratch rewrite of source control backend
• Immediate goal: solve predicted scaling problem
• Also enable lots of new functionality
• Not just a “Mercurial rewrite”
• Correctness a primary requirement: corrupting source files would be terrible
• Also had to be much faster than Python
• And something that the rest of the team could pick up
• Super early adopter of Tokio+async - day after it appeared on crates.io
• Groundwork: targeted, high-quality integrations
• Build system (Buck)
• Test environment (#[test] -> Testpilot)
• Path to using crates.io

Mononoke production experience

• Rust claims checked out:
• No time spent debugging memory corruption or race conditions
• Deployed as source of truth for one of the large repos
• Still core dumps in production
• Stack overflows
• Unhandled C++ exceptions hitting noexcept FFI boundaries
• Buggy C++ bindings, and underlying C++ code
• Rest of source control team adopted Rust for new development

�Good enough to justify further investment in Rust

The FB Environment

• (Context: backend service codebase)
• Primary languages: C++, Java and Python
• But highly polyglot - there are lots of languages in use
• Teams have a lot of freedom to choose languages and technologies
• But they're typically pragmatic - they have to have a good reason to do so
• Huge codebase
• Millions of files, hundreds of millions of lines, all one big source repository
• General tooling goal: everything feels like the scale of your project, not the scale of the repo
• Lots of specialized tooling to handle this scale
• Basically everything is custom, from source control, build, CI, testing, deployment
• Lots of production infra for monitoring, profiling

How does Rust fit into all this?

Rust view of the world

• Safety. Safety safety safety. Safety is the foundation.
• New language = greenfield to experiment
• Core stability: language + std have careful compatibility story
• Core stability offset by crates.io: vibrant third-party ecosystem for experiments
• Build + dependency managed by Cargo, very low friction path to Rust
• Cargo is the center of the Rust universe
• Cargo treats everything as either “Rust” or “other”
• Non-Rust is a non-goal
• Cargo alone is not enough

How to retain Rust’s essential benefits while working within a wider ecosystem?

Cost of bugs

• When an FB outage hits the headlines, it represents a LOT of lost revenue
• At least some of those root cause to “use after free in some vital service”
• Therefore FB spends a lot of resources on code quality:
• Code review, static analysis (eg Infer), test infrastructure
• But these are all out of the inner development loop
• Engineer has already changed contexts, and needs to return to fix them
• May only flag problems after the code has landed
• Rust solves large classes of errors in the inner loop before a context switch
• While looking like C++ operationally => nearly a drop in replacement

Open Challenge: How to quantify the cost of bugs which didn’t happen?

Expressiveness and review

Some secondary benefits:

• Code reviews are higher level
• Reviewers don’t need to be concerned with low-level details
• Can focus on higher-level architectural design details, why not how
• Lower risk of code maintenance
• Many bugs introduced by the next engineer who doesn’t have the full context in their head
• (Even if it’s the original author a few months later)
• Encoding those constraints into types+lifetimes makes

Surprise converts: dynamic language users

• Most aggressive early adopters of Rust were coming from Python
• They were looking for:
• Startup time improvements
• Regain control of their codebases
• Runtime performance improvement
• Also keen interest from Node/Javascript users
• Fun learning experience: “convert your favorite Python script to Rust”
• Lots of loving rebukes from the compiler, but the end result is working code
• vs C++ where the compiler accepts a lot more, but the bugs are inexplicable

Advantage: responsive CLIs and cross-platform

• Lots of CLI-based programs
• Either:
• C++: responsive but restricted to Linux systems, or
• Python: cross platform, but deathly slow
• Why not both?
• Libraries like clap+structopt give a very good initial impression

Supporting new Rustaceans

• Most teams adopting Rust have no experienced Rust users
• Ramp-up time a universal concern
• Typical experience
• 2-3 weeks: little programs and fighting the compiler
• 3-5 weeks: can generally work out a way to get things done
• 8+ weeks: starting to be consistently idiomatic
• Quite dependent on language background
• Functional => Rust’s constraints are easy to conceptualize
• C++ => Lifetimes make sense
• Go/Java => Harder to migrate from GC’d heap soup
• Danger later on: “Trait mania” sets in, things get needlessly complex
• Not much internal training material needed so far

Building a community

• A local community which extends the external Rust community
• Depends on the external community being vibrant and healthy
• Safety as a foundational principle for a healthy community
• Bootstrapping a review culture
• Teams newly adopting Rust need to get to reviewing their own code
• #rustreviewers to the rescue
• Learning culture - active “learning” group with a wide range of participation
• Solid documentation - hybrid of external and internal
• Remove gratuitous “weirdness”
• Codebase must be approachable by non-Rust programmers
• Solving a bug in production means anyone needs to be able to dive in and make some sense of code - even if they’re not super-familiar with the codebase or language
• Adopt existing conventions where it makes sense, but nothing super alien to Rust

Rust - not just for fans

• Early adopters are already excited
• Time to reach out to people who just want to get their job done
• Rust is just one option among many, and not necessarily the best
• Some arguments for:
• Undetected bugs are expensive
• Intrinsic security, esp for untrusted inputs
• Easy to attract talent - people want to use Rust
• Operationally familiar
• Steep learning curve = get higher faster
• Async is really nice
• ...
• General observations:
• Everyone who has evaluated Rust has ended up using it
• No regrets

Where are we now?

Solid starting point:

• Rust being actively used for several high-value projects
• Many standard APIs available in Rust
• General development experience feels comfortable to FB devs
• Also feels natural to Rust devs
• Lots of ambient enthusiasm

Where are we now?

But:

• Rust not yet the default language for any broad domain
• Non-enthusiasts beginning to pay attention, but will need more convincing
• Not making good use of Rustdoc at all
• Lots of polish still needed

​

This journey is 1% finished.��Want to help? We’re hiring!

EXTRA

Challenges for Rust adoption

• C++ gets a reasonable compile-time safety story in the next few years
• We need to keep an affirmative case for Rust, not just in opposition to C++
• Rust keeps changing quickly
• 6 week cycle is great for small codebases, but unsustainable for large ones
• Back-compat is great, but updates are mandatory if third-party tracks releases closely
• Rust “activation energy” remains too high
• Too hard to use with existing libraries/codebase
• We need to keep refining the FFI story
• Lack of other implementations means there’s a worrying single point of failure
• Rustc should remain the reference implementation, but we should start to encourage others

Language Diplomacy

• Rust interop with other languages will always be important
• Direct FFI function calls has the highest coupling:
• Highest performance
• But hardest to bridge language semantics
• Automating as many of the details as possible very important
• With nice clean build breakages
• Lots of work still to do
• RPC is a good middle ground
• Decoupled services are language-agnostic
• Pervasive Thrift at FB, lots of other options elsewhere
• Also useful in-process
• In any case, need uniform cross-language dependency management

The Tao of Buck

• If you have:
• Explicitly declared dependencies
• Deterministic build steps
• You get:
• Uniform cross-language builds
• Aggressively distributed caching
• Not just cached targets, but entire portions of the dependency graph
• Distributed builds*
• Quick incremental builds
• Essential glue for the codebase
• Rust needs solid integration to fit in
• Easy because it looks a lot like C++ in the details that matter
• Main problem: need to list all sources as dependencies, rather than auto-discover

Testing

• Rust inline unit tests are great
• Low-friction tests are a huge incentive to adding tests
• FB test infra is very comprehensive
• Keeps track of all tests, their results, their flakeyness
• Tracks which tests are affected by which rules, and reruns a minimum
• Answer: interface Rust test framework with test infra
• Auto-generate build targets for tests gives Cargo-like experience
• Small changes to Rust testing framework to complete the integration
• One test per process
• Precise control of which tests get run per invocation
• C++ singletons and statics the main pain

Third Party Code

• Crates.io ecosystem an essential part of the language
• Clearest boundary between internal and external tooling
• Current approach:
• Cargo.toml with dummy package; immediate deps are the curated list for use
• Cargo does version resolution, vendoring and building
• Publish as prebuilt rlib files + Buck dependency rules
• Has worked OK so far, but not scaling to 600+ crates
• Precompiled => stuck with fixed version of rustc
• New approach:
• Convert from Cargo => Buck build rules
• Build from source with Buck
• Main blocker: build.rs
• Need manual intervention because there’s no explicit dependencies, inputs or outputs

RLS

• Deploying RLS took a long time
• RLS’s close coupling with Cargo makes it hard
• Works in a fairly cumbersome way with a large amount of hackery
• Rust-analyzer looks very promising
• Interesting challenge in finding a model which works equally well for
• Typical Rust use case, where most code is Rust
• An FB codebase, where Rust code is interspersed with much more non-Rust