1 of 44

Rust and rustc intro

Daniel Cumming

1

Runtime Verification Inc.

2 of 44

Overview

  • What is Rust
  • rustc (Rust compiler) general overview
  • rustc intermediate representations
  • Interrupting rustc to examine intermediate representations (Callbacks)
  • Replacing the code generation
  • Hacking on rustc and compiling incrementally

2

3 of 44

What is Rust?

3

4 of 44

What is Rust

4

“A language empowering everyone to build reliable and efficient software.” - rust-lang.org

5 of 44

What is Rust

5

“A language empowering everyone to build reliable and efficient software.” - rust-lang.org

  • Systems level language
  • Control over memory allocation
  • Compiles to many targets
  • Interoperable with foreign languages

6 of 44

What is Rust

6

“A language empowering everyone to build reliable and efficient software.” - rust-lang.org

  • Can run on embedded systems
  • No garbage collector
  • Systems level language
  • Control over memory allocation
  • Compiles to many targets
  • Interoperable with foreign languages

7 of 44

What is Rust

7

“A language empowering everyone to build reliable and efficient software.” - rust-lang.org

  • Systems level language
  • Control over memory allocation
  • Compiles to many targets
  • Interoperable with foreign languages
  • Can run on embedded systems
  • No garbage collector
  • Type system delivers strong guarantees:
    • Memory safety (no buffer overflow, no use-after-free, etc..)
    • Thread safety (data race freedom)
  • All enforced at compile time:
    • No mutation of shared data
    • No aliasing of mutable data

​

8 of 44

What is Rust

8

  • Type system delivers strong guarantees:
    • Memory safety (no buffer overflow, no use-after-free, etc..)
    • Thread safety (data race freedom)
  • All enforced at compile time:
    • No mutation of shared data
    • No aliasing of mutable data

​

“A language empowering everyone to build reliable and efficient software.” - rust-lang.org

  • Systems level language
  • Control over memory allocation
  • Compiles to many targets
  • Interoperable with foreign languages
  • Can run on embedded systems
  • No garbage collector

Assuming no inappropriate use of unsafe keyword

9 of 44

What is Rust

9

  • Type system delivers strong guarantees:
    • Memory safety (no buffer overflow, no use-after-free, etc..)
    • Thread safety (data race freedom)
  • All enforced at compile time:
    • No mutation of shared data
    • No aliasing of mutable data

​

“A language empowering everyone to build reliable and efficient software.” - rust-lang.org

  • Systems level language
  • Control over memory allocation
  • Compiles to many targets
  • Interoperable with foreign languages
  • Can run on embedded systems
  • No garbage collector

Safely possible with appropriate use of unsafe keyword

Assuming no inappropriate use of unsafe keyword

10 of 44

What is rustc?

10

11 of 44

What is rustc?

11

  • rustc is the program that compiles rust source language to a target binary
  • Github for rust compiler is here
  • Rust compiler has stable and nightly releases (in progress features and bugs)
  • The rust compiler is bootstrapped (the rust compiler is written in rust)
  • Active community, mostly at the Rust Zulip (great place for compiler questions)
  • Probably if you used rust in the past you used cargo which calls rustc for you
  • Self reminder: -Z flags, compiler, library, x

12 of 44

Stages of the Rust compiler

12

13 of 44

Rust Source Code

13

fn main() {

println!("Hello, world!");

}

​

  • NEXT: Transform source code into AST (Abstract Syntax Tree)

14 of 44

Rust IR: Abstract Syntax Tree

14

An abstract syntax tree (AST) is a data structure used in computer science to represent the structure of a program or code snippet. It is a tree representation of the abstract syntactic structure of text (often source code) written in a formal language. Each node of the tree denotes a construct occurring in the text. - Wikipedia (AST)

15 of 44

Rust IR: Abstract Syntax Tree

15

  • Lexing
  • Parsing

​

  • Macro expansion
  • Name resolution

​

  • NEXT: Transform the AST to HIR (High Intermediate Representation)

An abstract syntax tree (AST) is a data structure used in computer science to represent the structure of a program or code snippet. It is a tree representation of the abstract syntactic structure of text (often source code) written in a formal language. Each node of the tree denotes a construct occurring in the text. - Wikipedia (AST)

16 of 44

Rust IR: High IR

16

The HIR – "High-Level Intermediate Representation" – is the primary IR used in most of rustc. It is a compiler-friendly representation of the abstract syntax tree (AST)... Many parts of HIR resemble Rust surface syntax quite closely, with the exception that some of Rust's expression forms have been desugared away. - rustc-dev-guide

17 of 44

Rust IR: High IR

17

The HIR – "High-Level Intermediate Representation" – is the primary IR used in most of rustc. It is a compiler-friendly representation of the abstract syntax tree (AST)... Many parts of HIR resemble Rust surface syntax quite closely, with the exception that some of Rust's expression forms have been desugared away. - rustc-dev-guide

  • Lowering
  • Desugaring

​

  • Type inference
  • Trait solving
  • Type checking

​

  • NEXT: Transform the HIR to THIR (Typed-High Intermediate Representation)

18 of 44

Rust IR: Typed-High IR

18

THIR is a lowered version of the HIR where all the types have been filled in, which is possible after type checking has completed. But it has some other interesting features that distinguish it from the HIR… - rustc-dev-guide

19 of 44

Rust IR: Typed-High IR

19

THIR is a lowered version of the HIR where all the types have been filled in, which is possible after type checking has completed. But it has some other interesting features that distinguish it from the HIR… - rustc-dev-guide

  • Desugaring

​

  • Unsafety checks

​

  • NEXT: Transform the THIR to MIR (Middle Intermediate Representation)

20 of 44

Rust IR: Middle IR

20

[MIR] is a radically simplified form of Rust that is used for certain flow-sensitive safety checks – notably the borrow checker! – and also for optimization and code generation. - rustc-dev-guide

21 of 44

Rust IR: Middle IR

21

[MIR] is a radically simplified form of Rust that is used for certain flow-sensitive safety checks – notably the borrow checker! – and also for optimization and code generation. - rustc-dev-guide

  • Transform from AST-like to CFG

​

  • Drop Elaboration
  • Borrow Checking

​

  • NEXT: Transform MIR to LLVM IR (Assuming LLVM is the Codegen Backend)

22 of 44

Code Generation

22

Code generation (or "codegen") is the part of the compiler that actually generates an executable binary. Usually, rustc uses LLVM for code generation… - rustc-dev-guide

23 of 44

Code Generation

23

Code generation (or "codegen") is the part of the compiler that actually generates an executable binary. Usually, rustc uses LLVM for code generation… - rustc-dev-guide

  • Different options (LLVM, GCC, Cranelift, …)
  • Constant Evaluation
  • More Lowering (SSA)
  • Monomorphisation

​

  • Optimisation

​

  • NEXT: Transform from LLVM IR to ASM (target binary)

24 of 44

Target / Binary

24

  • The normal rustc / cargo experience

25 of 44

Callbacks

25

26 of 44

rustc_driver module

26

  • Enables us to run the rustc as part of a running rust program
    • fn main() {� let _ = rustc_driver::RunCompiler::new(...).run();�}

​

  • Requires adding rustc crates as foreign modules
    • extern crate rustc_driver;

​

  • Requires adding some components to rustup to add rustc crates in
    • rustup component add rust-src rustc-dev llvm-tools-preview

27 of 44

Callbacks

27

  • after_crate_root_parsing
    • Post parsing
    • Pre macro expansion

​

​

  • after_expansion
    • AST

​

​

  • after_analysis
    • Post borrow checking
    • Pre monomorphisation

28 of 44

CodegenBackend

28

29 of 44

CodegenBackend

29

  • Codegen can be replaced

​

​

  • Can be replaced with another supported backend
    • GCC for example

​

​

  • Can be replaced with your our custom backend
    • Kani is a good example

30 of 44

Building rustc

30

31 of 44

31

Questions?

https://runtimeverification.com/

​

@rv_inc

​

https://discord.com/invite/CurfmXNtbN

​

contact@runtimeverification.com

​

​

32 of 44

  • Lexing

32

Lexing takes strings and turns them into streams of tokens. … This is implemented in rustc_lexer. - rustc-dev-guide

33 of 44

  • Lexing

33

fn main() {

println!("Hello, world!");

}

​

Lexing takes strings and turns them into streams of tokens. … This is implemented in rustc_lexer. - rustc-dev-guide

fn

main

(

)

{

println!

“

Hello, world!

”

)

;

}

34 of 44

  • Parsing

34

Parsing takes streams of tokens and turns them into a structured form which is easier for the compiler to work with, usually called an Abstract Syntax Tree (AST) . - rustc-dev-guide

35 of 44

  • Parsing

35

Parsing takes streams of tokens and turns them into a structured form which is easier for the compiler to work with, usually called an Abstract Syntax Tree (AST) . - rustc-dev-guide

fn

main

(

)

{

println!

“

Hello, world!

”

)

;

}

pub struct Stmt {

pub id: NodeId,

pub kind: StmtKind,

pub span: Span,

}

36 of 44

  • Macro Expansion

36

Given a raw source code for a crate, the compiler will produce a massive AST with all macros expanded, all modules inlined, etc. - rustc-dev-guide

37 of 44

  • Macro Expansion

37

Given a raw source code for a crate, the compiler will produce a massive AST with all macros expanded, all modules inlined, etc. - rustc-dev-guide

println!("Hello, world!");

std::io::_print(

std::format_args_nl!("Hello, world!")

);

  • rust-analyzer: Expand macro recursively at caret

38 of 44

  • Name Resolution

38

The name resolution in Rust is a two-phase process. In the first phase, which runs during macro expansion, we build a tree of modules and resolve imports. Macro expansion and name resolution communicate with each other via the ResolverAstLoweringExt trait.

The input to the second phase is the syntax tree, produced by parsing input files and expanding macros. This phase produces links from all the names in the source to relevant places where the name was introduced. It also generates helpful error messages, like typo suggestions, traits to import or lints about unused items.

- rustc-dev-guide

39 of 44

  • Lowering / Desugaring

39

The AST lowering step converts AST to HIR. This means many structures are removed if they are irrelevant for type analysis or similar syntax agnostic analyses. - rustc-dev-guide

40 of 44

  • Lowering / Desugaring

40

The AST lowering step converts AST to HIR. This means many structures are removed if they are irrelevant for type analysis or similar syntax agnostic analyses. - rustc-dev-guide

for _i in 0..10 { continue; }

match #[lang = "into_iter"](#[lang = "Range"]

{ start: 0, end: 10 }

mut iter => loop {

match #[lang = "next"](&mut iter) {

#[lang = "None"] {} => break,

#[lang = "Some"] { 0: _i } => { continue; }

}

}

};

​

41 of 44

  • Type Inference

41

Type inference is the process of automatic detection of the type of an expression. It is what allows Rust to work with fewer or no type annotations, making things easier for users… - rustc-dev-guide

42 of 44

  • Type Inference

42

Type inference is the process of automatic detection of the type of an expression. It is what allows Rust to work with fewer or no type annotations, making things easier for users… - rustc-dev-guide

fn main() {

let mut things = vec![]; // things: Vec<&str>

things.push("thing");

}

43 of 44

  • Trait Solving

43

44 of 44

  • Type Checking

44

Type "collection" is the process of converting the types found in the HIR (hir::Ty), which represent the syntactic things that the user wrote, into the internal representation used by the compiler (Ty<'tcx>) - rustc-dev-guide

The process of verifying and enforcing the constraints of types - Wikipedia (Type Checking)