Go Protobuf

Michael Stapelberg, Zürich Gophers 2025-April-2

Agenda

5-minute refresher on Protobuf

01

Newly released: The Opaque API

Migrating millions of lines of code

02

03

5-minute refresher on Protobuf (Protocol Buffers)

• “Google’s language-neutral data interchange format”
• define what data to encode/decode in .proto files:

edition = "2023"; // successor to proto2/3

package log;

​

message LogEntry {

string backend_server = 1;

uint32 request_size = 2;

string ip_address = 3;

}

5-minute refresher on Protobuf (Protocol Buffers)

The Protobuf workflow:

5-minute protobuf refresher: Logs Analysis Pipeline decodes a message

for {

b := … // read from network or disk�

var entry logpb.LogEntry // protoc made this!

err := proto.Unmarshal(b, &entry)

if err != nil { … }

� requestsByIP[entry.GetIpAddress()]++

}

​

// Store aggregated statistics report…

5-minute protobuf refresher: Why not JSON?

• You might be thinking: Why not just use JSON?!
• Protobuf includes schema validation
• backwards- and forwards-compatible!
• Protobuf has cleaner type semantics, e.g. int64
• (JSON numbers are arbitrary-precision floats)
• Protobuf encodes more efficiently (faster, smaller output)
• but: not human-readable like JSON

Latest development:�Opaque API launch�in December 2024

Opaque API: What’s changing?

Let’s recap the Protobuf workflow:

The Opaque API is a new

version of the generated code

Background: Generated code (old Open Struct API)

package logpb // generated by protoc

​

type LogEntry struct {

BackendServer *string

RequestSize *uint32

IPAddress *string

// …internal fields elided…

}

​

func (l *LogEntry) GetBackendServer() string { … }

func (l *LogEntry) GetRequestSize() uint32 { … }

func (l *LogEntry) GetIPAddress() string { … }

​

Background: Field presence

• “field presence”: is a message field set or not set?�Go Protobuf models presence using pointers
• entry.BackendServer = proto.String("zrh01.prod")�→ field is set and contains string zrh01.prod
• entry.BackendServer = proto.String("")�→ field is set and contains an empty string
• entry.BackendServer = nil // field not set

Background: Field presence over the years

• proto2 used explicit presence (pointers)�
• proto3 used implicit presence (no pointers)�→ field considered “set” if different from its zero value�proto3 later got explicit presence (optional keyword)�
• edition 2023+ uses explicit presence by default

The Opaque API uncouples the�Generated Code API from the�in-memory representation!

Generated code (new Opaque API)

package logpb // generated by protoc, Opaque API

​

type LogEntry struct {

xxx_hidden_BackendServer *string // not exported

xxx_hidden_RequestSize uint32 // not exported

xxx_hidden_IPAddress *string // not exported

// …internal fields elided…

}

func (l *LogEntry) GetBackendServer() string { … }

func (l *LogEntry) HasBackendServer() bool { … }

func (l *LogEntry) SetBackendServer(string) { … }

func (l *LogEntry) ClearBackendServer() { … }

​

Opaque API: Why? Less memory usage!

• Using pointers adds a 64-bit word for each field
• The Opaque API uses bit fields for presence�…for elementary fields (less profitable for other types)
• Requiring fewer allocations is faster:

│ Open Struct API │ Opaque API │

│ allocs/op │ allocs/op vs base │

Prod#1 360.3k ± 0% 360.3k ± 0% + 0.00% (p=0.002 n=6)

Search#1 1413.7k ± 0% 762.3k ± 0% -46.08% (p=0.002 n=6)

Search#2 314.8k ± 0% 132.4k ± 0% -57.95% (p=0.002 n=6)

​

Opaque API: Why? Lazy decoding!

• All field access now goes through accessors�
• We can safely delay decoding until actual field access�…but messages are still verified at Unmarshal() time�
• For applications that only work with a subset�of a large message hierarchy, this saves lots of work!

Opaque API: Why? Safety! More optimizations!

• The new API prevents safety mistakes like accidental sharing, incorrect pointer comparison or accidental use of Go reflection. → go.dev/blog/protobuf-opaque �
• More optimizations: Using profiles to choose the ideal memory layout for a specific program…?

Migrating well over 100 million�lines of Go code

Migration: the open2opaque tool

• Migration Guide: protobuf.dev/reference/go/opaque-migration/�
• open2opaque rewrite <Go-packages>�
• Complexity ladder of changing Go code:�find&replace < regexp < AST < typed AST < wholeprog

Migration: the open2opaque tool (2)

• uses the standard library to work with Go code:�go/parser (read) + go/types (type checking),�go/ast (change) + github.com/dave/dst (comments),�go/format (write)�
• uses golang.org/x/tools/go/packages �…which uses the go tool, or blaze (Google-internally),�to load an entire Go package for analysis/rewriting

Migration: the open2opaque tool (3)

• open2opaque is a multi-pass rewriting tool! 🤯�if f := entry.BackendServer; f != nil {� *f = strings.Replace(…) }��the has pass first rewrites the code to:�→ if entry.HasBackendServer() { � *entry.BackendServer = strings.Replace(…) } ��then, the set pass picks that up and rewrites it to:�→ if entry.HasBackendServer() { entry.SetBackendServer(strings.Replace(…)) }

Migration: scaling it up!

• Tell all developers to migrate the code and wait? No!�
• “Churn policy”: 80% of the migration work must be covered by the team, 20% can be pushed to others�
• Large-Scale Changes (LSC) team helps with expertise, process reviews, Global Approvals�

Migration: scaling it up!

• open2opaque + MapReduce* = nightly full analysis�(Static analysis teams build all code every night!)�
• We can query dependencies and produce reports:�“What code in my dependencies do I need to migrate�before my project can flip the switch?”�
• Dashboard for current status is great for motivation

Migration: scaling it up!

• Next step: don’t just analyze, rewrite that code!�→ Nightly pipeline produces diffs�
• We have an internal tool (think like GitHub’s dependabot) which takes such diffs and sends PRs*

Opaque API

• → saved hundreds of thousands of CPU cores! 🚀�→ landed over 100k commits with .go code changes!�→ 98% of Google Production is on the Opaque API!�
• We hope you’ll migrate to the Opaque API, too!

Thank you for�your attention!

→ protobuf.dev �→ go.dev/blog