From Python To Go:�How Khan Academy is saving time and money

With your host, Steve Coffman

​

​

What is Khan Academy?

Who am I?

(not a brief philosophical interlude)

I love my job.�I love my co-workers.

I love what we do together.

Where did we start? �GCP App Engine �Python 2 Monolith

​

​

Where did we start? �GCP App Engine �Python 2 Monolith

​

​

This didn’t change. It’s awesome!

Where are we going?

GCP App Engine

Multiple Go Services (27)

GraphQL Between

Where are we going?

GCP App Engine

Multiple Go Services (27)

GraphQL Between

This didn’t change. It’s awesome!

EOL for Python 2 forced us to choose...

We Chose Go

• Fast (which translates to cheaper)
• Rich Ecosystem (and standard lib)
• Simple and Productive
• Maintaining someone else's code is a pleasure in Golang

​

What platform should we run on?

We Chose AppEngine (again)

• Avoid undifferentiated heavy lifting
• Scales Great! (e.g. covid-19)
• Reliable as heck (all-volunteer pager duty)

​

Stick with Monolith or switch to services?

We Chose Services (27)

• Human Scaling as We Grew
• Independent Deployability
• Granular Cost Tracking Aligned to Value

​

How do services communicate between each other?

We Chose GraphQL

• Great Success with React + GraphQL on Frontend
• AppEngine ≠ HTTP/2 (gRPC) (yet)
• Ubiquitous Language Across Go/JavaScript Divide
• Schema Driven, Code Generated

​

How should we divide up into services?

Meet (some of) The 27 Services

How have we learned and transitioned to Go?

Learning Go at Khan

• TDD forecast app
• Learn Go With Tests
• Ardan Labs training
• Cookbook
• Web-workshop

Ways to Safely Transition

• Traffic Shadowing
• Github Scientist
• Side-by-Side Testing
• Using GraphQL Federation!

Side-by-Side Transition at Khan

1. Python monolith
2. Develop in a Go service
3. Python primary, Go secondary
4. Go primary, Python secondary
5. Go only

Side-by-Side

• Python monolith
• Develop in a Go service
• Python primary, Go secondary
• Go primary, Python secondary
• Go only

Side-by-Side

• Python monolith
• Develop in a Go service
• Python primary, Go secondary
• Go primary, Python secondary
• Go only

Side-by-Side

• Python monolith
• Develop in a Go service
• Python primary, Go secondary
• Go primary, Python secondary
• Go only

Side-by-Side

• Python monolith
• Develop in a Go service
• Python primary, Go secondary
• Go primary, Python secondary
• Go only

Side-by-Side

• Python monolith
• Develop in a Go service
• Python primary, Go secondary
• Go primary, Python secondary
• Go only

It’s code time!

GraphQL Schema Changes

Step 1: Python and beginning of Go

extend type Assignment @key(fields: "id") {

id: ID! @external

​

createdDate: Time�}

extend type Assignment @key(fields: "id") {

id: ID! @external

​

createdDate: Time @migrate(from: "python", state: "manual")

}

GraphQL Schema Changes

Step 2: Python and Go Side-by-Side

extend type Assignment @key(fields: "id") {

id: ID! @external

​

createdDate: Time @migrate(from: "python", state: "manual")

}

extend type Assignment @key(fields: "id") {

id: ID! @external

​

createdDate: Time @migrate(from: "python", state: "side-by-side")

}

Side-by-Side Differences

Side-by-Side Differences

Side-by-Side Differences

GraphQL Schema Changes

Step 3: Go serving, Python ignored

extend type Assignment @key(fields: "id") {

id: ID! @external

​

createdDate: Time @migrate(from: "python", state: "side-by-side")

}

extend type Assignment @key(fields: "id") {

id: ID! @external

​

createdDate: Time @migrate(from: "python", state: "migrated")

}

Side-by-Side GraphQL Schema Changes

Step 4: Go only, Python deleted

extend type Assignment @key(fields: "id") {

id: ID! @external

​

createdDate: Time @migrate(from: "python", state: "migrated")

}

extend type Assignment @key(fields: "id") {

id: ID! @external

​

createdDate: Time

}

Side-by-Side Transition at Khan

• Python monolith
• Develop in a Go service
• Python primary, Go secondary
• Go primary, Python secondary
• Go only

What about Mutations?

• Altering persistent data Side-by-side testing is a poor fit
• We need a safe, gradual traffic shift Ideally, internal traffic first

Blue/Green

Release a new version alongside the old version, then switch traffic.

Side-By-Side (Traffic Shadowing)

Release a new version alongside the old version.

Incoming traffic is mirrored to the new version and doesn't impact the response.

Canary

Release a new version to a subset of users, then proceed to a full rollout.

Canaries and You

Canaries and YOU

• Selective release of a feature to a random subset of requests, for testing purposes
• Gradually increase traffic from 0 to 100%
• More than 0%, all ‘internal’ users will get the canary treatment

GraphQL Schema Changes

Step 1: Python and beginning of Go

extend type Assignment @key(fields: "id") {

id: ID! @external

​

createdDate: Time�}

extend type Assignment @key(fields: "id") {

id: ID! @external

​

createdDate: Time @migrate(from: "python", state: "canary", color: "red")

}

Canary GraphQL Schema Changes

Step 2: Go only, Python deleted

extend type Assignment @key(fields: "id") {

id: ID! @external

​

createdDate: Time @migrate(from: "python", state: "canary", color: "red")

}

extend type Assignment @key(fields: "id") {

id: ID! @external

​

createdDate: Time

}

Sooo…

• Engineers are happy and productive
• Efficiency went up
• Costs bending down

What’s Next

• 65% of production traffic served by Go
• Open Source by default
• Sharing our onboarding process

�https://www.khanacademy.org/careers

​

49

Thanks! Any questions?