2 of 23

Last week you measured.

This week you diagnose.

RECAP: SESSION 1

Produced a baseline: numbers and observations from a real run of the OrbitTasks pipeline.

TODAY: SESSION 2

Take that data and figure out why it looks the way it does. Identify bottlenecks and underlying causes.

3 of 23

10 min: Profiling techniques: how to read what your machine is telling you
20 min: Apply profiling to your baseline data from Session 1
10 min: Root cause analysis: the 5 Whys and the fishbone
10 min: Apply root cause analysis to the OrbitTasks pipeline
10 min: Prioritize the bottlenecks and discuss how senior engineers do this work

Agenda

4 of 23

Profiling,�not guessing.

5 of 23

THE HONEST DEFINITION

What is�profiling?

Profiling is the practice of measuring where a program actually spends its time, rather than guessing.

01. THE GUESS (OPINION)

"I bet the database is slow"

An assumption made without real data. This leads to wasted effort, incorrect diagnoses, and endless speculation.

02. THE MEASUREMENT (EVIDENCE)

"Step X took 14s, Step Y took 0.2s"

Concrete, empirical numbers that reveal the exact truth. Once you have these, the engineering conversation shifts from opinion to evidence.

6 of 23

Two flavors of profiling.�Both are useful.

01. INSTRUMENTATION

Explicit timing around blocks of code

Cheap, simple, and surgical, but you only see what you remember to measure.

`scripts/measure.sh` is pure instrumentation.

02. SAMPLING

Periodic stack-trace recording

Heavier and noisier, but reveals unexpected performance bottlenecks you'd never think to measure.

Tools: Node's `--inspect`, `perf`, `time`

7 of 23

Today's tools are deliberately simple.

THE APPROACH

You don't need a flame graph generator to make real progress. The most useful profiling tool is the one you actually run.

We will rely on three key inputs plus your own perspective.

01. HISTORICAL BASELINE

`baseline.log`

The timing output from last week's `npm run ci` run.

02. TEST RUNNER LOGS

Raw Test Output

Jest and Vitest both print individual per-test durations.

03. SHELL INSTRUMENTATION

`time` and `date +%s`

Wrap any command to capture precise wall-clock execution time.

04. HUMAN INTUITION

Your Own Brain

Key patterns matter just as much as raw statistical numbers.

8 of 23

How to read a build log line by line.

What jumps out

The point isn't memorizing format; it's learning to scan for outliers and to be suspicious of any single number that's much larger than its neighbors.

01. STAGE DURATIONS

Identify long stages

Walk through the baseline.log to find stages that are observably long.

02. TEST DOMINANCE

Analyze the test stage

Pinpoint exactly which individual tests dominate the test stage.

03. ANOMALIES & FLAKES

Spot non-deterministic outcomes

Scan for outliers and trace any unexpected variation in results.

9 of 23

Apply it�to your data.

10 of 23

Workshop • 15 Minutes

Now together.�Open your�`baseline.log`.

We will analyze and rank your pipeline stages live to establish our baseline target.

01. LOGBOOK & RANKING

Rank your pipeline stages

Fill the W2 section of your logbook as we rank stages together from fastest to slowest.

02. FIND THE BOTTLENECK

Identify the primary target

Circle the single biggest contributor. For example, `test:api` typically accounts for ~95% of total runtime—this is our first target.

11 of 23

Hot spots and the long tail

A pattern you'll see for the rest of your career

Most pipelines are dominated by a small number of slow steps. Senior engineers learn to spot both the obvious bottlenecks and the silent accumulators to protect system health.

01. TODAY'S FOCUS

The "Hot Spot"

Usually one or two slow operations take more time than everything else combined. These are our first targets for optimization.

02. FUTURE SESSIONS

The "Long Tail"

Dozens of small operations that each look cheap individually, but add up to create major delays. We will address these starting in Session 3.

12 of 23

Why the long tail is dangerous

It hides in plain sight

Senior engineers learn to be just as suspicious of "lots of small things" as they are of "one big thing." Today, we must learn to note both to protect our pipeline's long-term health.

01. The Obvious Problem

"One Big Thing"

A single 30-second slow test is easy to spot. It stands out immediately in any performance log as the primary culprit.

02. The Hidden Danger

"Lots of Small Things" (The Long Tail)

Ten tests taking 3 seconds each look perfectly fine individually. Together, they create the exact same 30-second problem in disguise.

13 of 23

Symptoms�vs causes.

14 of 23

"The test is slow" is a symptom.

The root cause lives further down

When test:api takes ~12 minutes, "the tests are slow" is true but not useful. Why does it take so long?

By performing root cause analysis, we dig past the obvious symptoms to find a process or convention that we can actually change.

01. THE SYMPTOM

test:api takes ~12 minutes

02. THE MECHANISM

Integration tests make real HTTP round-trips

03. THE GAP

There is no mock layer for external services

04. ACTIONABLE ROOT CAUSE

No test-infra convention existed

15 of 23

The 5 Whys:�the simplest tool that works.

Invented at Toyota in the 1950s. Still in active use at every engineering org you'll work at.

The technique is exactly what it says: ask "why?" five times, in sequence. By the fourth or fifth "why?" you've usually moved from a symptom to something you can actually change.

01. The Symptom

Why is the pipeline slow?

Because test:api takes ~12 minutes.

02. Digging Deeper

Why does test:api take ~12 minutes?

Because a suite of integration tests each make many real HTTP round-trips.

03. Identifying the Barrier

Why real HTTP?

Because there's no mock layer for the external service clients.

04. Root Cause ACTIONABLE

Why is there no mock layer?

Because there was no test-infra convention.

16 of 23

Worked example, together

5 Whys on a real OrbitTasks bottleneck

As a group, we apply the 5 Whys to apps/api/tests/reports.test.ts and the tests/integration/ suite, specifically analyzing the many real HTTP round-trips each test makes.

01. THE CRITERION

Don't move on until everyone has traced it back to a root cause about process or convention, not just the specific line of code.

02. THE GOAL

Process root causes are what you actually design and implement fixes for.

17 of 23

What it is

A multi-branch diagram for analyzing categories of causes

Categories typically include People, Process, Tools, Environment, and Materials. Each branch holds the contributing factors you can think of.

The visual structure forces you to look beyond the most obvious answer.

When to use it

When a single-line 5 Whys is no longer enough

Use it when 5 Whys gives you one answer but you suspect there are several contributing causes.

Flaky tests are a classic fit: the failure is rarely just one thing. We'll draw one together for the flaky test on the next slide!

When 5 Whys isn't enough,�draw a fishbone.

18 of 23

Apply RCA�to your repo.

19 of 23

WORKSHOP • 10 MINS • TOGETHER

Pick your top three.

Select Bottlenecks

Take the three biggest bottlenecks you identified in Part 1 to analyze.

Run 5 Whys

Drill down into each bottleneck, recording the analysis in your logbook as we go.

Root Cause

Target process, not code.

Land on how the team works. If you get stuck on "someone wrote it that way," push one more level.

20 of 23

Not every bottleneck is worth fixing.

A 2×2 of impact and effort. Top-right wins are obvious.

The interesting question is the rest: how we evaluate trade-offs to prioritize what actually matters.

We rank your three together now and record them in your logbook.

High Impact • Low Effort

Fix it this sprint

No discussion needed. These are your obvious quick wins.

High Impact • High Effort

Write a proposal

Get the team aligned. These projects need executive buy-in.

Low Impact • Low Effort

Good for warm-ups

Great tasks for new hires and nice low-stakes wins.

Low Impact • High Effort

Kill it. Politely.

These tasks almost always lose priority to something else.

21 of 23

Why senior engineers care about this so much

CAREER RELEVANCE

The Junior to Senior Shift

Work shifts from "fix things" to "diagnose what to fix."

The Staff Interview Focus

Every staff engineer interview asks you to trace a hard bug to its source (or about how you did it)

Mastery Through Practice

The underlying technique is the same. The depth comes from practice.

22 of 23

SESSION 03

CI/CD pipeline design and configuration.

You'll take the bottlenecks you identified today and start fixing them, for real, using GitHub Actions.

The next session is hands-on heavy.

Pipeline Structure

Understand how a modern CI pipeline is structured.

GitHub Workflows

Write your own .github/workflows/ci.yml for OrbitTasks.

Optimization & Speed

Apply caching and parallelization, then measure the improvement.

1 of 23