Why haven’t we solved measuring

engineering organizations?

Will Larson. May, 2023

Some ways measuring engineering goes wrong:

• Measuring what worked at the last job (“At Stripe, we …”)
• Measuring when there’s no trust (“If they look good, they must be wrong.”)
• Worrying about measurements being misused (“But they could misuse this!”)
• Waiting for the perfect metric (“Let’s measure nothing for now!”)
• Judging organizations on optimization metrics (“Let’s grade them on DORA!”)
• Measuring individuals instead of teams (“How many PRs did they ship?”)
• Selecting measurements in isolation (“Let’s just use the ones I like.”)

Example 1: Metrics push at Stripe

Example 2: Eng cost accounting at Uber

Example 3: DORA metrics at $friendsco

Question

Why are there so many recurring challenges in measuring engineering?

Diagnosis

Engineering has too many stakeholders for a universal approach to measurement.

Solution

Think about measurement differently for each group of stakeholders.

Stakeholders for engineering measurement

• Yourself (aka Engineering)
• CEO / Board
• Finance
• Strategic peer organizations (Product, Sales, etc)
• Tactical peer organizations (Customer Success, etc)

Engineering

• Measure to plan
• Measure to operate
• Measure to optimize
• Measure to inspire

Engineering

• Measure to plan: what would help you align with cross-functional stakeholders to decide what to ship in a given quarter, half or year? Critically, this is not about how you ship (“are we using agile?”), rather your goal is to support project selection and prioritization.��Start by tracking the number of shipped projects by team, and the impact of each of those projects
• Measure to operate
• Measure to optimize
• Measure to inspire

Engineering

• Measure to plan
• Measure to operate: what do you need to know to be confident that your software and teams are operating effectively on a day-to-day basis? It can be helpful to think of these as quality measures.��e.g. tracking the number of incidents, minutes of user-facing API and website downtime, latency of user-facing APIs and websites, engineering costs normalized against a core business metric, user ratings of your product, and a broad measure of whether your product’s onboarding loop is completing
• Measure to optimize
• Measure to inspire

Engineering

• Measure to plan
• Measure to operate
• Measure to optimize: what do you need to know to effectively invest time into improving engineering productivity? If you look at engineering as a system, how would you understand that system’s feedback loops?��Start with SPACE or its predecessor, Accelerate. If those are too heavy, then start with a developer productivity survey
• Measure to inspire

Engineering

• Measure to plan
• Measure to operate
• Measure to optimize
• Measure to inspire: how can you concisely communicate engineering’s transformational impact to the business? How would you energize a potential hire about engineering’s contribution to making the business possible?��Maintain a list of technical investments that turned something impossible into something so obvious that it isn’t really worth mentioning

CEO

• Most CEOs manage through inspection, so just something
• Pick a measure, set a goal, report regularly on progress
• Avoid measuring optimization

Finance

• Answer three questions:
• How is actual headcount expense trending towards budgeted headcount expense?
• How are actual vendor costs trending towards budgeted vendor costs?
• Which engineering costs can be capitalized instead of expensed?
• Meet Finance where they are
• Don’t fixate on costs growing slower than revenue
• Create a 1-2 engineer Performance/Efficiency team when you can (cost attribution and fixed allocation towards cost reduction)

Strategic peer organizations

• Use the same metrics that you use for planning

Tactical peer organizations

• Tactical organizations are tactical by design, not by choice
• Appreciate how these organizations are evaluated
• Work to their measurements where it’s rational
• Work with executive team to clarify tactical team’s approach when not rational

Q&A

See lethain.com/measuring-engineering-organizations/