1 of 20

Run Towards the Fire

Why Incidents Supercharge Teams

Lisa Karlin Curtis

@LisaKC_�@lisa-kc.bsky.social

2 of 20

@LisaKC_�@lisa-kc.bsky.social

One of my first coding jobs was at a company called GoCardless. I’d been there for a few months when we had a major incident - our API had slowed to a crawl.�I was pretty curious so I jumped into the incident channel.

One endpoint in particular was consistently timing out - so we disabled it to get the system back up and running. Step 1 complete.

Now, we had to understand what had happened. There weren’t any recent changes that looked suspicious, so our attention shifted to the database. It turned out the query plan for this particular query had changed, from something that was expensive but manageable, to something that was not at all manageable - doing large sequential scans on the database and eating up all our CPU.

We made a subtle change to the query which made the database revert to the ‘good’ query plan. Everything was back up and running: we’d fixed it.

Well, I say we. I watched quietly from the sidelines furiously taking notes. After the incident was over, I turned to a senior engineer in my team.

Daniel, what is a query plan?

We’ll come back to this in a second

3 of 20

Lisa Karlin Curtis

Founding Engineer at incident.io

@LisaKC_�@lisa-kc.bsky.social

4 of 20

@LisaKC_�@lisa-kc.bsky.social

5 of 20

Incidents broaden

your horizons

@LisaKC_�@lisa-kc.bsky.social

Incidents broaden your horizons

As engineers, we live in a world full of black boxes. Whether that’s a programming language, a framework, or a database - we learn how to use the interface and move on; if we tried to understand how everything worked down to the metal we’d never get to ship …well … anything.

Incidents force you to open the black boxes around you, peek inside, and learn just enough to solve the problem.

After this incident, I read up on query plans. This proved very useful: this was not our last query plan incident. It was also useful for building new things - I was suddenly able to write code that scaled well the first time, rather than using trial and error to discover I was missing an index.

Incidents give you great signal about which of these black boxes are worth opening, and give you a real-world example that you can use as a starting point.

6 of 20

Incidents teach you to build things that fail gracefully

@LisaKC_�@lisa-kc.bsky.social

Incidents teach you to build systems that fail gracefully

One of the key follow-ups from the API incident was to add a default statement timeout for all of our database calls. That meant that, if we issued a bad query, Postgres would try for a few seconds and then give up.

This is an excellent example of resilient engineering: our system can now handle unexpected failures. We don’t need to know what will issue a bad query. Just that it’s likely that something will.

Of course, it’s possible to learn this by reading about these ideas in a book, but nothing compares to seeing it in action.

During this incident, I learned a whole set of tools that I could employ to reduce the blast radius of potential failures - not just the statement timeouts which we implemented, but all the other options that the incident response team discussed and discarded.

7 of 20

Incidents are a chance for blue sky thinking

@LisaKC_�@lisa-kc.bsky.social

8 of 20

Incidents teach you to make systems easier to debug

@LisaKC_�@lisa-kc.bsky.social

9 of 20

Incidents build your network

@LisaKC_�@lisa-kc.bsky.social

10 of 20

Incidents are a chance to learn from the best

@LisaKC_�@lisa-kc.bsky.social

11 of 20

@LisaKC_�@lisa-kc.bsky.social

12 of 20

@LisaKC_�@lisa-kc.bsky.social

13 of 20

How do we make incidents accessible?

@LisaKC_�@lisa-kc.bsky.social

14 of 20

Encourage everyone

to participate

@LisaKC_�@lisa-kc.bsky.social

Encourage everyone to participate

Incidents are great learning opportunities, and they should be accessible to everyone. Incident channels should be public by default, and engagement encouraged at all levels.

Of course there can be too much of a good thing: having 20 people descend into a minor incident channel may not be the best outcome. But most incidents can comfortably accommodate a few junior responders tagging along.

It doesn’t have to come at the cost of a good response - you can get this experiences in a low-risk environment - either by asking questions to someone who’s not actively responding to the incident or saving them up for after it’s resolved.

There are also other ways to gather learnings: reading debrief documents or attending post-incident reviews are both great ways of getting value from your team’s incidents.

You could even compile a list of the best incident debriefs to share with new joiners - they’re a great way to started in a new company.

15 of 20

Show your working

@LisaKC_�@lisa-kc.bsky.social

Get into the habit of showing your working

In an incident, your team should put as much information as you can into the incident channel. What command did you run, what theory have you disproved.

If you’re debugging on your own this admittedly can feel a bit strange: I’ve been sat at 10pm in an incident channel having a conversation with myself. But it’s worth it, I promise.

It’s useful for your response as it means you don’t have to rely on your memory to know exactly what you’ve already tried and when.

It’s also beneficial for your teams. By writing everything down, you’re enabling everyone to learn from your experience.

Just because it’s obvious to you, doesn’t mean it’s obvious to everyone.

That means using public Slack channels wherever possible, and having a central location where everyone can go to find incidents that they might be interested in.

I know I’m biased, but using an incident management platform like incident.io really does help with this one.

16 of 20

Watch out for anyone playing the hero

@LisaKC_�@lisa-kc.bsky.social

Watch out for anybody playing the hero.

Often, a single engineer takes on a lot of the incident response burden, fixing everything before anyone knows it’s broken. Maybe that was you.

This doesn’t end well for the hero: they’ll stop getting as much credit as they expect for fixing things as it becomes normalised, and are at risk of burning out.

But it also causes problems for the rest of the team. Without meaning to, the hero is taking away these learning opportunities from everyone else by fixing things quietly in the corner. That means that nobody else is ever going to be able to do what they do as effectively - no-one’s had enough practise. While that’s perhaps an effective job preservation tactic, it’s not going to result in a high performing team.

If you think you get a lot of recognition for resolving incidents, imagine how much you’ll get for levelling up your whole team.

17 of 20

Declare

lots of incidents

@LisaKC_�@lisa-kc.bsky.social

18 of 20

Declare lots of incidents

Keep it simple

Make it easy to declare and run an incident: avoid unnecessary�red tape.

Reward transparency

Build a culture where raising an incident is seen as responsible, not as admitting failure.

Develop champions

Cultivate leaders across your organisation who will model great incident response.

1

2

3

@LisaKC_�@lisa-kc.bsky.social

To do this we need to

Keep it simple: make it easy to declare and run an incident. If there’s too much red tape, and it doesn’t feel valuable, your team will avoid the unnecessary process. Make the process clear and concise: use a tool to automate as much of it as possible to help guide people through the process.
Reward transparency: build a culture where you celebrate those who identify and escalate issues, instead of blaming or even shaming them. Far better to know what’s gone wrong and start fixing it as a team, than to not even know something’s wrong at all
Develop champions: like all organisational change, you need to nurture leaders on-the-ground to model the behaviour you’re looking for. Choose a handful of your team who can model great incident culture, and invest in them first.

19 of 20

incident.io/guide

@LisaKC_�@lisa-kc.bsky.social

20 of 20

Thanks for listening

Workshop: Incident Resolution in 2025 @ 15:55

@LisaKC_�@lisa-kc.bsky.social