1 of 53

Engineering Beyond Blame

July 11, 2019

SRCCON 2019

Joe Hart & Vinessa Wan

Hello, everyone! Welcome.
If you're here, if you've shown up for our discussion today it means you're here because the idea of an organization that operates without blame something that interests you. Maybe your organization already operates this way and you're looking to share how successful that's been. Or maybe you're hoping to bring some of the things we'll talk about back to your team. Either way, we're both very happy to have all of you here in this room today.�
My name is Joe Hart and I'm a software engineer at The New York Times. My team at the Times is dedicated entirely to running experiments on how we engage with registered users on our website who haven't yet subscribed. Our work is backed partially by data and partially by hunches, and a lot of times we're wrong about what we think users are going to read, and it isn't a mistake if we spent time knowing what didn't work.
I'm Vinessa...(Vinessa talk about Delivery Engineering and SRE (1 - 2 min))

2 of 53

Before we begin

3 of 53

Our time together

DISCUSSION

EXERCISE

FRAMEWORK

JOE

VINESSA

Our time together will consist of four parts. Vinessa and I discussed how best to co-facilitate our discussions. We did a couple of practice runs and we settled on splitting this in half. I'm gonna warm y'all up and she's gonna bring it home.
We're gonna kick things off with a group discussion. Our topic: failure. It wont be difficult. I promise I wont put anyone under pressure to confess any event they don't feel comfortable sharing. I'm hoping we talk about a time you depended on something to happen just right and it didn't work out.
I'd like to use that time to get us to share our stories and identify with one another's experiences. That'll segue nicely into a group exercise that I think is going to be really fun. You're all going to encounter some issues during this exercise that'll take us through varying levels of failure and that'll set things up nicely for Vinessa to talk to you all about Learning Reviews.
I'm going to discuss the framework, what problems it intends to solve, and how we apply that framework at The Times.
Once you get a feel for how they work, we're going to revisit a specific failure in the exercise and apply what we've learned.

4 of 53

Use the restroom

Totally don't ask for permission! Listen to your bodies.

Ask questions

No need to wait until the optimal time.

Laptops are okay

We have paper and pens here if you need to take notes, but if you prefer a laptop, that's okay.

Timeboxed discussions

So that you can enjoy the other talks. If we missed anything, talk to us after.

Phones, not so much

If there's an emergency, no problem. Step out and come back when you're ready.

Some ground rules

We have a couple of ground rules for today

First off, please use the bathroom when you need to. Don't feel like you have to ask permission or feel guilty for standing up in the middle of us talking. Please listen to your own bodies.
As we're talking, please feel free to jump in with a question. Don't wait until the end. We've got a lot of material to cover so just ask away and we'll do our best to answer. We may respond that, we'll cover that in the next section, but please don't take that as us trying to dismiss you.
We'd like for you to all get out of here on time, so please don't take it personally if we timebox some discussions. I'm gonna use my phone here with a chime that'll ring. If it goes off, it means let's talk after.
Laptops are ok if you're taking notes, though I do request that you keep your phones on vibrate. So if you can, take a moment now to do it. If there's something urgent that buzzes your phone and you need to take it, again, no problem, just step out and come back when you're ready

How's every feel? Yeah? Alright, let's get started.

5 of 53

Discussion: Joe

6 of 53

Share your story about breaking production on election night 2012. I had just joined the times.

When I was managing the CMS team, I broke the editing tool the newsroom uses to write their stories, and during that downtime people weren't able to save their work and file their stories.
I've broken so many things at The Times, I'm known for those very public pratfalls more than I'm known for the successes I've had. It's true. People who were around for that 2012 election production bug remember it vividly and continue to tell stories about how much mayhem there was.
I often make this joke "It's amazing they haven't fired me yet". Has anyone here ever heard or even said that about themselves?

So how many of you all here, have made huge mistakes, whether in your current job or a previous job.

For those of you who have your hands up, how many of you are able to talk about it, of course withholding any details your organization may not feel comfortable talking about?
(Can you share)?

Follow up questions to ask the sharerer

Was there ever a post mortem or a retro about what happened?
Was anyone at "fault"?

If no, ask if they or anyone felt like a person was at fault.
If yes, ask how they arrived at that conclusion

Can anyone think of a time where they depended on someone on their team and that person couldn't deliver. They utterly failed and there was palpable disappointment and possibly resentment towards that person, if not from you then perhaps from someone on your team who was really upset with them.

Can you talk about that incident? Why did someone think
What did it feel like to be on this team? Did you feel comfortable talking about your mistakes?

7 of 53

Why is it easy for humans to resort to blame?

Are we just innately driven towards retribution?

So let me ask y'all this. Why is it so easy for human beings to fall into this trap? Why do think you think we resort to blame? Are we just innately driven towards retribution?

Any thoughts?

If no thoughts

Can someone share a time they showed compassion in the face of failure? Can someone talk about a time someone was having a really bad day and you showed them infinite patience and compassion?

What did that feel like?
Did you have to train your mind to not find fault with them?

I often do. Perfectionism is a piece of armor I wear to guard against my own personal feelings of inadequacy and anxiety. I feel like I can never overprepare for something. There's no such thing. And when things go wrong I have to actively fight against thoughts of how I should've known better or prepared better. My mind instantly goes to the blaming place to try to make sense of it. Does this behavior resonate with anyone here?

8 of 53

Why is blame so common?

Explanation: We have a deep need to understand and explain why things happen.

Motivator: Likely to result in reduction in the actions that we believe caused something.

Lench, H. C., Domsky, D., Smallman, R., & Darbor, K. E. (2015). Beliefs in moral luck: When and why blame hinges on luck. British Journal Of Psychology, 106(2), 272-287. doi:10.1111/bjop.12072

Why do we tend to blame ourselves or other people when things go wrong?

We're human beings. We have deep desire to explain the world around is. It helps us predict the future. It provides us a sense of security to be able to predict if something will go wrong. "Don't give the big assignment to James and Susan because they screwed up last time." It also makes us feel powerful because it allows us to claim expertise. And with that expertise it leads to people looking up to us, either in admiration (or fear) of our power.
We also use it as a form of punishment and as a way to reduce the likelihood of an outcome by reducing the actions a person took which we believe caused an outcome. It also conditions people into behaving according to a certain set of rules. Just the fear of being blamed, and the consequences of this, is often enough to gain compliance. In this way, blaming is also a way of sending signals to others.

9 of 53

Why is blame so common?

Incapable: We’re not very good at figuring out the causes of other people's behavior, or even our own.

It's Easy: It’s easier to blame someone else than to accept responsibility.

Lench, H. C., Domsky, D., Smallman, R., & Darbor, K. E. (2015). Beliefs in moral luck: When and why blame hinges on luck. British Journal Of Psychology, 106(2), 272-287. doi:10.1111/bjop.12072

10 of 53

Some organizations default to the path of least resistance when there's an outage or when a mistake happens that's very public. Some of the tools we bring out in these instances are blame, hindsight bias, correspondence bias. They give us a narrative that is easy to digest, easy to tell our bosses, and easy on us because it absolves us of doing any hard work. And if that story reinforces any preexisting beliefs about the world or the organization or system we're in, we believe that it's true. We like easy answers that follow a pattern. We're pattern-seeking organisms. Patterns make us feel comfortable because we can give predictability to something unknown. But the downfall here is that when we do this, we're missing out on an opportunity for growth. By avoiding the difficult and uncomfortable feelings of dealing with the complexity of our organizations, we're cheating ourselves and our organization of what we could've learned. Some outcome happened as a result of a failure. By narrowing it down to one root cause, and that root cause being a human, we're in-fact compounding the problem by never discussing the areas of fragility within it.

11 of 53

Exercise: Joe

12 of 53

Let's do a group exercise

to demonstrate this point.

13 of 53

14 of 53

Your goal

Work together to open your restaurant

15 of 53

16 of 53

It's Opening Day!

How have you prepared?

17 of 53

The toaster isn't working.

Take a moment: How would you solve this problem?

18 of 53

You run out of

Chocolate Pop Tarts.

19 of 53

Ok, talk to me. Let's start here.

How did you troubleshoot and solve the toaster oven?

How about y'all. Did you have similar method?

How about the Pop Tarts running out. How did you troubleshoot and solve this?

So if I'm hearing everyone correctly, there were clear cause-and-effect relationships that were easily discernible by everyone. The right answers were self-evident and I imagine undisputed. Since you all have access to the information necessary for dealing with the situation, directives were straightforward (plug the toaster in), decisions can be easily delegated (should we buy more chocolate pop tarts). Some of these problems can be automated, (if the number of boxes goes below this amount, order more.)

So this was an easy one. I wanted to give us something we could all do well with for our first restaurant. Let's look at a slightly more challenging restaurant.

20 of 53

So congrats on your amazing pop shop launch! It was such a success that you caught the attention of a reality-tv star. This a chef who was featured in a popular TV cooking show, and they became world famous with their signature hamburger. This chef wants to partner with you to open their first hamburger restaurant. They want to be inclusive to folks with dietary restrictions so you're offering a few options on the menu, not just beef hamburgers. You accept cash, credit, and mobile payments with Venmo.

If you pop over to the next sheet in your handouts you'll see the menu.

We've got

Hamburger (options: with cheese, fries)
Meat-free burger (options: dairy-free cheese, gluten free bun)
Fountain soda
Coffee
Water (bottles)

This time we've got a few constraints.

Your chef is the only one who knows how to make the items, but they recently published a cookbook that has all of their recipes. It's a New York Times best seller.
You must have separate utensils and cooking areas for meat and meatless dishes.

Any questions? Alright, same as before. Let's do 5 minutes. Work together and open this shop. Go!

21 of 53

It's Opening Day!

How have you prepared?

22 of 53

The french fries aren't coming out as crispy.

23 of 53

Your chef leaves abruptly to do a book tour.

24 of 53

Ok let's come back together, y'all. Talk to me. How did you troubleshoot the french fry issue.

Interesting. Did anyone else have a different way of solving this? Let's hear from this group.

So if I'm hearing you all correctly, it sounds like there may be multiple right answers. And though there is a clear relationship between cause and effect, not everyone can see it. Did anyone have an answer that was, "call a repair person to check out the fryer?" That's my first reaction. I don't really know how the fryer works, but I know a subject matter expert can solve it.

How about your chef leaving. Who had some ideas about this problem? How about this group.

So it sounds like, if I'm hearing you all correctly, most of the data about the problem is in front of you, and you might need someone's help to solve it depending on which route you go. None of us know how to make the hamburgers, but thankfully someone can learn it. I don't know if the fryer's broken, but someone can fix it if we decide to call someone. There are a lot of right answers and you as leaders kept an open mind and heard everyone's perspective, and in the end, with a combination of your expert's input, you can reach a decision. Did that feel accurate?

Awesome

25 of 53

Alright y'all. Let me tell you. You are crushing it in the restaurant business. All of your ventures are a huge success and guess what? You've drawn the attention of a world-class, Michelin-star chef. This chef has two Michelin stars and they are gunning for a three-star restaurant with an elite French steakhouse. There's a famous interior designer who built out the space and it is truly remarkable. Your reservation system, ordering, and kitchen is all digital, networked, and state of the art. You accept only credit card.

You have one constraint in this round. You can't run this place without your chef. They are the only person who can deliver the quality that the items on the menu deserve. They cannot be replaced.

Your menu is a chicken braised with red wine or a filet mignon. You have a huge wine list. Folks, this is your final round in this exercise. This is the apex of your careers as restaurateurs. Do you have any questions before we start?

Ok. 4 minutes.

26 of 53

It's Opening Day!

How have you prepared?

27 of 53

Customers complain that their wine orders are incorrect.

28 of 53

The next day customers become violently ill.

29 of 53

Ok folks. Time is up. Time is up and I have a question for y'all. How many of you tried to nail down a single root cause for the customers getting sick? Can I hear them? Did you blame the chef or the cooking staff?

This problem is far more complex than a broken toaster. In our burger restaurant, at least one right answer exists. Maybe the fryers broken. Maybe the staff learns to cook from the recipe book. In this realm, right answers can’t be ferreted out because we're dealing with incomplete data. We don't know the whole story. We can't solve the problem in the moment; we can understand why things happen in retrospect. It feels dirty to impose a course of action in this state because all we have are assumptions.

We don't know until we get everyone who worked that night back into a room. We need to know who exactly got sick, what they ordered, who served them, who made the food, with what utensils, with what procedure, maybe call the butcher and see what's up with the cuts. Maybe it's all one table that got sick? We wont know until everyone comes forward and is open an honest about what happened. And so you all do that. You tell the staff, let's all come in and talk about everything that happened.

But there's a problem.

30 of 53

"Whoever sabotaged my third star is finished!"

31 of 53

Eliminate Punishment

Remove blame and punishment from the equation. Without safety, it's unlikely there will be honesty.

32 of 53

Information is Accountability

When we remove blame, we're value the information people have more than we value punishing them.

And some of you here might be thinking that without punishment, we're not holding people accountable. And you're not wrong. That's one way to look at it. But if you're looking to improve your organization's health and increase the stability of your system, then consider the notion that the information they give us is how we are going to hold them accountable. Everyone involved in the incident is absolved of all wrongdoings provided they give us the full story. They give us the missing data we need as leaders to see the full picture because they're accountable for the system's health. We all are. We adopted this method of accountability in The New York Times tech organization two years ago. And I'd like to turn this discussion over to my friend, colleague, and our Principal Program Manager at The Times, who is a gladiator in this arena of rumbling with large-scale failures and accountability, Vinessa Wan.

33 of 53

Learning Reviews: Vinessa

34 of 53

Agenda

Introduction
Exercise
Structure & Framework
Discussion
Wrap-up

35 of 53

Who’s Involved?

Organizer

Participants

Report Writer

Facilitator

Scribe

Involved in the Incident

Not Involved in the Incident

Organizer

Was directly involved in the incident. Organizes the date, time, room, and invites for the Learning Review (see guidance on who to invite here), as well as providing the timelines and pre-requisite documentation for distribution before the Learning Review.

To organize the meeting where the incident can be discussed, and clear action items going forward established to prevent it happening again.

Facilitator

Someone not involved in the incident.

A neutral party to facilitate and set expectations in the Learning Review. Helps to create a safe space where meaningful conversation can be had surrounding the incident.*however it should be noted that creating this “safe space” is everyone's responsibility. Makes sure that all perspectives are being heard. Their job is not to ask the

Scribe

Someone not involved in the incident. This role will allow all involved to be fully engaged in the session, not focusing on writing notes and recording the points and decisions made.

To take notes during the Learning Review as a record of the conversations had, and decisions made in terms of action items and next steps.

Report Writer

Summarize the Scribe’s notes and distribute to participants. Usually the same person as the Organizer.

To communicate the key decisions made, next steps and action items.

Participants

Everyone who was directly involved in the incident. Typically includes: engineers, designers, product owner, project manager, sometimes includes editors, marketing and/or advertising dependent on the nature of the incident.

To contribute directly to the discussion and establishment of action items and next steps.

*for details about who this should and should not include, and how many people to invite, see Who attends Learning Reviews?

36 of 53

Structure of a Learning Review

Typically a 60 minute meeting
3 phases

Timeline review (70%)
Success and failures (15%)
Action items (15%)

Not an exact calculation

A learning review typically lasts about 60 minutes or more.

It's uncommon for them to be 90 minutes, but sometimes it's necessary if the event we're learning about is quite large, like elections for example.
When Dave Zwieback gave his workshop, he encouraged us to do over 90 minutes, but he stressed that we should do what feels natural for us. There's a reason why we landed at 60 minutes, and we'll talk about that a little later in this section

A learning review consists of three phases

The bulk of the meeting is spent running through the events of what happened. Every learning review we attend or facilitate is based around a timeline of events that happened. When we talk about Roles and Responsibilities in this section, we'll dig into the timeline more thoroughly
We save about 15% of our time to review our successes and our failures because, again, to truly learn about a system means we have to make time to examine the things that went well and even the places where we got lucky
Once we've gotten a full picture of how our system was functioning, we're able to spend some time on remediation steps

Just note these are not exact calculations. It's not exactly 15% here or 70% there. I'm using these just to illustrate where your focus will be for most of the review.

37 of 53

Questions to Ask

What worked well?

What didn’t work?

Where did we get lucky?

Where can we improve?

1

2

3

4

5

Anything else?

38 of 53

Learning Review Structure

Typically a 60 minute meeting
3 phases

Timeline review (70%)
Success and failures (15%)
Action items (15%)

(Not an exact calculation)

Expectations

Get clarity on something unclear
Meaningful incident report

What worked well?

What didn’t work?

Where did we get lucky?

1

2

3

Where can we improve?

4

Anything else?

5

A learning review typically lasts about 60 minutes or more.

It's uncommon for them to be 90 minutes, but sometimes it's necessary if the event we're learning about is quite large, like elections for example.
When Dave Zwieback, author of Beyond Blame, gave his workshop, he encouraged us to do over 90 minutes, but he stressed that we should do what feels natural for us. There's a reason why we landed at 60 minutes, and we'll talk about that a little later in this section

A learning review consists of three phases

The bulk of the meeting is spent running through the events of what happened. Every learning review we attend or facilitate is based around a timeline of events that happened. When we talk about Roles and Responsibilities in this section, we'll dig into the timeline more thoroughly
We save about 15% of our time to review our successes and our failures because, again, to truly learn about a system means we have to make time to examine the things that went well and even the places where we got lucky
Once we've gotten a full picture of how our system was functioning, we're able to spend some time on remediation steps

Just note these are not exact calculations. It's not exactly 15% here or 70% there. I'm using these just to illustrate where your focus will be for most of the review.
When people attend a learning review, they have a set of expectations about how it's going to run and what they're going to get out of it

Typically people are in the room because an incident emerged out of situation where something we didn't quite understand or expect had results we didn't want. And so they expect to get a little more clear about something unclear or confusing
Once they get clarity, it's expected that each learning review produce a report of everything we learned so we can widely publish the results. The goal being that both successes and failures need to become part of institutional memory at any learning organization.

If we're able to meet these expectations, people will feel like the review was a success.

But this is very broad, right?

Get clarity.
Get a meaningful report
Can we maybe get specific here?

39 of 53

Complex Systems

Don’t apply a simplistic linear cause-and-effect model to investigating trouble with complex systems.

Failure is Normal

Failure is an expected part of the functioning of complex systems.

All systems fail.

Conditions, not causes

Look for the many conditions that allowed a particular situation to manifest.

Learn from Success

Pay attention to what went RIGHT too, to make our systems more resilient.

Human Error

Is A Symptom

Cognitive Biases

Help each other become aware of cognitive biases that influence the way we see the situation.

Seek to understand why it made sense for people to do what they did.

Mindset

40 of 53

How to Set Up Learning Reviews for Success

41 of 53

All systems will fail, it's just a matter of time. All of them.

42 of 53

The blameless mindset

is a dependency for a meaningful learning review.

43 of 53

Prioritize for psychological safety and transparency

44 of 53

Source: ThinkReliablity https://www.thinkreliability.com/cause-mapping/what-is-root-cause-analysis/

45 of 53

It is not a postmortem!

Postmortems revolve around a single root cause

(of death!)

and neglect the positives.

46 of 53

Learning Reviews will not solve all problems.

47 of 53

Discussion: Vinessa

48 of 53

Agenda

Introduction
Exercise
Structure & Framework
Discussion
Wrap-up

49 of 53

Let's talk about this.

Can we apply this to the exercise we just did?

50 of 53

Revisit the French Restaurant

How would you go about creating a psychologically safe space?
How would you hold people accountable in a blameless way and affect change?

51 of 53

Agenda

Introduction
Exercise
Structure & Framework
Discussion
Wrap-up

52 of 53

Wrap Up

53 of 53

Rolling Out Learning Reviews

Phase	Challenges
Awareness	Why change may be needed Blamelessness as a concept and mindset
Interest	Giving sufficient guidance, especially for Facilitator and Organizers Creating community of knowledge
Adoption and Growth	Building out community of knowledge to meet growing needs Faux LRs and bad practices LR on trend Encouraging sharing
Optimizing	Evaluating the process Checking in on progress - Are action items being address? Learning approved.