A filled-in example of this template is provided at the end of this document.

For the 7 Rules of Causation, visit https://delbius.com/rules-of-causation/.

For all related RCA resources, visit https://delbius.com/resources#rca-guide.

A. Incident Info

B. What happened?

B.1 Incident Summary

Briefly describe the event or issue that occurred. Indicate what happened vs what should have happened. When and where did it happen?

[Start write-up here]

B.2 Initial Actions

What immediate steps were taken to address or remediate the issue?

[Start write-up here]

C. Why did it happen?

State the key causes using simple causal statements, taking care to note the nature of the cause: policy, system, human error, or process failure. (See also: 7 Rules of Causation.)

[Start write-up here]

D. How can we prevent it from happening again?

Create an action plan, assign owners for each action, and set a timeline. Identify immediate remediation actions and actions that address the root causes and contributing factors (RC/CF).

These actions have been assigned with the relevant task tickets created/linked: 

E. Measuring success

E.1 Set quantifiable metrics

What metrics will you use to confirm that the action(s) worked?

[Start write-up here]

E.2 Define how data will be collected and by whom

Who will collect and report on these metrics?

[Start write-up here]

E.3 Monitor impact

Decide how long the impact of the action will be tracked.

[Start write-up here]

F. Follow-up: Review Outcome

Did the action resolve the issue? Are further improvements needed? Outline any additional actions if required.

[Start write-up here]

For all related RCA resources, visit https://delbius.com/resources#rca-guide.

FILLED-OUT TEMPLATE EXAMPLE BELOW

Texts that are styled as hyperlinks are used for illustration purposes only.

A. Incident Info

B. What happened?

B.1 Incident Summary

Briefly describe the event or issue that occurred. Indicate what happened vs what should have happened. When and where did it happen?

On 28 August 2024, the Policy team published OPSUPDATE-037 to add a new set of slang/terms to the Allow List. This policy change is intended to reflect how different marginalized communities have reappropriated these terms.

The policy change was scheduled to take effect on 2 September 2024 at 12:00 PM PT. Unexpectedly, the auto-classification bots continued to classify user posts that included the slang/terms as violative.

From 12:00 PM PT on 2 September 2024 until the auto-classification bots were paused, a total of 94,673 posts were wrongly flagged for content violations, affecting 43,862 unique users and resulting in these user accounts being placed under temporary account restrictions.

The issue was first identified because Anjali Bahri (T&S Policy) noticed on 3 September that user complaints about their accounts being restricted due to these slang/terms continued despite the planned policy change.

B.2 Initial Actions

What immediate steps were taken to address or mitigate the issue?

Upon discovering the issue on 3 September 2024, the Policy team filed incident ticket INC-544.

The Incident Manager On Call filed a ticket with the Heuristics team (HEUR-934) to request that the auto-classification of posts for the relevant terms be halted pending investigation.

The Heuristics team successfully paused the relevant auto-classification bots on 3 September 2024, at 2:23 PM PT, pending the integration of OPSUPDATE-037.

Actual integration of OPSUPDATE-037 was completed on 3 September at 3:45 PM PT (HEUR-935), at which point, the bots were unpaused, and normal auto-classification operations resumed. The bots were configured to start reviewing all posts made since the moment they had been paused.

On 4 September 2024, the Internal Tools team identified the affected users based on audit logs (INTTOOLS-1021) and lifted account restrictions (INTTOOLS-1022) for all accounts after confirming that the accounts had not also been flagged as violative by other mechanisms.

The affected users were notified about the error and subsequent lifting of account restrictions via an in-app notification (INTTOOLS-1023) on 4 September 2024.

C. Why did it happen?

State the key causes using simple causal statements, taking care to note the nature of the cause: policy, system, human error, or process failure. (See also: 7 Rules of Causation.)

Process Failures

• The moderation tool’s algorithm was not updated to handle the new slang and context variations documented in OPSUPDATE-037 because the Heuristics team member who was auto-assigned the ticket based on the on-call rotation had started parental leave earlier than expected.
• There is no established process to ensure that already-assigned tickets are reassigned when the on-call rotation is retroactively updated. As such, OPSUPDATE-037 was not bundled with other planned updates and did not ship as scheduled.
• There is also no established process within the Policy team for confirming that planned policy updates shipped as scheduled.

D. How can we prevent it from happening again?

Create an action plan, assign owners for each action, and set a timeline. Identify immediate remediation actions and actions that address the root causes and contributing factors (RC/CF).

These actions have been assigned with relevant task tickets created/linked.

E. Measuring success

E.1 Set quantifiable metrics

What metrics will you use to confirm that the action(s) worked?

The actions taken will be considered successful if:

1. No further incidents arise due to on-call tickets not being reassigned when a team member goes on leave.

Numerator: number of incidents arising from un-reassigned tickets = 0
Denominator: total number of incidents.

2. Daily email alerts are generated for 100% of the cases where an on-call ticket is not reassigned within 24 hours of the on-call rotation getting a retroactive update.

Numerator: number of email alerts sent
Denominator: total number of un-reassigned tickets that should have triggered an email alert.

E.2 Define how data will be collected and by whom

Who will collect and report on these metrics?

These steps will be used to track and monitor the relevant metrics.

• Incident Cause Logging. The ticket for any incident that is found to be caused by on-call tickets not being reassigned will have “On-Call Ticket Not Reassigned” tagged as the official incident cause.
• Quarterly Incident Post-mortem Meeting. The Quarterly Incident Post-mortem Meetings will now include a review of incidents caused by the non-reassignment of on-call tickets.

The owner of the Quarterly Incident Post-Mortem Meeting will be responsible for collecting and presenting the relevant metrics at the meeting.

E.3 Monitor impact

Decide how long the impact of the action will be tracked.

The impact of the [Address RC/CF] actions identified in earlier sections will be monitored for one quarter (21 September to 21 December) to evaluate effectiveness.

The effectiveness of the changes will be assessed at the next Quarterly Incident Post-mortem Meeting, currently scheduled for 23 December.

F. Follow-up

Did the action resolve the issue? Are further improvements needed? Outline any additional actions if required.

If needed, follow-up actions will be identified at the discretion of the Quarterly Incident Post-mortem Meeting owner during the next quarterly meeting.