1 of 13

Permissiongeddon As It Happened

2 of 13

Thursday, May 16 to Friday, May 17, 2019

8:45 PM Permissiongeddon begins

Salesforce executes script on all orgs that have or have had Pardot in the past. Goal of the script was to modify permissions for the integration user, B2BMA Integration.

For reasons not yet known, the script elevated permissions for all Profiles, giving all Profiles Modify All permissions on all objects within their org.

4:56 AM Salesforce discovers elevated permissions (8 hrs. 11 minutes later). Salesforce blocks access to all instances that have an org. affected (i.e., any org that is or was a Pardot customer)

Salesforce begins running a script to remove all permissions from all Profiles except for System Administrator. After script is executed, access is restored for all users (timing varied widely).

2

3 of 13

Friday, May 17, 2019

10:09 AM JLL users begin experiencing issues

3

4 of 13

Friday, May 17, 2019

11:40 AM Parker Harris releases one of first public statements from Salesforce about the critical incident

4

5 of 13

Friday, May 17, 2019

All Day The internet starts having fun

5

6 of 13

Friday, May 17, 2019

3:00 PM Salesforce Critical Incident Team begins a series of conference calls (30 minutes to 1 hour long, every 1.5 to 2 hours).

9:00 PM Salesforce confirms that they have developed a script to restore permissions to their state before the bad script was run, and that the “fixer script” was being tested.

Initially Salesforce told Admins to begin manually restoring permissions for business critical functions; the “fixer script” would overwrite those changes.

6

7 of 13

Issues and Challenges at this Stage

Ongoing Issues and Challenges:

Accessing the calls was challenging (what conference calls?)

Trust site inaccurate/inconsistent

Initially, some customers without Support were told their cases were not critical and would be addressed during normal business hours (eventually Salesforce said all customers would be given support)

There was lots of misinformation/miscommunication/conflicting information in the first set of calls

Challenges manually resetting permissions:

Platform profiles did not have the standard objects (Accounts, Contacts, etc.)

Managed package permissions cannot be edited

Many sandboxes were also impacted by the script

Did Admins have documentation of permissions prior to incident?

7

8 of 13

Saturday, May 18, 2019

2:30 PM Critical Incident Team still testing and validating fixer script; no ETA on completion of testing. Script takes 30-40 minutes per org (up to 2 hours) and although they plan to run in parallel, no one can answer how long they expect the fixer script to complete on all affected orgs

Ongoing No significant updates, continue to indicate script is being tested, and continue tell Admins to manually restore permissions for business critical functions

5:00 PM JLL begins manually restoring some profiles after a time-consuming, manual comparison with Sandbox

9:00 PM Fixer script successfully executed on NA46. Will execute script on three smaller instances, then execute on all affected instances (more than 30 hours since first discovering the issue)

8

9 of 13

Sunday, May 19, 2019

1:15 AM Critical Incident Team indicates that the fixer script has been run (or is still running) on all affected instances

11:30 AM Fixer script was run on all orgs but skipped 11% of orgs (because those orgs had changed profiles - keep in mind, Salesforce instructed admins to manually modify permissions to restore system functionality)

2:30 PM Salesforce begins executing “mop up script” which is supposed to update the 11% of orgs that were skipped due to manual changes

JLL’s org is partially restored, but profiles that had been manually modified were not restored/reset

8:30 PM Salesforce announces that all affected orgs have been restored to their previous state; this does not appear to be correct

9

10 of 13

Monday, May 20, 2019

4:45 AM Permissions for NA53, NA57 and NA59 are again elevated for all users, and then stripped away 4 hours later.

9:29 AM Access is restored for NA53, NA57 and NA59, but permissions are still wiped.

9:30 AM JLL’s org is still not fully restored even after “mop up” scripts (profiles that were manually modified have to be updated, and permission sets have to be assessed and some are manually fixed). We begin manually assessing and resetting our remaining permissions

2:40 PM JLL restores functionality to all users with only two support cases submitted

All Day Continued focus on restoring permissions for NA53, NA57 and NA59.

All other instances are believed to be complete with the exception of Pardot Sync, Managed Package permissions and Sandboxes

10

11 of 13

Tuesday, May 21, 2019 and Beyond

11:30 AM Permissions for NA47 and NA72 are again elevated for all users, and then stripped away later.

Orgs NA47, NA53, NA57, NA59 and NA72 are reset to “factory” permissions

Log-mining is ongoing to restore these instances

Sandboxes do not have backups to be mined, so Salesforce considering restoring Sandbox permissions using Production, but this is all TBD and timing is unknown

Pardot Sync is restored but some customers are still having issues

Current State

Salesforce still mopping up issues with these five orgs plus Pardot Sync customers

Sandboxes still an open issue

There is ongoing discussion of scripts to update managed packages

Customers asking to be added to a “blacklist” to opt out of mop up scripts

11

12 of 13

Lessons Learned

Keep a “clean” Sandbox

Explore data backup solutions

How can we better document our current config?

Can we backup just metadata?

12

13 of 13

Thank you