1 of 14

Lessons learned from a cyber-incident

cedars-sinai.org

2 of 14

IT system downtime preparedness

  • Business Continuity project underway
      • BIA completed – 200+ critical applications within 34 surveyed areas
      • Focus on department level-downtime procedures and interdependencies
      • Updating electronic medical record downtime policy and procedures
  • General downtime plans and tiered response plans exist
  • Last exercise conducted December 2021 on day/night shift
        • AAR developed with some follow-up

​

2

3 of 14

Downtime Tiers

3

4 of 14

Incident begins

      • ~2:45pm: Security notices some network issues. First reports of slowness to helpdesk.
        • Infant/pediatric abduction system – downtime plan implemented
      • ~3:00pm: Level 2 downtime. Impacting data center, electronic medical record, internet access, and other clinical applications.
        • Impacting Marina Del Rey hospital and our outpatient medical network
      • 3:08 (+23min) Everbridge sent to leadership and management. (Email, text, cell phones)
        • Work email may be impacted and message may not go through
        • Discussions about whether Teams is working if command center needs to be activated
        • Decision made that we will use backup system
      • 3:15 (+30min) EIS sends test WebEx message sent via Everbridge – can’t join meeting
      • 3:41 (+56 min) Emergency Department goes INT diversion

4

5 of 14

Incident unfolds

      • 3:46 (+1h1min) Level 1 downtime password sent via Everbridge to all staff and medical staff
        • Utilize downtime devices that have red keyboard and label
      • 3:57 (+1h12min) Hospital Command Center activated
        • Small IT outage command center group utilized
      • 4:07 (+1h22min) Full Hospital Command Center group activated
      • 4:14 (+1h29min) Expanded Hospital Command Center group notified
        • Repeat message for large number of people that are having problems joining
        • Asked management team to round on their areas and determine scope/impacts
        • ED to reopen from INT diversion (4:36, 54 minutes)
        • Command center will open physically on-campus, virtual will continue = hybrid HCC
        • Small team discusses reporting to law enforcement.

5

6 of 14

JRIC connection

      • 4:59 (+2hr14min) JRIC lead submitted via website
      • 5:09 (+2hr24min) Call back from JRIC tip/lead
      • Attacker identified, connect with our EIS technical leads
        • Discuss methods to respond to attack and prevent/mitigate re-occurrence
        • JRIC had a contact, but unable to reach them
      • Connect EIS with FBI and CISA - Cybersecurity and Infrastructure Security Agency
        • No cost assistance
        • Allowed us to make the decisions on sharing info about the incident with other agencies
      • Planning tips
        • Register IT cybersecurity with JRIC ahead of time
        • Register with InfraGard
        • Register with CISA

6

7 of 14

Incident continues and ending

      • 5:10 (+2h25min) Email sent to all CSMC reporting network service interruption
      • 5:30 (+2h45min) Hospital Command Center regroups
        • Most units are reporting systems are coming back, some systems are slow but operational
        • Technology countermeasures are working
        • Determined command center will close physical location, but continue to monitor
      • 8:10pm (+5h25min) E-mail sent to all CSMC reporting services restored, but intermittent slowness and issues may persist. Reminder to escalate issues and utilize downtime procedures if needed.
      • ~9:00pm: Signs attack is resuming; however, attack is successfully mitigated
      • Saturday morning: A few attack attempts occurred overnight, successfully mitigated
          • EIS confident that mitigation measures are working
          • Continue to monitor network health and activity

7

8 of 14

Lessons learned: Communications

  • We did not test the backup virtual command center meeting system/links
      • Costly to test, use only during emergencies
  • Everbridge voice messages stating phone numbers, meeting numbers, and links are not effective
        • Focus on text and email for sending meeting info
        • Short SMS text with no link is best (may not able to access rest of message)
        • Encourage people to enter a second personal email into Everbridge

​

8

9 of 14

Lessons learned: Communications

  • Connecting to virtual command center
        • Used cellular hotspot from laptop (disconnect ethernet cable)
        • Messaging for those on-site to use phones, disconnect from WiFi.
        • Call-in users are not identifiable
        • Multiple apps with similar names – one is used for routine management meetings, another for hospital command center
  • Investigating other off-network communication options
      • Two-way radios?
      • Analog phones?

9

10 of 14

Lessons learned: Resources and Assets

  • Response IT activities blocked legitimate traffic
      • Off-site VPN connections were not working
      • Downtime plans included remote-work, potentially would have had to bring people on-site instead
  • Timeclocks could not connect to internet
        • Back entry for timekeeping
  • Downtime computers
      • Validate computer locations, printing capabilities, and proper configurations
      • Desire for more downtime computers
      • Downtime printers need to be connected to computers, have toner and paper

​

10

11 of 14

Lessons learned: Patient care & Clinical Support

  • Patient care areas and support services need access to Business Continuity terminals
      • Validate the information available matches what is needed for essential care
      • Updating downtime checklists, including for Point of Care devices (complete)

11

12 of 14

Lessons learned: Roles and Responsibilities

  • More education on downtime response is needed
      • Identifying downtime computers
      • Process for using downtime passwords
        • Don’t use 0 and O or 1 and l
      • Downtime forms – familiarization and updates

​

12

13 of 14

Where we are headed

  • Drills and exercises
      • Mini-downtime drills – today!
      • Full-scale exercise in September / October
  • Plan Development
      • Continued work on department level plans
      • Update procedures for

​

13

14 of 14

Questions

14

Ryan.Tuchmayer@cshs.org