8 lessons learned building threat detection systems as an MLE

Jeremy Jordan

Machine Learning Engineer

@jeremyjordan

jeremyjordan.me

Preface

Today’s talk:

• Walk through the process of building a threat detection system
• Relate this to the general process of building ML systems
• Discuss some of the unique aspects of applying ML in the cybersecurity domain

Many thanks to: Brian Jones, Zachary Abzug, Jeremy Hedges, Dan Salo, Cameron Schmauch, Mike Ciavarella, Konstantin Klinger, Sherrod DeGrippo, Dan Rapp, Wes Drone, Kirk Soluk, Tim Hopper, Bronwyn Woods, Brian Lindauer, Rich Harang, Mike Moran, Becca Lynch, Jeremie Vallee, Josh Terry, Joe Duggan …and many more wonderful people I’ve had the opportunity to work with who have influenced how I think about the intersection of cybersecurity and machine learning.

Disclaimer

The views and opinions expressed in this talk are my own and do not necessarily reflect the views and opinions of my current or previous employers.

Cybersecurity products in a nutshell

Implement secure protocols

• Make it hard by default for threat actors to cause harm
• Usually involves various flavors of standards around public key cryptography (e.g. DKIM for email content, FIDO for authentication)

Detect and block threats

• Adoption of the latest secure protocols takes time, we still need to provide protection in the interim
• Helps uncover any weaknesses in current implementations
• Identify and respond to security incidents

+

Cybersecurity products in a nutshell – today’s focus

Implement secure protocols

• Make it hard by default for threat actors to cause harm
• Usually involves various flavors of standards around public key cryptography (e.g. DKIM for email content, FIDO for authentication)

Detect and block threats

• Adoption of the latest secure protocols takes time, we still need to provide protection in the interim
• Helps uncover any weaknesses in current implementations
• Identify and respond to security incidents

+

How do you build a threat detection system?

How do you build a threat detection system?

We require a system which:

• Doesn’t interfere with benign content
• Identifies and blocks threats from causing harm
• Without imposing an undue burden on the users of your product

Lesson 0: Understand the threats you’re trying to detect

• Have internal discussions about the threats relevant to your product
• Use a shared language and taxonomy describing these threats
• Leverage community-led standards in threat classification (e.g. MITRE ATT&CK framework)

Lesson 0: Understand the threats you’re trying to detect

All successful machine learning projects start with a well-defined task.

Lesson 1: Start with rules

​

• Encode your domain knowledge of threats into a set of detection rules
• Focus on “threat fingerprints” which can be specified with high precision
• Build out the data infrastructure required to process events and apply these heuristics

Lesson 1: Start with rules

From Martin Zinkevich’s “Rules of ML”

Lesson 2: Annotate your data

• These labels are critical for evaluating rules during development
• Label your data according to your defined threat taxonomy
• Design your system to collect necessary information to apply labels to your data

The sometimes painful, but always necessary part of building a threat detection system.

Lesson 2: Annotate your data

https://twitter.com/mat_kelcey/status/1413245940631183363

Labeling workflows can be complex as we strive to efficiently label a lot of data.

​

A big challenge in labeling cybersecurity data is having sufficient context.

Lesson 2: Annotate your data

Is this URL a phish? 🐟

​

https://interbank.pe/solicitar/tarjeta/extracash/inicio

Lesson 2: Annotate your data

Is this login attempt legit?

Lesson 2: Annotate your data

What context is required in order to confidently annotate the data? What information do domain experts use? Let’s make sure we’re collecting that!

Lesson 3: Scale your detections with ML

​

• ML models allow us to more easily increase recall when operating at a fixed precision constraint
• Machine learning complements rules-based detection engines, each have unique strengths
• ML models don’t need to immediately start blocking threats, also useful for threat hunting

Lesson 3: Scale your detections with ML

​

Machine learning complements rules-based detection engines, each have unique strengths

Lesson 4: Pay attention when detection systems disagree

• Ideally we want both rules-based and ML detection engines to identify threats
• Discrepancies between detection systems can be an interesting perspective for labeling data
• Highly imbalanced datasets force us to build sampling strategies for active learning

Lesson 5: Build cascading detection systems

• It’s often not feasible to have “one model to rule them all”; instead we build an ecosystem of models working to detect threats efficiently
• Need cheap detection models for high volume data
• Suspicious or uncertain content can be forwarded to more expensive detection models

Lesson 5: Build cascading detection systems

⚖️

efficacy

cost

There’s a need to balance our detection accuracy with the associated cost of detection.

Lesson 6: Focus on the threats (...not the anomalies)

• People do a lot of weird but perfectly benign things on the internet
• We should be evaluating content from the perspective of the threats we’re attempting to detect/block
• Anomalies can still be valuable as a threat hunting tool

weird ≠ malicious

Lesson 6: Focus on the threats (...not the anomalies)

​

Archive of a bank’s website…

​

Is this weird? ✅

Is it a threat? ❌

​

Lesson 7: Mitigate detection errors with design

• We need to have extremely high precision when blocking content
• Shift perspective from binary outcomes (allow or block) to a gradient of possible outcomes
• This is a design problem! How do we treat content that falls into the gray area between benign and malicious?

Lesson 7: Mitigate detection errors with design

Even when we can’t outright block content, we can redirect select users to more secure environments.

Lesson 7: Mitigate detection errors with design

We can make intentional choices about when to increase friction to separate the user experiences between benign and malicious users.

Lesson 8: Understand you’re playing an infinite game

• Threat detection exists in a naturally adversarial environment
• Threat actors are incentivized to find vulnerabilities in our current defenses
• Must stay nimble and adapt to new threats in an ever-evolving landscape

Lesson 8: Understand you’re playing an infinite game

https://twitter.com/maldr0id/status/1569744564642058240

Lesson 8: Understand you’re playing an infinite game

Cybercrime is big business. As we improve our detections against existing threats, there will always be new attack vectors that pop up.

Thanks for listening! (now go enable 2FA)

@jeremyjordan

jeremyjordan.me