1 of 55

Do you know where your secrets are? Exploring the problem of secret sprawl and secret management maturity

@mcdwayne

2 of 55

Hi, I’m Dwayne

Dwayne McDaniel

I live in Chicago
I’ve been a Developer Advocate since 2016
On Twitter @mcdwayne
mcdwayne@mastodon.social
Happy to chat about anything, hit me up
Outside of tech, I love improv, karaoke and going to rock and roll shows!

@mcdwayne

3 of 55

About GitGuardian

GitGuardian is the code security platform for the DevOps generation.

—

We help enterprises answer the issue of "Where are my hardcoded secrets and have they been leaked?"

@mcdwayne

4 of 55

Code Leaks Are A Problem

@mcdwayne

5 of 55

“I must not fear. Fear is the mind-killer. Fear is the little-death that brings total obliteration. I will face my fear. I will permit it to pass over me and through me. And when it has gone past I will turn the inner eye to see its path. Where the fear has gone there will be nothing. Only I will remain.”

― Frank Herbert, Dune

Bene Gesserit Litany Against Fear

@mcdwayne

6 of 55

CircleCI – January 2023

Development System

CircleCI Platform

Embedded Malware Used To Steal Secrets

Remote Engineer

Hardcoded AWS, GCP, Google Drive, Internal employee dashboards, unencrypted customer data …

Customer secrets stolen, unauthorised access to GitHub repos and third-party systems

CircleCI Internal Network

Attacker

Customer Application

Hardcoded AWS, GCP, Google Drive, Internal employee dashboards, unencrypted customer data …

Customer Application

Hardcoded AWS, GCP, Google Drive, Internal employee dashboards, unencrypted customer data …

Customer Application

GitHub

@mcdwayne

7 of 55

Uber Breach – September 2022

2FA/MFA spamming

VPN Access

Scanning

Uber’s Infra

PowerShell scripts in network share

PowerShell script contained hardcoded credentials

Credential access

Discovery

Initial access

Privilege escalation

Admin access to

Thycotic PAM

Access to internal apps

AWS, GCP, Google Drive, Slack, SentinelOne, HackerOne admin console, Internal employee dashboards…

Uber Internal Network

Lateral movement

1 Exposed Secret

One exposed secret leads to many others!

Some mistakes are more expensive than others. One hardcoded secret giving access to Uber’s PAM solution led to a an organization wide IT takeover…

@mcdwayne

8 of 55

A Few Incidents

Toyota

Reported: 7 October, 2022
A subcontractor hired to work on the Toyota T-Connect source code pushed a private codebase into a public GitHub repo.
The repo contained access credentials for a data server, which exposed the emails of 296,019 customers
The repo was public from December 2017 to September 2022 - 5 years!

@mcdwayne

9 of 55

A Few Incidents

AstraZeneca

Reported: 3 November 2022
Developer hardcoded credentials and pushed to GitHub in 2021, giving access to test environments
"User error" caused an undisclosed amount of patient data to be available in a test environment
Credentials were exposed for over a year

@mcdwayne

10 of 55

https://www.gitguardian.com/state-of-secrets-sprawl-report-2023

@mcdwayne

11 of 55

@mcdwayne

12 of 55

@mcdwayne

13 of 55

@mcdwayne

14 of 55

@mcdwayne

15 of 55

@mcdwayne

16 of 55

@mcdwayne

17 of 55

@mcdwayne

18 of 55

Stop Hardcoding Secrets�� Use secrets managers like �Hashicorp Vault or Azure Key Vault

�

@mcdwayne

19 of 55

Three pillars of a secrets management program

Train devs to use the tools and follow the processes

People

Tools

Automate detection and remediation

Use vaults, secrets managers and KMS to manage secrets

Processes

Document the steps to follow for incident remediation by devs

Create clear processes for provisioning, manaaging, and rotating secrets

Raise awareness around secrets sprawl and secure coding practices

@mcdwayne

20 of 55

blog.gitguardian.com/a-maturity-model-for-secrets-management

Level 0

Uninitiated

Level 1

Beginners

Level 2

Intermediate

Level 3

Advanced

Level 4

Experts

No processes or tools for managing secrets – secrets sprawl in the SDLC.

No detection (and remediation) in place.

Secrets are unencrypted at rest and shared across teams. Scanning for secrets is triggered manually at times,

Secrets are scoped, stored in a vault and shared using a secrets manager. Automated detection on shared repositories and final artifacts is continuous.

Secrets are checked encrypted into repositories with decryption keys stored in a secure vault.

Secrets scanning and rotation are performed periodically.

Secrets are scoped, stored and called from a vault

Detection is preventive and integrated into dev workflows.

@mcdwayne

21 of 55

Level 4 – Experts

No presence of valid hardcoded secrets in past or current revisions of source code

Scanning before pushing code (pre-commit) is adopted by all developers
Developers are systematically involved in the remediation process)

Secrets are stored in a vault with access controls and logging
Dynamic secrets with limited scope are used for development

Secrets management

All repositories are continuously monitored and blocking scans (pre-receive) are setup
Remediation workflows are automated and fixing is handled by developers

Pipeline secrets are short-lived, scoped, and stored in an external vault
If possible, build secrets are replaced by OpenID Connect (OIDC) tokens for auth.

Blocking scanning is enforced for all branches (feature, hotfix, etc.) in CI pipelines

Secrets are stored in a vault and loaded dynamically from a secrets manager
Restrictive access controls and logging are enforced

Secrets are scheduled for regularly automated rotation

Secrets detection

Developer environments

Source Control

(Source code

& Infra-as-Code)

CI/CD pipelines &

software artifacts

Runtime environments

Limited exposure risk

@mcdwayne

22 of 55

@mcdwayne

23 of 55

What Attackers Want

Machine Resources
Access To Data
Anything That Leads To 1 or 2

@mcdwayne

24 of 55

Cyber Deception

Use the attacker's SOP against them

@mcdwayne

25 of 55

A Brief History Of Cyber Deception

@mcdwayne

26 of 55

Trojan Horse

~ 1200 BCE

"I thought I was getting a horse, what I got was defeat"

- Trojan security officer

Brief History of Deception

@mcdwayne

27 of 55

Art of War

~ 400 BCE

"Appear weak when you are strong, and strong when you are weak"

- Sun Tzu

Brief History of Deception

@mcdwayne

28 of 55

Ghost Army

1942

"The first mobile, multimedia, tactical deception unit in U.S. Army history"

- James Linn - Curator National WWII Museum

Brief History of Deception

@mcdwayne

29 of 55

The First Honeypot

1985

"Hi, is this the FBI? At my girlfriend's suggestion, I used fake documents to trick someone working with the KGB into keeping their connection to a Lawrence Berkeley National Laboratory computer open long enough to trace their exact location."

- Cliff Stoll

Brief History of Deception

@mcdwayne

30 of 55

Fred Cohen’s Deception Toolkit

1991

"Under DTK, deceptions are spread among the normal systems in a network in such a way that unused services on those systems are consumed with deceptions. This increases the likelihood of an intelligence probe encountering a deception rather than a vulnerability"

- Fred Cohen

Brief History of Deception

@mcdwayne

31 of 55

First Commercial Honeypots

1998

"These hackers aren't kids on a digital joyride, ... It's clear their motive is financial gain."

- Alfred Huger,

Creator of CyberCop Sting

Brief History of Deception

@mcdwayne

32 of 55

"Honeytokens" is coined

1998

"I was developing an idea that I call 'honeytokens'... Basically, information that shouldn't be flowing over the network and, if you can detect it, something wrong is happening."

- Augusto Paes de Barros

Brief History of Deception

@mcdwayne

33 of 55

Canarytokens

2015

"Added aws token

Added svn + smtp tokens to generate"

- nickrohrbs, Thinkst developer - git commit message upon adding aws tokens to the code in 2016.

Brief History of Deception

@mcdwayne

34 of 55

Brief History of Deception

Honeytokens Becomes Default 2nd Line Defence at Google

2023

"Honeytokens are your early warning signs"

- Kevin Mandia from Mandiant /Google Cloud - The state of Cybersecurity - Year in Review talk at RSA 2023

@mcdwayne

35 of 55

What Is A Honeytoken?

@mcdwayne

36 of 55

Definition

—

Honeytokens are decoy credentials that do not allow any access to any resources or data. Instead they trigger alerts that reveal the IP address of the user who attempted to use the honeytoken.��Honeytokens look identical to real credentials to an attacker.

@mcdwayne

37 of 55

https://github.com/GitGuardian/ggcanary

@mcdwayne

38 of 55

Honeytoken Options

vs

@mcdwayne

39 of 55

Open Source - The DIY Route

GitGuardian/ggcanary - Requires Terraform and AWS
spacesiren/spacesiren - Requires AWS how
thinkst/canarytokens - Requires Docker experience
Complete DIY - see previous diagram - requires�Lambda knowledge and time to tinker with it

@mcdwayne

40 of 55

Commercial Options - Off The Shelf

Canarytoken.org - free, one off honeytokens
Thinkst - Canary.tools - The paid version of Canarytoken.org
GitGuardian Honeytoken Module - Requires GitGuardian account - in Beta
Microsoft Sentinel - Azure specific
CrowdStrike - Requires CrowdStrike

@mcdwayne

41 of 55

Honeytoken Best Practices

@mcdwayne

42 of 55

Honeytoken Best Practices�Do:

Put honeytokens in your private environments

Since they don't go to anything, there is no legit reason someone would attempt to use one

Code, CI environments, Jira, Slack, Vault

@mcdwayne

43 of 55

Honeytoken Best Practices�Do:

Use a 1:1 ratio of honeytokens to repo/environment

Keep it simple. When an alarm goes off, make it easy to tell exactly where, and only where, that honeytoken was embedded.

@mcdwayne

44 of 55

Honeytoken Best Practices�Do:

Use automation to scale deployment

One off honeytokens have value, but blue teams should be worried about defense at scale

Bash or Python scripting should be all you need
An example https://github.com/mcdwayne/honeytoken-putter

@mcdwayne

45 of 55

Honeytoken Best Practices�Do:

Think in terms of 'Blue Team'

Use the IP, UserAgent and other data points to block access
The goal is not to track down the individual attacker, it is to guard your stuff
If you think other credentials are at risk, time to rotate them

@mcdwayne

46 of 55

Honeytoken Best Practices�Do:

Remember this is a journey, not a one off exercise

Start with one repo. Worry about scale and automation once you understand and are comfortable with this, or any tech.

@mcdwayne

47 of 55

Honeytoken Best Practices�DO NOT:

List honeytokens in public

AWS, GitHub, GitLab, GitGuardian and many other public scanners are always on the lookout for public keys and will trigger them by scanning them

@mcdwayne

48 of 55

Honeytoken Best Practices�DO NOT:

Go hunting the attacker…unless you are LEA

Attacking an attacker feels good but it falls into illegal activity rather quickly as you start monitoring the IP and digging in.

See earlier slide about Blue Team usage

@mcdwayne

49 of 55

In Conclusion

@mcdwayne

50 of 55

@mcdwayne

51 of 55

Definition

—

Honeytokens are decoy credentials that do not allow any access to any resources or data. Instead they trigger alerts that reveal the IP address of the user who attempted to use the honeytoken.��Honeytokens look identical to real credentials to an attacker.

@mcdwayne

52 of 55

Honeytoken Options

vs

@mcdwayne

53 of 55

Hi, I’m Dwayne

Dwayne McDaniel

I live in Chicago
I’ve been a Developer Advocate since 2016
On Twitter @mcdwayne
mcdwayne@mastodon.social
Happy to chat about anything, hit me up
Outside of tech, I love improv, karaoke and going to rock and roll shows!

@mcdwayne

54 of 55

Do you know where your secrets are? Exploring the problem of secret sprawl and secret management maturity

@mcdwayne

55 of 55

A Few Incidents

Samsung

Reported: 7 March 2022 and 2 September 2022
160GB of data stolen by Lapsus$ hacking group and published in March, Including Galaxy source code containing over 6,000 secrets (API keys, passwords, credentials)
From July to August customer data was stolen
No reporting of how many individuals were impacted, nor details on how the threat actors gained access

@mcdwayne