Do you know where your secrets are?

Exploring the problem of secret sprawl and secret management maturity

​

@mcdwayne

Hi, I’m Dwayne

Dwayne McDaniel

​

​

• I live in Chicago
• Co Host - The Security Repo Podcast
• I’ve been a Developer Advocate since 2016
• On Twitter @mcdwayne
• mcdwayne@mastodon.social
• Happy to chat about anything, hit me up
• Outside of tech, I love improv, karaoke and going to rock and roll shows!

@mcdwayne

About GitGuardian

GitGuardian is the code security platform for the DevOps generation.

—

We help enterprises answer the questions "Where are my hardcoded secrets and have they been leaked?"

@mcdwayne

Don't Let Your Credentials Leak

@mcdwayne

The End

@mcdwayne

“I must not fear. Fear is the mind-killer. Fear is the little-death that brings total obliteration. I will face my fear. I will permit it to pass over me and through me. And when it has gone past I will turn the inner eye to see its path. Where the fear has gone there will be nothing. Only I will remain.”

​

​

― Frank Herbert, Dune

Bene Gesserit Litany Against Fear

@mcdwayne

Uber Breach – September 2022

​

2FA/MFA spamming

VPN Access

Scanning

Uber’s Infra

PowerShell scripts in network share

PowerShell script contained hardcoded credentials

Credential access

Discovery

Initial access

Privilege escalation

Admin access to

Thycotic PAM

Access to internal apps

AWS, GCP, Google Drive, Slack, SentinelOne, HackerOne admin console, Internal employee dashboards…

Uber Internal Network

Lateral movement

​

​

​

​

​

One exposed secret leads to many others!

Some mistakes are more expensive than others. One hardcoded secret giving access to Uber’s PAM solution led to a an organization wide IT takeover…

@mcdwayne

CircleCI – January 2023

Development System

CircleCI Platform

Embedded Malware Used To Steal Secrets

Remote Engineer

Hardcoded AWS, GCP, Google Drive, Internal employee dashboards, unencrypted customer data …

​

​

​

​

​

Customer secrets stolen, unauthorised access to GitHub repos and third-party systems

CircleCI Internal Network

Use of a honeytoken alerted the security team of the breach

Attacker

Customer Application

Hardcoded AWS, GCP, Google Drive, Internal employee dashboards, unencrypted customer data …

Customer Application

Hardcoded AWS, GCP, Google Drive, Internal employee dashboards, unencrypted customer data …

Customer Application

GitHub

@mcdwayne

Toyota – October 2022

Toyota

Subcontractor

Public GitHub Repo

​

​

​

​

​

Private T-Connect repo, that included a key to production data server, cloned and pushed to a public GitHub repo. This exposed data of 296,000 customers for 5 years.

Found by a third party security researcher

Publicly Accessible

Private GitHub

Repo

Customer Data Server

Data Server Key

Data Server Key

@mcdwayne

AstraZeneca – November 2022

Developer 1

Test Env

​

​

​

Test environment credentials pushed to GitHub. Another user pushed real patient data to the test environment.

Credentials were exposed for over a year before reported. Number of patients affected was never disclosed.

Unknown number of affected end users

Publicly Accessible

Public

GitHub

Repo

Developer 2

Test Env Credentials

Patient Data

@mcdwayne

@mcdwayne

​

​

"These hackers aren't kids on a digital joyride, ... It's clear their motive is financial gain."

- Alfred Huger

​

Creator of CyberCop Sting

First Commercial Honeypot

​

@mcdwayne

What Attackers Want:

​

1. Machine Resources
2. Access To Data
3. Anything That Leads To 1 or 2

@mcdwayne

Credential Leakage Is A Growing Problem

@mcdwayne

What are secrets in software development?

Definition

—

Secrets authenticate access and encryption of software components, such as: �

• API keys
• Username/password pairs
• Database connection URLs
• Browser session tokens
• Certificates

​

Sensitive files such as .env, .pem or .crt are also secrets themselves

@mcdwayne

@mcdwayne

https://www.gitguardian.com/state-of-secrets-sprawl-report-2023

@mcdwayne

Demo

@mcdwayne

@mcdwayne

@mcdwayne

https://docs.gitguardian.com/secrets-detection/detectors/supported_credentials

@mcdwayne

@mcdwayne

@mcdwayne

What Does This Mean For The Enterprise?

@mcdwayne

https://blog.gitguardian.com/voice-of-practitioners-the-state-of-secrets-in-appsec/

@mcdwayne

@mcdwayne

What Can I Do To Prepare Future Leaks?

@mcdwayne

"We recommend that you look for any case where you use url-encoded basic access authentication. If the credentials are in fact sensitive, you should immediately change them and redesign your app to avoid including them. " – Google Help

@mcdwayne

Stop Hardcoding Secrets�� Use secrets managers like �Hashicorp Vault or Azure Key Vault

�

@mcdwayne

Three pillars of a secrets management program

Train devs to use the tools and follow the processes

​

​

People

Tools

​

​

Automate detection and remediation

Use vaults, secrets managers and HMS to manage secrets

Processes

Document the steps to follow for incident remediation by devs

Create clear processes for provisioning, manaaging, and rotating secrets

Raise awareness around secrets sprawl and secure coding practices

@mcdwayne

A pragmatic approach to

secrets management in the SDLC

No processes or tools for managing secrets – secrets sprawl in the SDLC.

No detection (and remediation) in place.

Secrets are unencrypted at rest and shared across teams. Scanning for secrets is triggered manually at times,

Secrets are scoped, stored in a vault and shared using a secrets manager. Automated detection on shared repositories and final artifacts is continuous.

Secrets are checked encrypted into repositories with decryption keys stored in a secure vault.

Secrets scanning and rotation are performed periodically.

Secrets are scoped, stored and called from a vault

Detection is preventive and integrated into dev workflows.

@mcdwayne

Level 0 – Uninitiated

• Secrets are shared in clear text through private channels and stored unencrypted in local config files

Secrets management

• Secrets are unencrypted and checked into private repositories.

• No detection in place

• No detection in place

• Secrets are embedded in artifacts
• VCS and 3rd-party (e.g. code quality tools) access tokens are hardcoded in build scripts

• No detection in place

• Secrets are embedded in scripts
• Sensitive variables are printed in server logs

• No detection in place

Developer environments

Secrets detection

Source Control

(Source code

& Infra-as-Code)

CI/CD pipelines &

software artifacts

Runtime environments

High exposure risk

@mcdwayne

Level 1 – Beginners

• Source code and IaC templates are scanned for secrets periodically
• High-severity incidents are remediated with limited help from developers

• Secrets are unencrypted in config files but can be encrypted before sharing with other devs

Secrets management

• Secrets are grouped in config files and accessed using environment variables
• IaC secrets are stored externally – by the cloud services provider (e.g. AWS, GCP)

• No detection in place

• Build outputs (e.g. Docker images) are scanned for secrets manually before a release

• Final artifacts do not contain any secrets
• Pipeline secrets are stored in the build system

• Secrets are grouped in a common config file
• Sensitive variables are redacted from logs
• Secrets are not scoped by environment

• Secrets are rotated manually in case of exposure or compromise

Secrets detection

Developer environments

Source Control

(Source code

& Infra-as-Code)

CI/CD pipelines &

software artifacts

Runtime environments

High exposure risk

@mcdwayne

Level 2 – Intermediate

• Secrets are stored in a vault and shared through a secret manager
• Developer environment secrets are correctly scoped

Secrets management

• Secrets are encrypted and checked into repositories (master key is stored externally)

• Critical repositories are continuously scanned at the pull/merge requests stage
• Devs contribute to remediation but it is still not a well established and documented process

• Scanning is triggered manually at times

• Secrets are stored in the build process
• Secrets are scoped; permissions abide by the principle of least privilege

- Build outputs (e.g. Docker images) are scanned for secrets manually before a release

• Secrets (or master decryption key) are stored in a vault and dynamically loaded with a secrets manager with minimal access controls

• Secrets are rotated manually in case of exposure or compromise

Secrets detection

Developer environments

Source Control

(Source code

& Infra-as-Code)

CI/CD pipelines &

software artifacts

Runtime environments

Moderate exposure risk

@mcdwayne

Level 3 – Advanced

• Informative scanning is enforced for all branches (feature, hotfix, etc.) in CI pipelines

• Secrets are stored in a vault and shared exclusively through a secret manager
• Secrets rotation policy is well defined

Secrets management

• Secrets are no longer embedded in the current source code revision
• Secrets rotation policy is well defined

• All repositories are continuously scanned for hardcoded credentials
• Collaboration on incident remediation is mandatory for all developers

• Scanning before pushing code (pre-commit / pre-push) is adopted by security champions
• Developers are systematically involved in the remediation process

• Pipeline secrets are stored in a vault and loaded using a secret manager
• Secrets are scoped; permissions follow the principle of least privilege

• Secrets are stored in a vault and dynamically loaded from a secrets manager
• Secrets are scoped and access is monitored/logged

• Secrets are scheduled for regular rotation

Secrets detection

Developer environments

Source Control

(Source code

& Infra-as-Code)

CI/CD pipelines &

software artifacts

Runtime environments

Low exposure risk

@mcdwayne

Level 4 – Experts

• No presence of valid hardcoded secrets in past or current revisions of source code

• Scanning before pushing code (pre-commit / pre-push) is adopted by all developers
• Developers are systematically involved in the remediation process

• Secrets are stored in a vault with access controls and logging
• Dynamic secrets with limited scope are used for development when possible

Secrets management

• All repositories are continuously monitored and blocking scans (pre-receive) are setup
• Remediation workflows are automated and fixing is handled by developers

• Pipeline secrets are short-lived, scoped, and stored in an external vault
• If possible, build secrets are replaced by OpenID Connect (OIDC) tokens for auth.

• Blocking scanning is enforced for all branches (feature, hotfix, etc.) in CI pipelines

• Secrets are stored in a vault and loaded dynamically from a secrets manager
• Restrictive access controls and logging are enforced

• Secrets are scheduled for regularly automated rotation

Secrets detection

Developer environments

Source Control

(Source code

& Infra-as-Code)

CI/CD pipelines &

software artifacts

Runtime environments

Limited exposure risk

@mcdwayne

In Conclusion

@mcdwayne

@mcdwayne

A pragmatic approach to

secrets management in the SDLC

No processes or tools for managing secrets – secrets sprawl in the SDLC.

No detection (and remediation) in place.

Secrets are unencrypted at rest and shared across teams. Scanning for secrets is triggered manually at times,

Secrets are scoped, stored in a vault and shared using a secrets manager. Automated detection on shared repositories and final artifacts is continuous.

Secrets are checked encrypted into repositories with decryption keys stored in a secure vault.

Secrets scanning and rotation are performed periodically.

Secrets are scoped, stored and called from a vault

Detection is preventive and integrated into dev workflows.

@mcdwayne

Hi, I’m Dwayne

Dwayne McDaniel

​

​

• I live in Chicago
• Co Host - The Security Repo Podcast
• I’ve been a Developer Advocate since 2016
• On Twitter @mcdwayne
• mcdwayne@mastodon.social
• Happy to chat about anything, hit me up
• Outside of tech, I love improv, karaoke and going to rock and roll shows!

@mcdwayne

Do you know where your secrets are?

Exploring the problem of secret sprawl and secret management maturity

​

@mcdwayne