1 of 53

2 of 53

Supply Chain Security and the Real World:

Lessons from Incidents

​

Adrian Mouat

Technical Community Advocate

Chainguard

3 of 53

  • picture of stats, CVE reports

​

3

4 of 53

4

5 of 53

5

6 of 53

6

7 of 53

7

8 of 53

8

9 of 53

9

10 of 53

10

11 of 53

11

12 of 53

12

13 of 53

13

14 of 53

Adrian Mouat

  • DevRel Engineer at Chainguard
  • Wrote "Using Docker" for O'Reilly

​

14

15 of 53

Incident 1

CodeCov

June 2021

​

15

16 of 53

17 of 53

17

18 of 53

NEVER do this!

FROM debian

COPY mysecret ./

RUN curl -H 'Authorization: Bearer \

$(cat mysecret)' https://api.example.com

RUN rm mysecret

18

19 of 53

Instead

FROM debian

RUN --mount=type=secret,id=mysec \

curl -H 'Authorization: Bearer \

$(cat /run/secrets/mysec)' \

https://api.example.com

​

$ docker build --secret id=mysec,src=./mysecret .

19

20 of 53

20

21 of 53

bash <(curl -s https://codecov.io/bash)

21

22 of 53

Not totally awful...

  • Uses https
    • pretty sure it is from codecov
    • unlikely mitm
  • No sudo
  • But really want checksum and/or GPG check
  • What if the transfer is interrupted?

22

23 of 53

What to do with these

  • Split download and exec steps
  • Take a look at code
  • Add in checksum for automation
    • (or GPG if available)

​

23

24 of 53

Using Checksums

ENV REDIS_DOWNLOAD_URL http://download…redis-7.4.2.tar.gz

ENV REDIS_DOWNLOAD_SHA \

4ddebb…9767dd89dbe712d2b68e808af6a1f

RUN wget -O redis.tar.gz "$REDIS_DOWNLOAD_URL"

RUN echo "$REDIS_DOWNLOAD_SHA *redis.tar.gz" \

| sha256sum -c -

24

25 of 53

Using GPG

Redis Dockerfile good example

​

wget -O /usr/local/bin/gosu.asc "$url.asc"

wget -O /usr/local/bin/gosu "$url"

gpg --batch --keyserver hkps://keys.openpgp.org \

--recv-keys B42F6819007F00F88E364FD4036A9C25BF357DD4

gpg --batch --verify /usr/local/bin/gosu.asc \

/usr/local/bin/gosu

25

26 of 53

For Distributing Software

  • Put it on a Package Manager
    • e.g. homebrew tap
  • Provide checksums and signatures
    • typically on releases page

26

27 of 53

27

28 of 53

28

29 of 53

Secrets and Environment Variables

​

​

  • Generally not a great idea
  • Too easily shared
  • Accidentally printed in logs

​

29

30 of 53

Secrets and Environment Variables

​

​

  • Prefer
    • Identity Federation
    • Using a Secret Manager
    • Temporary files
      • secret volumes in k8s

​

30

31 of 53

Incident 2

ChangedFiles

Mar 2025

​

31

32 of 53

tj-actions/changed-files

  • GitHub Action modified to leak secrets in logs
  • All users of latest version affected
  • How were they compromised?

32

​

​

33 of 53

Like all the best stories, it has a (review)dog

33

Photo by charlesdeluvio on Unsplash

34 of 53

reviewdog

  • Had automated invites for contributors!
  • A PAT for an account was compromised
  • Code committed that dumped secrets
  • Tags modified to point to malicious commit

34

35 of 53

35

36 of 53

36

37 of 53

Verify Contributors & Contributions

  • Hard to vet people
  • But automated workflows set you up as a target
    • probably A Bad Idea™
  • Enforce commit signing
  • Run checks for malware
    • malcontent

​

37

38 of 53

Restrict Tag Updates

  • Use rulesets to stop tags being updated
  • Also check your GitHub security configuration

38

39 of 53

Pin Actions to Digests

- uses: chainguard-dev/digestabot@v1.2.2

- uses: chainguard-dev/digestabot@432222…1c95ce2cedac # v1.2.2

  • Won't immediately use new version
  • But still not perfect
    • probably use dependabot to automatically update
    • will you really spot the change?
  • Also do this for container images

39

40 of 53

Avoid Long-Lived Credentials

  • Prefer identity federation
  • And short-lived tokens
  • And/or revoke credentials on completion
  • OIDC
  • Chainguard created OctoSTS to replace PATs with short-lived tokens
    • https://github.com/apps/octo-sts

40

41 of 53

Incident 3

Shai-Hulud

Sep 2025

​

41

42 of 53

Shai-Hulud

  • 500+ NPM Packages compromised
    • and GitHub repos
  • "Worm" behaviour
    • each compromise published further hacked packages
  • Forced GitHub to take action
      • stopped packages being pushed with IoC

42

43 of 53

Exfiltration

  • Made public copies of private repos
  • Used trufflehog to identify secrets
    • API keys and tokens
  • Sent secrets to webhook SaaS
    • on free tier so got rate limited!
    • still exposed in workflow log

​

43

44 of 53

44

45 of 53

Friends don't let friends publish from a laptop

45

46 of 53

Trusted Publishing

  • Don't have NPM tokens!
    • Or cargo, PyPI
  • OIDC based
    • short-lived and tightly scoped tokens
  • Works with GitHub, GitLab etc

46

47 of 53

Trusted Publishing

  • Configure via NPM/PyPI/Cargo etc
  • Example from PyPI

47

48 of 53

Trufflehog

  • Legitimate tool
  • Scans for secrets
  • Use it!
    • Or the attackers will

48

49 of 53

Multi Factor Authentication (MFA)

  • Do I really need to explain
  • Enforce on GitHub & NPM etc
  • Buy keys for your devs

49

50 of 53

Wrap Up

​

50

51 of 53

Recap

  • Be careful with secrets in container builds
  • Verify downloads
  • Avoid secrets in environment variables
  • Verify contributors and contributions
  • Pin actions (and images) to digests
  • Avoid long-lived credentials
  • Check GitHub Security Config
  • Enforce MFA

51

52 of 53

Tooling

52

53 of 53

"If all ransomware groups suddenly became cybersecurity specialists, and those who are now cybersecurity specialists began pentesting networks, the ransomware would end."

​

wazawaka

53