Security in DevOps-Strategies

Timo Pagel

devopssec@timo-pagel.de

​

The OWASP Foundation

OWASP

http://www.owasp.org

OWASP

Timo Pagel

Agenda

• Introduction
• Generic DevOps Security Maturity Model
• Conclusion

2

Timo Pagel

OWASP

Timo Pagel

Agenda

• Introduction
• Generic DevOps Security Maturity Model
• Conclusion

​

3

Timo Pagel

OWASP

Timo Pagel

About Me

• DevOps Security Consultant
• Lecturer for Security in Web Applications at University of Applied Sciences Kiel
• Open Source Enthusiast

​

• Hiring Software Developers

​

Timo Pagel

OWASP

Timo Pagel

5

OWASP

Timo Pagel

6

Wall of Irritation

OWASP

Timo Pagel

7

OWASP

Timo Pagel

8

OWASP

Timo Pagel

9

OWASP

Timo Pagel

10

What about quality when deploy automatically multiple times a day?

OWASP

Timo Pagel

11

OWASP

Timo Pagel

Problem Statement

• How to enhance information security?
• In DevOps-Strategies
• Through DevOps-Strategies
• How to prioritize?

12

Information Security

Timo Pagel

OWASP

Timo Pagel

DevOps-Dimensions

• Build and Deployment
• Culture and Organisation
• Information Gathering
• Infrastructure
• Test and Verification

13

Timo Pagel

OWASP

Timo Pagel

Agenda

• Introduction
• Generic DevOps Security Maturity Model
• Conclusion

14

Timo Pagel

OWASP

Timo Pagel

Generic DevOps Security Maturity Model

http://gdosmm.timo-pagel.de

• Build and Deployment
• Culture and Organisation
• Information Gathering
• Logging & Monitoring
• Infrastructure
• Test and Verification
• Dynamic depth
• Static depth
• Consolidation
• ...

​

15

Timo Pagel

OWASP

Timo Pagel

Levels

• Level 1: Basic understanding of security practices
• Level 2: Understanding of security practices
• Level 3: High understanding of security practices
• Level 4: Advanced understanding of security practices at scale

Timo Pagel

OWASP

Timo Pagel

Identification of the Implementation Maturity

17

Timo Pagel

OWASP

Timo Pagel

Implementation Point

• Risk
• Countermeasure
• Outcome for Security / Hartness of Implementation
• Dependencies
• Implementation Hints

18

Timo Pagel

OWASP

Timo Pagel

I am using Docker, now I am safe

Timo Pagel

OWASP

Timo Pagel

Infrastructure

• Virtual Environments
• KVM/Docker: No network restrictions

​

20

Virtual Environment

Virtual Environment

Virtual Environment

Internet

Firewall

Physical Server

Timo Pagel

OWASP

Timo Pagel

Infrastructure

• Virtual Environments
• KVM/Docker: No network restrictions

​

21

Virtual Environment

Virtual Environment

Virtual Environment

Internet

Firewall

Physical Server

Timo Pagel

OWASP

Timo Pagel

Infrastructure

• Virtual Environments
• KVM/Docker: No network restrictions

​

22

Virtual Environment

Virtual Environment

Virtual Environment

Internet

Firewall

Physical Server

Not a new problem

• More containers -> higher risk

Timo Pagel

OWASP

Timo Pagel

Controlled networks in virtual environments

OWASP

Timo Pagel

I hardened my production environment, now I am safe

Timo Pagel

OWASP

Timo Pagel

Build and Deployment Process

25

Developer

Version

Control

Build and Deployment

Production System

Internal Repository

Production near System

Libraries from external repositories

Timo Pagel

OWASP

Timo Pagel

Build and Deployment Process

26

Developer

Version

Control

Build and Deployment

Production System

Internal Repository

Production near System

Libraries from external repositories

Timo Pagel

OWASP

Timo Pagel

Build and Deployment Process

27

Developer

Version

Control

Build and Deployment

Production System

Internal Repository

Production near System

Libraries from external repositories

Timo Pagel

OWASP

Timo Pagel

Build and Deployment Process

28

Developer

Version

Control

Build and Deployment

Production System

Internal Repository

Production near System

Libraries from external repositories

Timo Pagel

OWASP

Timo Pagel

Build and Deployment Process

29

Developer

Version

Control

Build and Deployment

Production System

Internal Repository

Production near System

Libraries from external repositories

Accounting

Secretary

….

Timo Pagel

OWASP

Timo Pagel

Build and Deployment Process

30

Developer

Version

Control

Build and Deployment

Production System

Internal Repository

Production near System

Libraries from external repositories

Accounting

Secretary

….

Attacker

Timo Pagel

OWASP

Timo Pagel

Build and Deployment Process

31

Developer

Version

Control

Build and Deployment

Production System

Internal Repository

Production near System

Libraries from external repositories

Accounting

Secretary

….

Attacker

Timo Pagel

OWASP

Timo Pagel

CVE-2017-1000117

Source: https://about.gitlab.com/2017/08/10/gitlab-9-dot-4-dot-4-released/

Timo Pagel

OWASP

Timo Pagel

Simple access control for systems

Applications run in virtual environments

Test of server side application comp.with known vuln.

​

OWASP

Timo Pagel

Build and Deployment Process

34

Developer

Version

Control

Build and Deployment

Production System

Internal Repository

Production near System

Libraries from external repositories

Accounting

Secretary

….

Attacker

Timo Pagel

OWASP

Timo Pagel

Build and Deployment Process

35

Developer

Version

Control

Build and Deployment

Production System

Internal Repository

Production near System

Libraries from external repositories

Accounting

Secretary

….

Attacker

Timo Pagel

OWASP

Timo Pagel

Build and Deployment Process

36

Developer

Version

Control

Build and Deployment

Production System

Internal Repository

Production near System

Libraries from external repositories

Accounting

Secretary

….

Mostly credentials.xml can be read!

Attacker

Timo Pagel

OWASP

Timo Pagel

Demo

Developer

Version

Control

Build and Deployment

Production System

Internal Repository

Timo Pagel

OWASP

Timo Pagel

Simple access control for systems

Applications run in virtual environments

Checking the sources of used libraries

OWASP

Timo Pagel

Build and Deployment Process

39

Developer

Version

Control

Build and Deployment

Production System

Internal Repository

Production near System

Libraries from external repositories

Timo Pagel

OWASP

Timo Pagel

Build and Deployment Process

40

Developer

Version

Control

Build and Deployment

Production System

Internal Repository

Production near System

Libraries from external repositories

Timo Pagel

OWASP

Timo Pagel

Build and Deployment Process

41

Developer

Version

Control

Build and Deployment

Production System

Internal Repository

Production near System

Libraries from external repositories

Timo Pagel

OWASP

Timo Pagel

Building and testing of artifacts in virtual environments

Checking the sources of used libraries

Defence metrics

OWASP

Timo Pagel

Build and Deployment Process

43

Developer

Version

Control

Build and Deployment

Production System

Internal Repository

Production near System

Libraries from external repositories

Accounting

Secretary

….

Attacker

Timo Pagel

OWASP

Timo Pagel

Build and Deployment Process

44

Developer

Version

Control

Build and Deployment

Production System

Internal Repository

Production near System

Libraries from external repositories

Internet

Timo Pagel

OWASP

Timo Pagel

Private Registry, Public Access

Public Registry, Attack in July 2016

Source: http://blog.aquasec.com/vines-docker-registry-hack-a-bad-case-of-rtfm

Timo Pagel

OWASP

Timo Pagel

Simple access control for systems

OWASP

Timo Pagel

47

• Integration Tests: HttpUnit
• Static Analysis
• FindSecurityBugs
• OWASP Dependency Check

Developer

Version

Control

Build and Deployment

Production System

​

Internal Repository

Production near System

​

Static Test-System

OWASP

Timo Pagel

48

• Integration Tests: HttpUnit
• Static Analysis
• FindSecurityBugs
• OWASP Dependency Check

• Active Tests / Vulnerability Scans
• Arachni
• OWASP Zap

Developer

Version

Control

Build and Deployment

Production System

​

Internal Repository

Production near System

​

Static Test-System

OWASP

Timo Pagel

49

• Integration Tests: HttpUnit
• Static Analysis
• FindSecurityBugs
• OWASP Dependency Check

• Passive Vulnerability Scans:

OWASP Zap

• Passive Infrastructure Tests:

SSLyze

• Active Tests / Vulnerability Scans
• Arachni
• OWASP Zap

Developer

Version

Control

Build and Deployment

Production System

​

Internal Repository

Production near System

​

Static Test-System

OWASP

Timo Pagel

Detection of Components with Known Vulnerabilities

Through 2020, 99% of vulnerabilities exploited will continue to be ones known by security and IT professionals for at least one year.

​

Gartner, 2016

Source: Gartner’s Top 10 Security Predictions 2016

Timo Pagel

OWASP

Timo Pagel

Equifax Incident September 2017

Impacting approximately 143 million consumers

​

Reason: Using library with known vulnerabilities

(Apache Struts 2 vulnerability CVE-2017-5638)

Timo Pagel

OWASP

Timo Pagel

Detection of Components with Known Vulnerabilities

• What to scan?
• How to scan?
• When to scan?
• Who to report to?
• Which actions to take?

Timo Pagel

OWASP

Timo Pagel

Detection of Components with Known Vulnerabilities: What/How?

• Applications
• Operating System Environments
• Infrastructure/Firmware

​

Timo Pagel

OWASP

Timo Pagel

Detection of Components with Known Vulnerabilities: What/How?

• Applications
• Java/Ruby: OWASP Dependency Check
• PHP: Sensio Lab Security Advisor (composer)
• Javascript: retire.js
• Operating System Environments
• Root-Server / Virtual Machine
• Vulns, OpenVAS
• Docker
• anchore.io, clair, openScap
• Infrastructure/Firmware
• ?

Timo Pagel

OWASP

Timo Pagel

Scan During Build-Process

Build Application

Build Image

Scan Image

Deploy-

ment

Build Trigger

Scan Application

Timo Pagel

OWASP

Timo Pagel

Scan During Build-Process

• Fast Feedback on Changes

​

Build Application

Build Image

Scan Image

Deploy-

ment

Build Trigger

Scan Application

Timo Pagel

OWASP

Timo Pagel

Vulnerability Lifecycle

t

Build

Vulnerability Discovered

Patch Published

Start Container

Build

Run Container

Timo Pagel

OWASP

Timo Pagel

Source: https://security-tracker.debian.org/tracker/CVE-2017-8804

Timo Pagel

OWASP

Timo Pagel

When to report? Quality Gates

• Criticality >= High
• Risk Avoidance
• Upgrade
• Offline
• Risk Limitation
• Web Application Firewall
• Risk Acceptance
• Unstable Build
• Risk Transfer
• Get an Insurance

​

​

Timo Pagel

OWASP

Timo Pagel

Reporting

• Build Failed/Unstable
• E-Mail
• Monitoring
• Jira

Timo Pagel

OWASP

Timo Pagel

Test of infrastructure comp. with known vuln.

Treatment of defects with classification critical

Test of server side application comp. with known vuln.

​

OWASP

Timo Pagel

Dynamic Scanning

• Tools:
• Arachni, OWASP Zap, sqlmap, …

​

​

• Orchestration
• secureCodeBox, BDD-Security, gauntlt

Timo Pagel

OWASP

Timo Pagel

Integration into Jenkinsfile

[...]

httpRequest

httpMode: 'POST',

contentType: 'APPLICATION_JSON',

outputFile: 'scan_process.json',

requestBody: scanConfig,

Url:

'http://192.168.122.1:8080/engine-rest/process-definition/key/scanJobNmapDemoExternalBodgeIT/start'

[...]

OWASP

Timo Pagel

Simple Scan

Coverage of client side dynamic components

OWASP

Timo Pagel

Agenda

• Introduction
• Generic DevOps Security Maturity Model
• Conclusion

65

Timo Pagel

OWASP

Timo Pagel

Conclusion

• DevOps-Strategies must integrate security:

Hardening

• DevOps-Strategies offer security benefits:

Automated testing before release

• Prioritization:

Generic DevOps Security Maturity Model

Timo Pagel

OWASP

Timo Pagel

Building and testing of artifacts in virtual environments

Simple access control for systems

Controlled networks in virtual environments

Checking the sources of used libraries

Test of infrastructure comp. with known vuln.

Defence metrics

Treatment of defects with classification critical

Test of server side application comp.with known vuln.

​

Simple Scan

Coverage of client side dynamic components

Applications run in virtual environments

OWASP

Timo Pagel

Building and testing of artifacts in virtual environments

Simple access control for systems

Controlled networks in virtual environments

Checking the sources of used libraries

Test of infrastructure comp. with known vuln.

Defence metrics

Treatment of defects with classification critical

Questions?

Test of server side application comp.with known vuln.

​

Coverage of client side dynamic components

Simple Scan

Applications run in virtual environments

http://gdosmm.timo-pagel.de

Applications run in virtual environments

OWASP

Timo Pagel

Backup-Slides

Timo Pagel

OWASP

Timo Pagel

Comparison of Scanners (August 2017)

Criteria

• Coverage
• Scanning
• Automation
• Report

Tools

• Arachni
• OWASP Zap

Timo Pagel

OWASP

Timo Pagel

Comparison of Scanners (August 2017)

Timo Pagel

OWASP

Timo Pagel

Comparison of Scanners (August 2017)

Timo Pagel

OWASP

Timo Pagel

Comparison of Scanners (August 2017)

Timo Pagel

OWASP

Timo Pagel

Comparison of Scanners (August 2017)

Timo Pagel

OWASP

Timo Pagel

Remember Version: Scan Nightly

Build Application

Build Image

Remember Image:Tag

Deploy-

ment

Fetch Image/Tag

Scan Image

Build Trigger

Nightly Trigger

Build of Application

Scanning

Image:Tag

Timo Pagel

OWASP

Timo Pagel

Remember Version: Scan Nightly

Build Application

Build Image

Remember Image/Tag

Deploy-

ment

Fetch Image/Tag

Scan Image

Build Trigger

Nightly Trigger

Build of Application

Scanning

Build Application

Build Image

Remember Image/Tag

Deploy-

ment

Build Application

Build Image

Remember Image/Tag

Deploy-

ment

Image:Tag

Fetch Image/Tag

Scan Image

Fetch Image/Tag

Scan Image

Timo Pagel

OWASP

Timo Pagel

Remember Version: Scan Nightly

Fetch Image/Tag

More?

Yes

Scan Image

Image:Tag

Nightly Trigger

Build Application

Build Image

Remember Image/Tag

Deploy-

ment

Build Trigger

Build Application

Build Image

Remember Image/Tag

Deploy-

ment

Build Application

Build Image

Remember Image/Tag

Deploy-

ment

Timo Pagel

OWASP

Timo Pagel

Remember Version: Cluster

​

Container

Container

Container

Node 1

Container

Container

Container

Node 2

Container

Container

Container

Node X

Timo Pagel

OWASP

Timo Pagel

Remember Version: Cluster

​

App1

ImageA:Z

App2

ImageB:Z

App3

ImageC:Z

Node 1

App1

ImageA:Z

App2

ImageB:Z

App3

ImageC:Z

Node 2

App1

ImageA:Z

App2

ImageB:Z

App3

ImageC:Y

Node X

Timo Pagel

OWASP

Timo Pagel

Remember Version: Cluster

Blue / Green Deployment

App1

ImageA:Z

App2

ImageB:Z

App3

ImageC:Z

Node 1

App1

ImageA:Z

App2

ImageB:Z

App3

ImageC:Z

Node 2

App1

ImageA:Z

App2

ImageB:Z

App3

ImageC:Y

Node X

Timo Pagel

OWASP

Timo Pagel

Remember Version: Cluster

​

App1

ImageA:Z

App2

ImageB:Z

App3

ImageC:Z

Node 1

App1

ImageA:Z

App2

ImageB:Z

App3

ImageC:Z

Node 2

App1

ImageA:Z

App2

ImageB:Z

App3

ImageC:Y

Node X

Master

Timo Pagel

OWASP

Timo Pagel

Fetch Running Containers/Images

Fetch all

running

Get Image

oc get pod --all-namespaces -o wide

oc describe pod <pod>

More?

Yes

docker ps

docker inspect <container>

Scan Image

Nightly Trigger

Timo Pagel

OWASP

Timo Pagel

Combination: During Build + Get Running

Fetch all

running

Get Image

More?

Yes

Scan Image

Nightly Trigger

Build Application

Build Image

Scan Image

Deploy-

ment

Build Trigger

Team/Build

Image:Tag

Remember�Team/Img.

Report

Timo Pagel

OWASP

Timo Pagel

Combination: During Build + Get Running

Fetch all

running

Get Image

More?

Yes

Scan Image

Nightly Trigger

Build Application

Build Image

Scan Image

Deploy-

ment

Build Trigger

Team/Build

Image:Tag

Remember�Team/Img.

Report

docker run smartapp:1.0 --label team="xyz"

Timo Pagel

OWASP

Timo Pagel

Fetch Vulnerable

Trigger Vulnerable Projects

Fetch all

running

Scan Images

Nightly Trigger

Remember Vulnerable

Vulnerable Image

Vulnerable?

Unstable/

Fail Build

Yes

Build

Scan

Timo Pagel

OWASP

Timo Pagel

Fetch Vulnerable

Trigger Vulnerable Projects

Fetch all

running

Scan Images

Nightly Trigger

Remember Vulnerable

Vulnerable Image

Vulnerable?

Unstable/

Fail Build

Yes

Build

Scan

Relation Image<>Job Name needed to trigger

Timo Pagel

OWASP

Timo Pagel