1 of 22

Continuous Status Monitoring of your Agile and DevOps Programs

Arlene F. Minkiewicz

— 1 —

2 of 22

Agenda

Introduction
What is Continuous Monitoring for DevOps?
Types of Continuous Monitoring
Benefits of Continuous Monitoring
Continuous Monitoring Tools for DevOps
DevOps Metrics from Continuous Monitoring
Wrap Up

— 2 —

3 of 22

Introduction

In Atlassian’s report DevOps Trends Survey over half of the respondents reported that:

Their organizations had dedicated DevOps teams

99% indicated that DevOps has had a positive impact on their organizations
Gartner report that 87% of business leaders believe digitation is an important priority

DevOps transformations require a dedicated, continuous learning process and successful implementation to reach maturity
Continuous monitoring is important part of the DevOps lifecycle that is frequently overlooked

— 3 —

4 of 22

Introduction

Continuous monitoring should be an important part of the success plan for organization’s pursuing DevOps
This Webinar covers…

The concepts of DevOps and Continuous Monitoring (CM)
Benefits organizations have realized through CM
Popular CM Tools
Common metrics that differentiate high performing DevOps organizations

— 4 —

5 of 22

What is DevOps?

DevOps Approach merges the skills of ‘development’ (Dev) with ‘Operations’ (Ops) to optimize software delivery efficiency and customer satisfaction
DevOps Teams, through common practices and principles, aided by certain software tools, develop a highly automated code distribution pipeline:

Enables the practice of methodically monitoring numerous development related areas
Extends from formulation of an operational plan through deployment and into operations
Incorporates several engineering best practices such as

Continuous process improvement
Better resource management
Cost optimization
Rapid delivery

— 5 —

6 of 22

What is Continuous Monitoring (CM)?

With agile, Development Teams are frequently delivering new content to end users.
Success with such a paradigm requires automation to provide continuous feedback to the team

Continuous Integration
Continuous Testing
Continuous Deployment
Continuous Monitoring

CM is key to the sustained health of an application
CM tools observe and evaluate system functionality and state and report to Development and QA teams so they can react quickly

— 6 —

7 of 22

What is Continuous Monitoring?

CM is the process of routinely and vigilantly scanning for performance degradations in:

Application Performance
Security
Compliance

The primary goals of CM are:

Enhance transparency and visibility of IT and network operation, especially focused on discovery and addressing potential security breaches
Monitor Software Operations, especially focused on identify and solving performance issues
Provide rapid feedback to the Development, Operations and QA teams to optimize the performance of applications and ensure a great customer experience

— 7 —

8 of 22

Three Types of Continuous Monitoring for DevOps

Infrastructure Monitoring – data centers, networks, hardware and software required to deliver applications and services
Application Monitoring – health and performance of applications released to production
Network Monitoring – network activity and other components such as servers, firewalls, routers, and switches, identify security threats

— 8 —

9 of 22

Benefits of Continuous Monitoring

Improved Security through automation
Early detection of performance errors improving response time
Reduced downtime – issues are detected and addressed quickly
Better business performance – improving user experience
Increased clarity throughout the team

— 9 —

10 of 22

Continuous Monitoring Tools for DevOps

A plethora of CM Tools for DevOps….

All types of monitoring
Various environments and platforms
Open source, licensed and pay per use models

Way too many to discuss them all
Multi-platform support (AWS, Azure, Kubernetes, Docker, etc.)
Descriptions of several popular ones follows….

— 10 —

11 of 22

Continuous Monitoring Tools for DevOps

AppDynamics

Application Monitoring
Cloud and on-premise
Enables optimization of applications by monitoring APIs, key business metrics and code level problems
Licensed with free lite version available

Dynatrace

Application, Infrastructure and Network Monitoring
Hybrid and cloud
Artificial Intelligence (AI) powered automation monitoring cloud based PaaS and container technologies
Various pricing schemes

— 11 —

12 of 22

Continuous Monitoring Tools for DevOps

Splunk

Application and Infrastructure Monitoring
Cloud, on-premise and hybrid monitoring
Automatic, continuous, real-time monitoring through application lifecycle
Various pricing models

Nagios

Network, Application, and Infrastructure monitoring
Cloud, web and storage service monitoring
Services easily find network outages, protocol errors and provide for monitoring for audit and compliance issues
Open source (Nagios Core) and paid version (Nagios XI)

— 12 —

13 of 22

DevOps Metrics from Continuous Monitoring

DevOps Research and Assessment (DORA)

Largest academic study exploring DevOps principals and applications
Purchased by Google in 2018
Employs data driven insights to promote DevOps best practices
Over seven years have identified four DevOps Success metrics thought to differentiate high performing DevOps teams with respect to software delivery performance

Deployment Frequency
Lead Time for Changes
Mean Time to Recovery
Change Failure Rate

— 13 —

14 of 22

DevOps Metrics from Continuous Monitoring

Deployment Frequency (measure of throughput)

Number of deployments in a given time frame
The higher the number the better – more value delivered more often
To measure deployment frequency…

Use pipeline tool to determine total builds (TOTAL_BUILD) and successful deployment to production (DEPLOY_BUILD)
Development Frequency = (DEPLOY_BUILD/TOTAL_BUILD * 100)

2022 State of DevOps Report from DORA

High performing teams deploy multiple times per day
Medium performing teams deploy between once a week and once a month
Low performing teams deploy between once a week and once every 6 months

— 14 —

15 of 22

DevOps Metrics from Continuous Monitoring

Lead Time for Changes (measure of throughput)

Time it takes for an idea to progress from implementation to production
Entire lifecycle including development, testing and delivery
To calculate Lead Time for Changes you need to identify start of work and its finish
This metric can be improved by shortening the deployment time through improved automation and test integration
The shorter the lead time the better
2022 State of DevOps Report from DORA

High performing teams have a mean time between one day and one week
Medium performing teams have a mean time between one week and one month
Low performing teams have a mean time of between one month and six months

— 15 —

16 of 22

DevOps Metrics from Continuous Monitoring

Mean Time to Recovery (measure of stability)

Time it takes an organization to restore a system when there has been a production failure
Mean Time to Recovery can be calculated by tracking average time between a defect report and a fix being deployed to production.

CM tools can be configured to enable this tracking - providing alerts about potential problems

The shorter the Mean Time to Recovery is the better
2022 State of DevOps Report from DORA

High performing team average less than one day
Medium performing teams average less between one day and one week
Low performing teams average between one week and one month

— 16 —

17 of 22

DevOps Metrics from Continuous Monitoring

Change Failure Rate (measure of stability)

Number of code changes that result in any sort of production failure
This can be calculated by dividing the number of deployment failures by the number of total deployments
Change Failure Rate can be decreased through robust monitoring and through strategies such as delivering in small increments and increased test automation
The lower the Change Failure Rate the better
2022 State of DevOps Report from DORA

High performing teams are between 0 – 15%
Medium performing teams are between 16-30%
Low performing teams are between 46-60%

— 17 —

18 of 22

Wrap-Up

Continuous Monitoring is an important part of the success plan for organization’s pursuing DevOps Successes
Continuous Monitoring is key to the sustained health of an application
Continuous Monitoring tools observe and evaluate system functionality and state enabling Development and QA teams to react quickly
Many software tools are available to implement Continuous Monitoring – teams should evaluate their monitoring goals to determine the best tool for their project(s)
For the last eight years, DORA has studied DevOps teams to identify what differentiates high performing teams – they have identified 4 key software delivery metrics….

Deployment Frequency
Lead Time for Changes
Mean Time to Recovery
Change Failure Rate

— 18 —

19 of 22

— 19 —

arlene.minkiewicz@unisonglobal.com

20 of 22

Infrastructure Monitoring (IM)

Focus on creating and maintaining IT Infrastructure components such as:

Servers
Storage
Software Units
Hardware Units
Data Centers
Networks

— 20 —

IM Tools should monitor

Server availability and efficiency
Effective use of disks and CPU
Reliability
Response time to errors
User Permissions
Storage
Database Health
Security
Network switches
Performance Trends

21 of 22

Application Monitoring (AM)

Focuses on all aspects of an application….

Application uptime
Security
Performance
Log time

AM Tools should monitor…

API and User Responses
Service Level Agreement (SLA) Status
Transaction time and volume
Error Rate
Throughput
Third party resource speed

— 21 —

22 of 22

Network Monitoring (NM)

Focus is on the evaluation of switches, servers, virtual machines, firewalls, and router via the following tasks

Discovering
Delineating
Detecting
Observing
Reporting

NM Tools need to monitor

Latency
Server bandwidth
Network packet flows
CPU use of hosts
Uptime

— 22 —