S3-Pipelines 02
DevOps Easy Learning
Application: Stomer
Problem to solve
Deploy and setup the application STOMER in all environments using robust automation
Core Areas pipelines
1- Development
2- Testing (QA)
3- Pre-production
4- Production
Requirements from Customer
1- High availability
2- zero downtime
3- Biweekly deployment
4- On demand scalability
Requirements from Development team
1- Provide an automated pipeline to build and test branches individually and provide notification to slack for any failure
2- Provide a sidekick quick testing pipeline with manual intervention, the pipeline should be capable of taking as input the branch name and deploy
3- Provide an automated pipeline to build test and deploy the main branch (develop) provide all build status notification to slack
4- Provide a 5 min deployment warning to user about upgrade.
5- Provide a Jenkins job capable of taking as input the branch name and tag , that will build, push images to dockerHub, tag the git branch and commit., with slack notification
Requirements from Quality Assurance (Testing) team
1- Provide a platform that will be capable to taking as input release tag from dev and perform some test ------- pipeline
2- provide a pipeline that will scan and test image provided
3- Provide and announcement job that will describe the state of the images with go-no-go to pre production. ------ slack notification
Requirements from Program management
1- Be mindful about resources cost. (cost optimization)
DevOps Expectation from Development team
1- What is the application made of
2- Language that each part of the application is written of
3- which library used (useful for base images)
4- app default port
5- how does the application run
6- demo of application running using libraries and binaries only
7- how to start the application
8- list of important file
9- important files locations and permission inside the container for the application to work properly
DevOps Expectation from QA team
1- list of tools to incorporate in their pipeline.
Requirements from Customer
1- High availability
2- zero downtime
3- Biweekly deployment unless otherwise specified
4- On demand scalability
5- Full backup all databases ( grand father, father, son)
6- The recovery time objective (RTO) 4h
7- Recovery Point Objective (RPO) 8h
8- Disaster recovery plan (DRP)
9- use AWS as cloud provider
10 - no requirements for self or manage service
Requirements from Database team
Acceptance
Criteria
Security team requirements
Application image is secured
All OS-level dependencies are added via a package manager
Image runs as unprivileged user
Does not listen on privileged ports
Debug logs are disabled
All security features are enabled by default (NIST 800-53 CM-6)
All security-related configuration options are documented
Administrative APIs (e.g. metrics, admin consoles) listen on a dedicated port (NIST 800-53 SC-2)
Audit logs are written separately from application logs
Helm Chart is secure
Default values result in most-secure install/configuration
All security-related values are documented
Applications deploy with app-specific service accounts (not default/shared)
Service account permissions follow least-privilege principles
Secrets are used for all sensitive information
All workloads define appropriate Requests and Limits
Helm chart is highly compatible
Helm’s built-in Capabilities object is used to appropriately customize releases for target environments
High Availability
All workloads have well-defined liveness and readiness probes
All critical workloads are deployed with multiple replicas and topological skew
Workloads requiring persistent volumes are defined by StatefulSets
Stateful services perform replication using headless services (allowing multiple StatefulSets to coordinate)
All non-batch workloads have a PodDisruptionBudget defined
Helm releases are managed separately from application images and support RollingUpdate deployments
Others requirements
Critical observability metrics are exposed via Datadog endpoints
Pod/Service Monitors are defined
Alerting thresholds are defined
Devops team : GET FAMILIAR WITH PROJECT
Devops team REVIEW: MANAGEMENT REQUIREMENTS
Brainstorm in couple of strategies we can implement to save cost.
Devops team planning: MANAGEMENT REQUIREMENTS
Goal reduce the cost:
1- use on demand VM on Jenkins
2- use on demand container as much as possible
3- Carefully select resources involved into the process
4- use docker compose in place maybe in dev or testing areas if possible
Devops team REVIEW: DEVELOPMENT REQUIREMENTS
1- Provide an automated pipeline to build and test branches individually and provide notification to slack for any failure — Multibranch pipeline integrated with slack.
2- Provide a sidekick quick testing pipeline job with manual intervention, the pipeline should be capable of taking as input the branch name and deploy – Parameterized pipeline (string).
3- Provide an automated pipeline job to build test and deploy the main branch (develop) provide all build status notification to slack Pipeline job set with webhook and integrate with slack.
4- Provide a 5 min deployment warning to user about upgrade. — Notify all users about upgrade (software change) add a sleep stage on pipeline indicated on 3.
5- Provide a Jenkins job capable of taking as input the branch name and tag , that will build, push images to dockerHub, tag the git branch and commit., with slack notification – Parameterized pipeline (string & choice & boolean) job to build and tag images.
Devops team planning: DEVELOPMENT REQUIREMENTS
Jenkins:
Devops team REVIEW:
SECURITY REQUIREMENTS
All OS-level dependencies are added via a package manager —---- Use package manager only to install packages (dockerfile, VMs)
Image runs as unprivileged user —---- drop root user in dockerfile ( example USER S3)
Does not listen on privileged ports —-- Do not use port under 1024
Debug logs are disabled —-- pass flag debug: false into all k8s workload
All security features are enabled by default (NIST 800-53 CM-6) – go online and read
All security-related configuration options are documented —--------- create documentation for security
Administrative APIs (e.g. metrics, admin consoles) listen on a dedicated port (NIST 800-53 SC-2) – go online and read
Audit logs are written separately from application logs – go online and read
Devops team REVIEW:
SECURITY REQUIREMENTS
Default values result in most-secure install/configuration – make is very secure
All security-related values are documented — write documentation on security
Applications deploy with app-specific service accounts (not default/shared) —-- create service account for each workload
Service account permissions follow least-privilege principles —- Only assign permissions that a user or application is entitled to nothing less nothing more
Secrets are used for all sensitive information — create secret if need be
All workloads define appropriate Requests and Limits – set cpu and memory usage
Devops team planning: SECURITY REQUIREMENTS
Dockerfile:
Kubernetes:
set debug to false into all k8s workloads manifest
create service account for each workload
set appropriate permission on service accounts
create secret file instead of using env variable in workloads
set cpu and memory usage for each workloads
create probe for all workloads
set replicas, PDB on all workloads
only use statefulset for DB and enable headless
use RollingUpdate upgrade strategy
Devops team REVIEW:
SECURITY REQUIREMENTS
Helm’s built-in Capabilities object is used to appropriately customize releases for target environments —-- create values file per environment.
Devops team planning: SECURITY REQUIREMENTS
Helm:
Devops team REVIEW:
SECURITY REQUIREMENTS
All workloads have well-defined liveness and readiness probes —- add liveness and readiness probes to deployment, statefulset, Deamonset
All critical workloads are deployed with multiple replicas and topological skew —- Set replicas to at least 2 and and add topological skew to deployment, statefulset, Daemonset
Workloads requiring persistent volumes are defined by StatefulSets —---- deploy all DB by statefulset
Stateful services perform replication using headless services (allowing multiple StatefulSets to coordinate) —if having more than one 1 sts allow healess and enable communication between
All non-batch workloads have a PodDisruptionBudget — defined add PDB to all workloads
Helm releases are managed separately from application images and support RollingUpdate deployments – appversion and chart version manage separately and all deployment strategy be RollingUpdate
Devops team REVIEW:
SECURITY REQUIREMENTS
Critical observability metrics are exposed via Datadog endpoints
Pod/Service Monitors are defined
Alerting thresholds are defined
Monitor all infrastructures and applications and services using datadog
Devops team REVIEW: DB TEAM REQUIREMENTS
Devops team planning: DB TEAM REQUIREMENTS
Devops team REVIEW: CUSTOMER TEAM REQUIREMENTS
1- High availability —-- always available no matter what
2- zero downtime —--- chose a upgrade strategy that does not make the app unavailable
3- Biweekly deployment —- new release or feature added biweekly
4- On demand scalability —--- auto scaling alone
Devops team planning: CUSTOMER TEAM REQUIREMENTS
High availability:
HA cluster
cross region replication
use cluster as much as possible
zero downtime:
blue green
canary
rollingUpdate.
Biweekly deployment:
make sure the infrastructure is ready for new release all the time
On demand scalability:
add HPA to all deploy or statefulset
https://docs.google.com/spreadsheets/d/1XF-M8k_OoMxrP6Zsl1VS2d3SkWM3d-9dM1FNhnBqlgc/edit#gid=0
https://drive.google.com/drive/folders/1tQiHZp4vJdd8snVz1C87xkKAaHPmmQ_U?usp=sharing
Devops At Work: Infrastructure planning
Devops At Work: Infrastructure planning
Tool that maybe used:
CICD:
1- Jenkins
REGISTRY:
1- Nexus
2- Harbor
ORCHESTRATION:
1- kubernetes (EKS)
2- Helm
MONITORING:
- datadog
DEPLOYMENT TOOL:
- ArgoCD
COMMUNICATION:
- slack
CODE SCANNING:
- SonarQube
CLOUD:
- AWS
VERSION CONTROL:
- Gitlab
CONTAINER MANAGEMENT:
- Docker-compose
CONTAINERIZATION:
- Docker
BUILD TOOL:
- maven
- golang
Tool that maybe used:
CICD:
1- gitlab
REGISTRY:
1- gitlab-CR
2- docker Hub
MONITORING:
- cloudwatch
DEPLOYMENT TOOL:
- Flux
CODE SCANNING:
- Veracode
VERSION CONTROL:
- Github
CONTAINER MANAGEMENT:
- ECS
BUILD TOOL:
- gradle
Tool that maybe used:
CICD:
1- Bambo
REGISTRY:
- ECR
List tools and their functionality
Code repos
Version control
containerization
Image repos
Continue integration
notification
Image repos
Code report
Code analysis
Continue deployment
Kubernetes package manager
Container orchestration
monitoring
Devops At Work: SOURCE CODE REPOSITORY
https://github.com/devopseasylearning/S3-projects/tree/develop/stormer-project02
Application Architecture
The application has been deliberately over-engineered to generate multiple de-coupled components. These components generally have different infrastructure dependencies, and may support multiple "backends"
Devops At Work: code review planning
JAVA:
- build with maven
GO:
- build with golang
NODE:
- research for now
NGINX:
- research for now
Devops At Work: DevOps Files planning
FILES:
- Dockerfiles
- Jenkinsfile
- Helm Chart
- Docker-compose file
- gitlab ci file
- Sonarqube Property file
- scripts
- config files
Devops At Work: Tools Adopted
DEPLOYMENT TOOL:
- ArgoCD
COMMUNICATION:
- slack
CODE SCANNING:
- SonarQube
CLOUD:
- AWS
CICD:
1- Jenkins
REGISTRY:
1- Nexus
2- Harbor
ORCHESTRATION:
1- kubernetes (EKS)
2- Helm
MONITORING:
- datadog
VERSION CONTROL:
- Github
CONTAINER MANAGEMENT:
- Docker-compose
CONTAINERIZATION:
- Docker
BUILD TOOL:
- maven
- golang
Devops At Work: Tools list planning
CICD:
1- Jenkins ------ install and configure plugins
REGISTRY:
1- Nexus -------- Launch and configure nexus server
ORCHESTRATION:
1- kubernetes (EKS) ------- Launch and configure EKS
2- Helm ----------------- NOT need will be ran by argocd
MONITORING:
- datadog ---------------- install and configure in k8s and create monitors
DEPLOYMENT TOOL:
- ArgoCD ------------------- install and configure in k8s
COMMUNICATION:
- slack ------------------- create a channel
CODE SCANNING:
- SonarQube ---------------- create quality gate for the projects
CLOUD:
- AWS ----------------- provision ressources
VERSION CONTROL:
- Github ----------------- configure the repo
CONTAINER MANAGEMENT:
- Docker-compose ------------ launch and configure on a VM
CONTAINERIZATION:
- Docker ------------ Install where required
BUILD TOOL:
- maven ---------- prebuild maven image
- golang ----------- prebuild golang image
Devops At Work: Launch all tools
Code repos
Version control
containerization
Image repos
Continue integration
notification
Image repos
Code report
Code analysis
Continue deployment
Kubernetes package manager
Container orchestration
monitoring
Devops At Work: Tools connection planning
Github
Jenkins
PagerDuty
Dockerhub
Sonar-cli
Sonarqube
maven
Golang
docker
Argocd
EKS
datadog
docker-compose
Helm
Slack
Person on call
Devops At Work: DEV WORK
Devops At Work: MAJOR DEVOPS FILES
FILES:
- Dockerfiles
- Jenkinsfile
- Helm Chart
- Docker-compose file
- gitlab ci file
- Sonarqube Property file
- scripts
- config files
Deploy on EKS
Test: go test
Test: maven test
Build .war
Build .exe
DEVELOPMENT PIPELINE (When pull request merged to develop branch)
1
2
6
5
4
3
7
8
9
14
12
13
11
17
16
15
18
10
1
2
3
DEVELOPMENT PIPELINE (When pull request merged to develop branch)
A pull request (PR) is merged in the develop branch (develop is the main branch for development and accept only code change by pull request)
notice the change on stormer repository via a preset webhook then clone the repository
Jenkins use the docker agent to launch a golang container to scan the all go code
1
2
Jenkins send build start notification on
3
4
Jenkins use the docker agent to launch a container to scan the all code
5
DEVELOPMENT PIPELINE (When pull request merged to develop branch)
use the docker agent to launch container to scan the code and the scan result sent back to
sent the build report to to be compared with quality gate
Jenkins use the docker agent to launch containers to build an artifact
6
7
sent the code analysis result back to
8
9
Jenkins use the docker agent to launch containers to build an artifact
10
DEVELOPMENT PIPELINE (When pull request merged to develop branch)
read to build all docker then push all of them to
Jenkins the charts on then update value.yaml file with new image tag
11
Jenkins commit back the new charts back into
12
13
notice the change on the on helm charts located on
14
DEVELOPMENT PIPELINE (When pull request merged to develop branch)
Invoke
15
to deploy in to
to deploy pull docker from
17
16
is available on the browser for
18
DEVELOPMENT PIPELINE (When pull request merged to develop branch)
agent notice and incident on the cluster and notify
ag notify the devops engineer on call by
ag notify the devops team on
1
2
3
Test: go build
Test: maven test
Build .war
Build .exe
DEVELOPMENT PIPELINE (When a code is changed on all branches except develop)
1
2
6
5
4
3
7
8
9
11
10
This pipeline on make sure an image can be built from the change made on the code
DEVELOPMENT PIPELINE (When pull request merged to develop branch)
a new branch is created or a commit is made to on of the branch
notice the change on stormer repository via a preset webhook the clone the repository
Jenkins use the docker agent to launch a golang container to scan the all go code
1
2
Jenkins send build start notification on
3
4
Jenkins use the docker agent to launch a container to scan the all code
5
DEVELOPMENT PIPELINE (When pull request merged to develop branch)
use the docker agent to launch container to scan the code and the scan result sent back to
sent the build report to to be compared with quality gate
Jenkins use the docker agent to launch containers to build an artifact
6
7
sent the code analysis result back to
8
9
Jenkins use the docker agent to launch containers to build an artifact
10
DEVELOPMENT PIPELINE (When pull request merged to develop branch)
read to build all docker ONLY no push required
11
5
4
3
2
1
DEVELOPMENT PIPELINE (build and push specific tag )
This pipeline is used to tag image and push to Nexus when everything is ok on dev side
Test: go build
Test: maven test
Build .war
Build .exe
DEVELOPMENT PIPELINE (Deploy a specific branch using K8S )
1
5
4
3
2
6
7
8
13
11
12
10
16
15
14
17
9
1
2
3
DEVELOPMENT PIPELINE (Deploy a specific branch using Docker-compose )
1
3
4
2
5
6
7
8
Devops At Work: QA WORK
Devops At Work: MAJOR DEVOPS FILES
1
4
3
2
5
QA PIPELINE
Devops At Work: PREPROD WORK
Devops At Work: MAJOR DEVOPS FILES
PREPROD PIPELINE
5
2
5
3
4
8
7
6
9
1
2
3
1
Devops At Work: PROD WORK
Devops At Work: MAJOR DEVOPS FILES
PROD PIPELINE
2
5
3
4
8
7
6
9
1
2
3
1
Devops At Work: DB WORK
Devops At Work: MAJOR DEVOPS FILES
1
3
2
4
DB PIPELINE automate backup
1
4
2
3
DB PIPELINE Restore backup
https://github.com/devopseasylearning/stomer/tree/main/src
CODE URL
Application Architecture
The application has been deliberately over-engineered to generate multiple de-coupled components. These components generally have different infrastructure dependencies, and may support multiple "backends"
Application Architecture
The application has been deliberately over-engineered to generate multiple de-coupled components. These components generally have different infrastructure dependencies, and may support multiple "backends"
Explain the whole pipeline scenario 01
Explain the whole pipeline scenario 02
Explain the whole pipeline scenario 03
Issues encountered while building the pipeline
No | Issue encountered | How did you fix it ? |
| | |
| | |
| | |
| | |
| | |
Issues encountered while building the pipeline
No | Issue encountered | How did you fix it ? |
| | |
| | |
| | |
| | |
| | |
Issues encountered while building the pipeline
No | Issue encountered | How did you fix it ? |
| | |
| | |
| | |
| | |
| | |
Take personal notes here