1 of 20

GitOps on scale

with ArgoCD

Denys Zhyhulin | DevOps Engineer

dzhyhulin@bostonsd.com

https://www.linkedin.com/in/deniszn

2 of 20

Table of Contents

GitOps

Overview
The GitOps Terms
The GitOps Principles
An Example GitOps Workflow
Existing GitOps Controllers

ArgoCD

Overview
The “App of Apps” Pattern
The “Application Set” Pattern

Demo

Demo Environment

3 of 20

GitOps Overview

4 of 20

Observable

Standard enforcer

Relatively young

DevOps native

Scalable

We love GitOps because it’s…

Git-centric

Universal

Rollback-capable

Relatively young: First introduced in 2020 by Weaveworks. Yet, GitOps nowadays has a constantly growing community of practitioners, multiple regular conferences, and a set of production-grade tools.

DevOps native: GitOps is DevOps. GitOps is the natural progression of DevOps, and it implements the best of what DevOps practitioners have already been doing, having no idea what it is.

Where standardization is enforced: GitOps heavily relies on source repository structure. Thus, it forces practitioners to plan ahead and use standardized approaches that follow the well-known best practices.

Observable: Most GitOps controllers expose metrics of any managed application state, which enables practitioners to configure monitoring and alerting to get an immediate feedback about possible failures.

Scalable: Scaling GitOps-powered infrastructure is as easy as copying a directory and renaming it. Moreover, some GitOps controllers (i.e., ArgoCD ApplicationSet) enable practitioners to simplify the process even more.

Git-centric: GitOps mainly relies on (but not limits to) Git as the single source of truth. This feature pushes the Infrastructure as Code approach one step ahead, making sure that a Git repository is always up to date and contains current infrastructure state.

Universal: It’s possible to use anything as a source of a deployment manifest. While the most popular tools are Helm and Kustomize, it’s also possible to use arbitrary sources that render a manifest (i.e., Jsonnet, Python script, HTTP response).

Rollback-capable: With GitOps, rolling back an application is as easy as reverting a commit. Thus, a rollback step can be easily added to existing pipelines or automated with a metric-based tools (i.e., Argo Rollouts) or platforms.

5 of 20

The GitOps terms

System
Desired State
Declarative Description
(Configuration) Drift
Reconciliation
GitOps Controller

System

Is defined as one or more runtime environments consisting of the following entities:

Resources under management
The management agents within each runtime
The policies for controlling access and management of repositories, deployments, and runtimes

Desired State

Is the representation of the way you want the system to work in an “end state”

Declarative Description

Describes the desired operating state of a system without specifying procedures for how that state will be achieved

(Configuration) Drift

When a system's actual state has been moved or is in the process of shifting away from the desired state

Reconciliation

A continuous process of ensuring the actual state of a system matches its desired state

GitOps Controller

GitOps controllers are tools that continuously reconcile a system state to detect and remediate drifts

6 of 20

The GitOps Principles

Declarative
Versioned and Immutable
Pulled Automatically
Continuously Reconciled

7 of 20

An Example GitOps Workflow

8 of 20

Existing GitOps Controllers

9 of 20

ArgoCD Overview

10 of 20

We should evaluate ArgoCD because it’s…

Observable

Kubernetes native

Minimalistic

Configurable

Fully declarative

Scalable

Both CD and CD capable

A part of an ecosystem

Kubernetes native: ArgoCD runs in Kubernetes, leveraging Custom Resources to define and manage applications. This enables us to apply any Kubernetes-native practice to ArgoCD resources.

Minimalistic: While a standard ArgoCD installation provides everything required to build and operate a GitOps system, only 3 Custom Resource Definitions are shipped and required for it to operate.

Configurable: Almost every part of ArgoCD can be easily configured. By default, ArgoCD supports Helm, Jsonnet, and Kustomize to render manifests, but the list can easily be extended with Config Management Plugins.

Observable: ArgoCD exposes metrics about every application managed by it as well as controller-specific metrics. This enables us to configure alerts, dashboards, notifications, receive immediate feedback about sync status, etc.

Fully declarative by design: All ArgoCD configuration can (and should) be managed declaratively. Applications, cluster definitions, Git/Helm/OCI credentials, etc. are stored as Kubernetes resources and can be managed with ArgoCD.

Scalable: ArgoCD supports remote cluster management, which enables us to deploy existing applications into multiple clusters at once. Application Sets makes this process even easier, enabling us to deploy applications into multiple clusters on demand.

User friendly: ArgoCD is shipped with a beautiful and functional Web UI, alongside with the CLI tool. While we use Git as a single source of truth, it’s very handy to give developers access to Web UI to check the state of their deployments, get some logs, etc.

Both CD and CD capable: ArgoCD can act as a Continuous Deployment or Continuous Delivery tool, or both at the same time. It’s quite handy to sync some critical (or just huge) applications manually, and let the rest be synced automatically.

Suitable for both small and large companies: Out of the box, ArgoCD supports SSO integration (OIDC, OAuth2, LDAP, SAML 2.0, GitHub, GitLab, Microsoft, LinkedIn), RBAC, custom CSS styles, and so on.

11 of 20

The ”App of Apps” Pattern

Apps of Apps (one per cluster). Usually gets synced by one more app.

Base applications to be deployed.

Cluster-specific overlays, i.e., ‘.spec.destination.name’.

12 of 20

The ”App of Apps” Pattern

Base Application

Cluster-level global overlay

.spec.destination.name is a placeholder

that MUST be overridden by a cluster-specific overlay

.spec.source.targetRevision can be overridden on demand

Cluster-level application overlay

Overrides .spec.destination.name

Overrides .spec.source.targetRevision

13 of 20

The ”App of Apps” Pattern

�Pros:

Easy to implement and operate
Overall customizability
Easy to validate and troubleshoot
Stability

�Cons:

Scaling Complexity
Dependency management
Code duplication

�Pros:

Easy to implement and operate: All that’s required is a single Git repository. The pattern itself is a structure-agnostic, so overlays and Applications can be grouped by projects, BUs, teams, etc., to fit your needs.
Customizability: Usage of separate overlays per cluster enables us to fine tune services for every cluster or environment.
Easy to validate and troubleshoot: It’s easy to validate the whole structure by just rendering a parent application and passing the output to kubeconform/kubectl apply/linters to ensure, that the setup works as expected.

�Cons:

Scaling: Having 20+ managed clusters creates a lots of overlays, service definitions, etc. Sometimes, it’s fairly hard to manage that amount of YAMLs which leads to errors because of the human factor.
Complexity: Kustomize is an industry standard for implementing the pattern, and for those who aren’t familiar with the tool it can create difficulties. While it’s possible to use Helm or Jsonnet to implement the pattern, it raises complexity even more.
Dependency management: A change in a base application (i.e., chart version or parameter) can affect multiple overlays if corresponding field is not explicitly overridden.

14 of 20

The ”Application Set” Pattern

An Application Set resource. Generates Applications based on Generators’ output.

15 of 20

The ”Application Set” Pattern

ApplicationSet

Enable Go Template for this Application Set

Fetch all clusters matched by matchLabel

Extract the value of the annotation as a variable,

provide a default value if not set

Generate an Application for each element of the resulting list:

- name: demo-worker1

values:

chartVersion: 7.1.256 ( the default value )

- name: demo-worker2

values:

chartVersion: 6.1.47 ( via the annotation )

16 of 20

The ”Application Set” Pattern

�Pros:

Universal
Fail-safe
Scalable by design
Go-template support

�Cons:

Troubleshooting complexity
Development complexity
Refactoring complexity
Allows insecure actions

�Pros:

Universal: Application Set Controller supports vast variety of generators: List, Cluster, Git, SCM, Pull Request, etc. Also, it’s possible to get output from any HTTP server and use it as a generator output (the Plugin generator). Moreover, Application Sets can be used alongside with the App of Apps pattern which enables us to make a really customizable GitOps system.
Fail-safe: An Application Set policy (create-only, create-update, create-delete, sync) or a Dry Run mode can be enabled during the development phase to ensure operational safety.
Scalable by design: Application Sets are designed to be scalable. When correctly configured, an Application Set can replace a huge amount of manually created Applications.
Go-template: Application Sets support Go Template engine with the Sprig library which enables quite advanced templating capabilities.

�Cons:

Development complexity: It takes some time to get used to Application Sets, and the documentation doesn’t cover all possible edge cases and difficulties. Also, Application Sets enforce standardizations, which can become a problem in some cases.
Troubleshooting complexity : There’s no other way to debug failed Application Set rather than checking an Application Set Controller logs or running a local controller instance with a debugger attached.

17 of 20

Demo

18 of 20

Demo Environment

�Tech Stack:

Kubernetes
ArgoCD
Kustomize
Helm
Git
Ingress Nginx (a demo service)
Cert-Manager (a demo service)
Reflector (a demo service)
Taskfile (Optional)
Terraform (Optional)

19 of 20

Links

20 of 20

Boston SoftDesign, Inc.�233 Needham Street, Suite 550

Newton Upper Falls, MA 02464-1605

Office: +1 (617) 546-8088

E-mail: info@bostonsd.com

Thank you!