1 of 30

Prometheus Operators

in the Wild

Observability Day; KubeCon; Paris 2024

Bartłomiej Płotka; Google

Mahmoud Amin; Google

@bwplotka

@maxamins

@bwplotka

@maxamins

Bartek

https://sched.co/1YFeL

Tuesday, March 19 • 14:40 - 15:05

Hello Everyone!

I hope you have a great time on this Observability Day, it grown a lot since we started first one in 2022 in Detroit. It’s amazing to see!

In this talk we are really excited to introduce you to the topic of Prometheus Kubernetes Operators.

But it won't be a standard Prometheus Operator talk because of two reasons:

We actually introduce you to TWO different open source Prometheus Operators: official Prometheus Operator we all love and use and bit newer Google Managed Prometheus operator, in short GMP Operator
The second reason is that we will focus on metric collection Prometheus use case on Kubernetes clusters, so scraping metrics and sending them to remote metric backend of your choice. More on that later

So in the next 20m you will learn

Why Operators can be useful for that collection use case
How to configure and use Prometheus Operator and GMP Operator and when to use which
And you will see a quick demo on how it looks to setup both!

But before, a short introduction.

2 of 30

Who are we?

Mahmoud Amin

Software Engineer @ Google

GMP Operator maintainer
OSS (e.g. GSoC) mentor

Fun fact: 10 patents

@bwplotka

@maxamins

3 of 30

Who are we?

Bartłomiej Płotka

Senior Software Engineer @ Google

Tech Lead for Google Cloud Managed Service for Prometheus

Maintainer for e.g. Prometheus, Thanos, GMP Operator

Tech Lead for CNCF TAG Observability

@bwplotka

@maxamins

4 of 30

Who are we?

Bartłomiej Płotka

Senior Software Engineer @ Google

Tech Lead for Google Cloud Managed Service for Prometheus

Maintainer for e.g. Prometheus, Thanos, GMP Operator

Tech Lead for CNCF TAG Observability

Efficient Go book: https://www.bwplotka.dev/book

🔥 Giveaway & signing tomorrow 10:40 G18 O’Reilly booth

@bwplotka

@maxamins

5 of 30

Prometheus (collection) in Kubernetes

@bwplotka

@maxamins

[3m] Bartek

We will be discussing today, in our experience, the most pragmatic way of designing your Prometheus metric experience in your organization. We see this getting traction in the community too.

As you might know Prometheus is a project that packs all metric functionality into one binary: scraping (so metric collection) storage, querying, alerting and so on. This is useful, but for more dynamic environments like Kubernetes, especially when hybrid story is into play with compute locations across the globe, it’s quite beneficial to partition collection logic and rest.

The best part of this design is that tons of complexity is somewhere else, perhaps even done by separate team or vendor. On Kubernetes you have only stateless processes that simply pull metrics from you applications as you configure them, group those metrics, perhaps attach some metadata and filtering and stream those metrics to remote backend be it another Prometheus or Thanos, Cortex or any Prometheus compatible vendor.

6 of 30

Do we need Operators?

@bwplotka

@maxamins

7 of 30

K8s Consistent & Multi-Tenant Configuration

global:

scrape_interval: 15s

external_labels:

prom: 1

scrape_configs:

-job_name: “Application A”

metrics_path: /metrics

scheme: http

kubernetes_sd_config:

role: pod

authorization: ..

relabel_config: …

-job_name: “Application B”

metrics_path: /internal/metrics

scheme: https

kubernetes_sd_config:

role: pod

@bwplotka

@maxamins

Bartek

The first reason is scrape configuration. As you might know Prometheus requires scrape to be configured in one big yaml file per Prometheus server. Each application can be encoded as individual jobs or one job and filtered and enriched by powerful relabelling configuration. Once you apply this configuration e.g. put in ConfigMap and mount to Prometheus, your application will get scraped.

3 problems:

First you have to understand relabelling and generally Prometheus yaml definition
Second if there are different teams or people who are responsible for monitoring of their applications they literally step on their toes and can accidentally break configuration of other teams
Final problem is that to debug or check if scrape is actually correct you have to go to that Prometheus process UI and see what’s going on. This requires this Prometheus to be accessible to everyone, users of this has to understand what they look for etc There is also security aspect where tenants that have access only to one namespace suddenly see what other applications / pods are monitored.

8 of 30

K8s Consistent & Multi-Tenant Configuration

apiVersion: …

kind: …

spec:

selector:

matchLabels:

app: App A

endpoints:

- port: metrics

interval: 15s

status:

targets:

- address: …

status: OK

lastError: “”

duration: 596ms

apiVersion: …

kind: …

spec:

selector:

matchLabels:

app: App B

endpoints:

- port: metrics

interval: 10s

path: /internal/metrics

status:

targets:

- address: …

status: OK

lastError: “”

duration: 270ms

@bwplotka

@maxamins

Bartek

This is where operators can come handy as they can expose custom resource e.g. that allows configuring scrape configuration per pod

Both operators we are talking about today come with well defined custom resources that swaps things around. It’s now application responsibility to configure some portion of the scrape e.g.

Where the application metric endpoint is, what’s it’s URL and port, what’s HTTP path
How to filter it’s metrics, what labels to remove or add
Optionally how Prometheus can authenticate to that endpoint

More-over the status of pod scrape can be outputed to the custom resource, which gives focused overview of what’s happening only with your application.

And this is beautiful because this way it’s much easier to use a shared metric collection pipeline for individual user

9 of 30

Deployment & Scalability

@bwplotka

@maxamins

10 of 30

Deployment & Scalability

prometheus:

hashmod: true

series_per_replica: 1M

@bwplotka

@maxamins

11 of 30

Prometheus Operators to Help!

For:

👉🏽 Kubernetes native, simplified, multi-tenant scrape configuration

👉🏽 Easy installation & operations for complex, scalable deployments

@bwplotka

@maxamins

12 of 30

Prometheus Operator

https://prometheus-operator.dev/

@bwplotka

@maxamins

@bwplotka

@maxamins

13 of 30

Prometheus/PrometheusAgent CR

apiVersion: monitoring.coreos.com/v1

kind: Prometheus

metadata:

spec:

podMonitorNamespaceSelector: {}

podMonitorSelector: {}

shards: 3

remoteWrite:

- url: https://remote-endpoint…

@bwplotka

@maxamins

14 of 30

PodMonitor CR

apiVersion: monitoring.coreos.com/v1

kind: PodMonitor

spec:

namespaceSelector: A

selector:

matchLabels:

app: App A

podMetricsEndpoints:

- port: metrics

interval: 15s

apiVersion:

monitoring.coreos.com/v1

kind: PodMonitor

spec:

namespaceSelector: B

selector:

matchLabels:

app: App B

endpoints:

- port: metrics

interval: 10s

path: /internal/metrics

@bwplotka

@maxamins

15 of 30

GMP Operator

https://github.com/GoogleCloudPlatform/prometheus-engine

@bwplotka

@maxamins

@bwplotka

@maxamins

16 of 30

GMP deploys Prometheus as a DaemonSet

@bwplotka

@maxamins

17 of 30

GMP OperatorConfig CR

apiVersion: monitoring.googleapis.com/v1

kind: OperatorConfig

metadata:

namespace: gmp-public

exports:

url:

- https://remote-endpoint…

collection:

compression: gzip

filter:

matchOneOf:

- '{__name__!~"container_foo.*"}'

@bwplotka

@maxamins

18 of 30

GMP PodMonitoring CR

apiVersion: monitoring.googleapis.com/v1

kind: PodMonitoring

metadata:

namespace: namespace-1

spec:

selector:

matchLabels:

app: App A

endpoints:

- port: metrics

interval: 15s

@bwplotka

@maxamins

19 of 30

GMP ClusterPodMonitoring CR

apiVersion: monitoring.googleapis.com/v1

kind: ClusterPodMonitoring

metadata:

spec:

selector:

matchLabels:

app: App A

endpoints:

- port: metrics

interval: 15s

@bwplotka

@maxamins

20 of 30

Optimizing for DaemonSet Deployment

👉🏽 Tailoring CRDs (PodMonitoring) for DaemonSet case (e.g. no ServiceMonitor).

👉🏽 Leveraging how Kubernetes caches Pod API (it’s cached per node).

👉🏽 (upcoming) Secure & efficient secret support.

https://kccncna2022.sched.com/event/182IV/stateless-collectors-for-stateful-data-scaling-prometheus-as-a-node-agent-danny-clark-google

@bwplotka

@maxamins

Max

The GMP operator CRD’s are tailored for Daemonset, it also optimizes the Prometheus configuration for it and this is the hard part we are hiding from the user. Every CRD can be used freely with Daemonset.

The technical challenge of using Prometheus as daemonsets is to make sure to utilize the Kubernetes API watch cache which indexes pods by nodenames. This way it’s cheap for prometheus to know which pods exist on it’s node, WITHOUT impacting K8s performance. For this same reason we don’t have service monitor abstraction, like Prometheus Operator has.

If you are more interested in how it works you can look up the talk stateless collections for stateful data, scaling Prometheus as a Node Agent from KubeCon 2022.

We have an upcoming feature, secrets management, this feature makes the gmp operator handle your secrets without exposing it to a prometheus instance that doesn’t need access to that secret. It’s really challenging to implement in a secure and efficient way. But we have some ideas we are experiment with. So Stay tuned.

21 of 30

Demo Time!

@bwplotka

@maxamins

@bwplotka

@maxamins

22 of 30

TL;DR: Prometheus vs GMP Operator

Focus on hashmod deployment (plans for more)

Only DaemonSet based deployment

Allows HA + query + storage + collection use case

Tailored CRDs for collection use case

Supports ~all the Prometheus knobs

Simpler CRDs (e.g. opinionated, limited relabelling)

Service-based monitoring

Target status propagation

Simpler stack (also enabled by default on GKE)

More options & useful when you need local query

@bwplotka

@maxamins

23 of 30

Thanks!

Questions…?

Feedback & contributions welcome on both!

Prometheus Operator

https://prometheus-operator.dev/

GMP Operator

https://github.com/GoogleCloudPlatform/prometheus-engine

@bwplotka

@maxamins

@bwplotka

@maxamins

24 of 30

Demo Reference

@bwplotka

@maxamins

25 of 30

Demo Reference

@bwplotka

@maxamins

26 of 30

Demo Reference

@bwplotka

@maxamins

27 of 30

Demo Reference

@bwplotka

@maxamins

28 of 30

Demo Reference

@bwplotka

@maxamins

29 of 30

Demo Reference

@bwplotka

@maxamins

30 of 30

Demo Reference

@bwplotka

@maxamins