Prometheus Workshop

Adam Chen, Owen Wu, Zz Chen

Outline

• Installation time
• Prometheus Overview & Details
• Service Discovery (Kubernetes)
• Familiar with Metric, Grafana , Pushgateway
• Alert Manager with Practical Cases

Chapter 0: Setup Cloud9 and EKS

Following pictures of installation guide comes from: https://github.com/pahud/amazon-eks-workshop

​

7. execute ‘aws configure‘ to configure the credentials for your IAM user. Make sure this IAM User has AdministratorAccess and run ‘aws sts get-caller-identity’ - you should be able to see the returned JSON output like this.

​

​

Create IAM key if you have no one (1)

Create IAM key if you have no one (2)

Create IAM key if you have no one (3)

Create IAM key if you have no one (4)

Run command in Cloud9

$ git clone https://github.com/Taipei-HUG/Prometheus-workshop

$ cd CH_0

$ ./step1.sh # get all binary

$ ./step2.sh # setup eks cluster

$ ./step3.sh # get and setup helm

$ ./step4.sh # install kube-prometheus and push-gateway with LBS

$ ./get_links.sh # show all components link

Prometheus Overview

Chapter I : Prometheus Overview

• Introduction
• Arch and components
• With Config
• Service discovery
• Metrics type (counter, gauge, histogram, summary) (5 mins hands-on)
• Push Gateway
• Data Store
• Local storage
• Remote storage
• PromQL overview
• WebUI and official site, functions (5 mins hands-on)

Prometheus

• Pull based monitoring system
• Service Discovery
• Time series database
• Alertmanager
• Plenty of exporters
• Hierarchical architecture
• Support remote storage

Basic Prometheus Config

global:

[ scrape_interval: <duration> | default = 1m ]

[ scrape_timeout: <duration> | default = 10s ]

[ evaluation_interval: <duration> | default = 1m ]

rule_files: # Trigger alert

[ - <filepath_glob> ... ]

scrape_configs: # Find Target

[ - <scrape_config> ... ]

alerting:

alertmanagers:

[ - <alertmanager_config> ... ]

remote_write:

[ - <remote_write> ... ]

remote_read:

[ - <remote_read> ... ]

Service Discovery

https://prometheus.io/docs/prometheus/latest/configuration/configuration/#scrape_config

​

POD

Exporters

https://github.com/prometheus/prometheus/wiki/Default-port-allocations

​

Check EKS Cluster

Time for step 3 & 4

Obverservability : metrics

<Metric Name>{label1=value1, label2=value2, ...}

Data type

• Counter 單調遞增
• Gauge 可增可減
• Histogram (計算於 Prom-Server)
• Summary (Client 直接提供)

Data type (Histogram)

alertmanager_http_response_size_bytes_bucket{handler="/alerts",method="post",le="100"} 374

alertmanager_http_response_size_bytes_bucket{handler="/alerts",method="post",le="1000"} 374

alertmanager_http_response_size_bytes_bucket{handler="/alerts",method="post",le="10000"} 374

alertmanager_http_response_size_bytes_bucket{handler="/alerts",method="post",le="100000"} 374

alertmanager_http_response_size_bytes_bucket{handler="/alerts",method="post",le="1e+06"} 374

alertmanager_http_response_size_bytes_bucket{handler="/alerts",method="post",le="1e+07"} 374

alertmanager_http_response_size_bytes_bucket{handler="/alerts",method="post",le="1e+08"} 374

alertmanager_http_response_size_bytes_bucket{handler="/alerts",method="post",le="+Inf"} 374

alertmanager_http_response_size_bytes_sum{handler="/alerts",method="post"} 7480

alertmanager_http_response_size_bytes_count{handler="/alerts",method="post"} 374

Data type (Summary)

alertmanager_nflog_gc_duration_seconds{quantile="0.5"} 3.273e-06

alertmanager_nflog_gc_duration_seconds{quantile="0.9"} 3.273e-06

alertmanager_nflog_gc_duration_seconds{quantile="0.99"} 3.273e-06

alertmanager_nflog_gc_duration_seconds_sum 1.9283e-05

alertmanager_nflog_gc_duration_seconds_count 6

Practice #01

Get metrics from prometheus server

Get metrics

• Get metrics from alertmanager
• Browse http://Prometheus_ELB_DNS_NAME:9090/metrics

# HELP alertmanager_alerts How many alerts by state.

# TYPE alertmanager_alerts gauge

alertmanager_alerts{state="active"} 12

alertmanager_alerts{state="suppressed"} 0

# HELP alertmanager_alerts_invalid_total The total number of received alerts that were invalid.

# TYPE alertmanager_alerts_invalid_total counter

alertmanager_alerts_invalid_total 0

Remote Storage and sidecar (Thanos)

https://prometheus.io/docs/operating/integrations/#remote-endpoints-and-storage

Remote Storage and sidecar (Thanos)

Remote Storage and sidecar (Thanos)

PromQL with functions

• sum
• avg
• bool
• rate
• ...

https://prometheus.io/docs/prometheus/latest/querying/functions/

​

Practice #02

Find out basic functions of PromQL

PromQL with functions

• Browse HTTP://Prometheus_ELB:9090
• pushgateway_http_requests_total
• pushgateway_http_requests_total[2m]
• increase(pushgateway_http_requests_total[2m])
• increase(pushgateway_http_requests_total[2m]) / 120
• rate(pushgateway_http_requests_total[2m])
• pushgateway_http_requests_total{code="200"}
• sum(pushgateway_http_requests_total{code="200"}) by (instance)
• sum(http_requests_total{code="200"}) by (instance)

Alertmanager

Break Time

Chapter II : Service Discovery & Kubernetes

• Introduce the Service Discovery for Prometheus
• kubernetes
• Kubernetes Introduction
• Where to run Prometheus ?
• Prometheus Operator : Prometheus in Kubernetes
• Service Monitor - Elegant Service Discovery
• Exporter

Service Discovery

• Service Discovery，說他是微服務架構的靈魂也當之無愧 By 安德魯大大
• In Cloud Native environment, there may be a lot instances ( VM / pod ) start or shutdown at any time
• For Prometheus, Service Discovery is a key path to find where & what is the target to fetch metric.
• Kubernetes & Prometheus is perfect match

Prometheus with Service Discovery on file or …….

Service Discovery Configs

• We want to integrate Prometheus with the SD that's already there in your infrastructure, not invent yet more ways to do service discovery.
• The general principle with SD is to extract all the potentially useful information we can out of the SD, and let the user choose what they need of it using relabelling. This information is generally termed metadata.
• Ref
• https://github.com/prometheus/prometheus/tree/master/discovery
• https://prometheus.io/docs/prometheus/latest/configuration/configuration/

Resources & Service Discovery in Kubernetes

• Pod
• Label
• Selector
• Service

Pods

Pods

Logical Application

• One or more containers and volumes
• Shared namespaces
• One IP per pod

​

Pod

nginx

monolith

10.10.1.100

Labels

Labels

Arbitrary meta-data attached to Kubernetes object

​

​

​

Pod

hello

Pod

hello

labels:� version: v1

track: stable

​

​

​

labels:� version: v1

track: test

​

​

​

Labels

selector: “version=v1”

​

​

​

Pod

hello

Pod

hello

labels:� version: v1

track: stable

​

​

​

labels:� version: v1

track: test

​

​

​

Labels

selector: “track=stable”

​

​

​

Pod

hello

Pod

hello

labels:� version: v1

track: stable

​

​

​

labels:� version: v1

track: test

​

​

​

Services

Kubernetes Service

49

Client

Service�selector: app=app2

Service�selector: app=myApp

Practice #03

Try to understand label selector with kubectl

Demo Selector with kubectl

• $ kubectl get pod -n kube-system --show-labels
• $ kubectl get pod -n kube-system --show-labels -l k8s-app=kube-dns
• $ kubectl get service -n kube-system
• $ kubectl describe service kube-dns -n kube-system
• $ kubectl get pod -n kube-system --show-labels -l k8s-app=kube-dns -o wide

Flow with service discovery

• `relabel_configs`
• For Service Discovery
• `metrics_relabel_configs`
• For metrics
• `keep` vs `drop`
• mostly is replacement
• Ref:
• https://blog.freshtracks.io/prometheus-relabel-rules-and-the-action-parameter-39c71959354a

Practice #04

Using kubernerte_sd_configs to discover coredns

Kuberentes Service Discovery - CoreDNS

• $ cd CH_2/coredns_scrape_configs
• $ sh generate_yaml.sh
• $ kubectl apply -f manifests/
• $ sh restart_prometheus.sh

​

• Ref
• https://github.com/prometheus/prometheus/blob/release-2.8/documentation/examples/prometheus-kubernetes.yml

Where to run Prometheus?

• In Kubernetes ?
• On a dedicated machine/VM?
• In Kubernetes, Easily to access the Pod, otherwise it would encounter a lot of difficulty.
• But Kubernetes prefers to treat application as stateless, the restart / upgrade need more care.
• On dedicated Machine, it should be more easy to management ?

Why use Kubernetes ?

Why use Kubernetes ?

• A universal platform for manage application
• Easy to scale
• Unified, powerful interface for operation
• Cost
• Leverage Infrastructure with other application
• Using Spot Instance https://eksworkshop.com/spotworkers/
• A lot of resources & support

Prometheus Operator

Operators

Operators

A Kubernetes Operator helps extend the types of applications that can run on Kubernetes by allowing developers to provide additional knowledge to applications that need to maintain state.

Mostly it focus on automating and the special know how of the application. Simply the deployment and maintain.

​

​

Prometheus Operator Architecture

Prometheus Operator Object(CRD)

• Prometheus
• which defines a desired Prometheus deployment.
• PrometheusRule
• which can be loaded by a Prometheus instance containing Prometheus alerting and recording rules.
• Alertmanager
• which defines a desired Alertmanager deployment.
• ServiceMonitor

Service Monitor

• Declaratively define how a dynamic set of services should be monitored.
• Which services are selected to be monitored with the desired configuration is defined using label selections.
• The ServiceMonitor object introduced by the Prometheus Operator in turn discovers those Endpoints objects and configures Prometheus to monitor those Pods.

Practice #05

Using Service Monitor to discover coredns

Service Monitor Demo

• $ cd CH_2/coredns_service_monitor
• diff ../coredns_scrape_configs/manifests/prometheus-prometheus.yaml manifests/prometheus-prometheus.yaml
• $ kubectl apply -f manifests/
• $ sh reload_prometheus.sh

For resource not in kubernetes

• https://devops.college/prometheus-operator-how-to-monitor-an-external-service-3cb6ac8d5acb
• https://cloud.google.com/blog/products/gcp/kubernetes-best-practices-mapping-external-services

​

​

How to supply metrics for Prometheus

Natively or Exporter

• Prometheus is pull-base monitoring system
• Prometheus’ best practices are to natively instrument the services .
• But for non-natively-instrumented services (such as Memcached, Postgres, etc.) it is possible to use an exporter.
• An exporter is a process that runs alongside your service and translates metrics from the service into the format Prometheus understands.

​

Exporter

More Exporter

• https://prometheus.io/docs/instrumenting/exporters/
• https://medium.com/@nachomillangarcia/prometheus-alerts-for-aws-daily-costs-3374137d1371
• https://github.com/prometheus/cloudwatch_exporter
• https://medium.com/kubecost/introducing-kubecost-a-better-approach-to-kubernetes-cost-monitoring-b5450c3ae940

​

Chapter III

Grafana & Pushgateway

Grafana

a analytics platform

to query, visualize and alert

Access Dashboards

./CH_0/get_links.sh

Preloaded dashboards from kube-prometheus

Practice: Monitor CoreDNS status

1. Create a new dashboard
2. Add a new query to get the status of CoreDNS

kube_pod_status_ready{pod=~"coredns-(.*)", condition="false"}

Pushgateway

allow ephemeral and batch jobs

to expose metrics

Pushgateway

• https://github.com/prometheus/pushgateway
• For ephemeral workloads and batch jobs
• They are not exist long enough to be scrapped
• Pushgateway keep metrics for these kind of jobs
• Prometheus can pull metrics from /metrics of pushgateway

​

​

Play with pushgateway

$ echo "some_metric 3.14" | curl --data-binary @- http://{URL_OF_PUSHGATEWAY}:9091/metrics/job/some_job

$ echo "progress 12" | curl --data-binary @- http://{URL_OF_PUSHGATEWAY}:9091/metrics/job/playing

​

Check the metrics

Metrics: http://<pushgateway-host>:9091/metric

UI: http://<pushgateway-host>:9091/

Check at prometheus console

Break Time

CH 4 : Alerting & Practical Cases

Let’s pull the trigger.

Alertmanager

Alertmanager

• Global setting
• Template
• Route
• Receiver
• Inhibit rules

global:

[ resolve_timeout: <duration> | default = 5m ]

[ slack_api_url: <secret> ]

[ http_config: <http_config> ]

​

templates:

[ - <filepath> ... ]

​

route: <route>

​

receivers:

- <receiver> ...

​

inhibit_rules:

[ - <inhibit_rule> ... ]

Alertmanager - Route

"route":

"group_by":

- "job"

"group_interval": "1m"

"group_wait": "30s"

"receiver": "slack_alert1"

"repeat_interval": "3m"

"routes":

- "match":

"alertname": "Watchdog"

"receiver": "slack_alert2"

Setup Slack

• Into Slack Workspace
• Create Incoming Webhook URL
• Create your own channel x 2

​

Setup Alertmanager & Nginx

$ vi CH_4/alertmanager.yaml

​

​

​

​

​

$ CH_4/apply_change.sh

​

​

"receivers":

- "name": "slack_alert1"

"slack_configs":

- "api_url": "https://hooks.slack.com/services/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"

"channel": "#alert_1"

- "name": "slack_alert2"

"slack_configs":

- "api_url": "https://hooks.slack.com/services/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"

"channel": "#alert_2"

$ CH_4/helm_nginx_install.sh

Install rule 1

$ cat alert_rule_1.yaml

apiVersion: monitoring.coreos.com/v1

kind: PrometheusRule

metadata:

labels:

prometheus: k8s

role: alert-rules

name: prometheus-example-rules

namespace: monitoring

spec:

groups:

- name: yourname.rules

rules:

- alert: YournameAlert

expr: vector(1)

$ kubectl apply -f alert_rule_1.yaml

Install rule 2

$ cat alert_rule_2.yaml

apiVersion: monitoring.coreos.com/v1

kind: PrometheusRule

metadata:

labels:

prometheus: k8s

role: alert-rules

name: prometheus-example-rules2

namespace: monitoring

spec:

groups:

- name: nginx_rule

rules:

- alert: NGINXAlert

expr: nginx_ingress_controller_nginx_process_requests_total > 1000

$ kubectl apply -f alert_rule_2.yaml

$ CH_4/trigger_nginx_alert.sh

Checkout Slack for alerts

Remove all resource in AWS

Please remember run below script, remove all AWS resource to prevent unnecessary cost.

$ cd CH_0

$ ./uninstall.sh

THANKS