1 of 49

Instrumenting applications and alerting with Prometheus

Simon Pasquier / Senior Software Engineer @ Red Hat

@SimonHiker

​

2 of 49

Who am I?

  • Working at Red Hat.
  • Prometheus team member.
  • Alertmanager maintainer.

3 of 49

“Prometheus is a systems and service monitoring system”

​

source: https://github.com/prometheus/prometheus/blob/master/README.md

4 of 49

Server

Application

Application

# HELP http_requests_total Total number of HTTP requests.

# TYPE http_requests_total gauge

http_requests_total{code=”200”,method=”GET”} 10

http_requests_total{code=”500”,method=”GET”} 0

...

HTTP

5 of 49

Timeseries data model

http_requests_total{code=”200”,method=”GET”,job=”app”,instance=”foo:8080”}

6 of 49

Timeseries data model

http_requests_total{code=”200”,method=”GET”,job=”app”,instance=”foo:8080”}

name

7 of 49

Timeseries data model

http_requests_total{code=”200”,method=”GET”,job=”app”,instance=”foo:8080”}

labels

8 of 49

Timeseries data model

timeseries

samples

​

t

http_requests_total{code=”200”,method=”GET”,job=”app”,instance=”foo:8080”}

10

http_requests_total{code=”400”,method=”GET”,job=”app”,instance=”foo:8080”}

0

http_requests_total{code=”500”,method=”GET”,job=”app”,instance=”foo:8080”}

0

9 of 49

Timeseries data model

​

t

t+1m

http_requests_total{code=”200”,method=”GET”,job=”app”,instance=”foo:8080”}

10

20

http_requests_total{code=”400”,method=”GET”,job=”app”,instance=”foo:8080”}

0

0

http_requests_total{code=”500”,method=”GET”,job=”app”,instance=”foo:8080”}

0

2

10 of 49

Timeseries data model

​

t

t+1m

t+2m

t+3m

t+4m

http_requests_total{code=”200”,method=”GET”,job=”app”,instance=”foo:8080”}

10

20

40

50

58

http_requests_total{code=”400”,method=”GET”,job=”app”,instance=”foo:8080”}

0

0

0

1

2

http_requests_total{code=”500”,method=”GET”,job=”app”,instance=”foo:8080”}

0

2

2

5

8

11 of 49

Querying

http_requests_total

​

t

t+1m

t+2m

t+3m

t+4m

http_requests_total{code=”200”,method=”GET”,job=”app”,instance=”foo:8080”}

10

20

40

50

58

http_requests_total{code=”400”,method=”GET”,job=”app”,instance=”foo:8080”}

0

0

0

1

2

http_requests_total{code=”500”,method=”GET”,job=”app”,instance=”foo:8080”}

0

2

2

5

8

12 of 49

Querying

http_requests_total

http_requests_total{code=”400”}

​

t

t+1m

t+2m

t+3m

t+4m

http_requests_total{code=”200”,method=”GET”,job=”app”,instance=”foo:8080”}

10

20

40

50

58

http_requests_total{code=”400”,method=”GET”,job=”app”,instance=”foo:8080”}

0

0

0

1

2

http_requests_total{code=”500”,method=”GET”,job=”app”,instance=”foo:8080”}

0

2

2

5

8

13 of 49

Querying

http_requests_total

http_requests_total{code=”400”}

http_requests_total{code=”500”}[5m]

​

t

t+1m

t+2m

t+3m

t+4m

http_requests_total{code=”200”,method=”GET”,job=”app”,instance=”foo:8080”}

10

20

40

50

58

http_requests_total{code=”400”,method=”GET”,job=”app”,instance=”foo:8080”}

0

0

0

1

2

http_requests_total{code=”500”,method=”GET”,job=”app”,instance=”foo:8080”}

0

2

2

5

8

14 of 49

Querying

http_requests_total

http_requests_total{code=”400”}

http_requests_total{code=”500”}[5m]

rate(http_requests_total{code=”500”}[5m])

​

t

t+1m

t+2m

t+3m

t+4m

http_requests_total{code=”200”,method=”GET”,job=”app”,instance=”foo:8080”}

10

20

40

50

58

http_requests_total{code=”400”,method=”GET”,job=”app”,instance=”foo:8080”}

0

0

0

1

2

http_requests_total{code=”500”,method=”GET”,job=”app”,instance=”foo:8080”}

0

2

2

5

8

15 of 49

Querying

http_requests_total

http_requests_total{code=”400”}

http_requests_total{code=”500”}[5m]

rate(http_requests_total{code=”500”}[5m]) =~ 0.0267 req/sec

​

t

t+1m

t+2m

t+3m

t+4m

http_requests_total{code=”200”,method=”GET”,job=”app”,instance=”foo:8080”}

10

20

40

50

58

http_requests_total{code=”400”,method=”GET”,job=”app”,instance=”foo:8080”}

0

0

0

1

2

http_requests_total{code=”500”,method=”GET”,job=”app”,instance=”foo:8080”}

0

2

2

5

8

16 of 49

Querying

http_requests_total

http_requests_total{code=”400”}

http_requests_total{code=”500”}[5m]

rate(http_requests_total{code=”500”}[5m])

sum(rate(http_requests_total{code=~”5..”}[5m])) / sum(rate(http_requests_total[5m])) > 0.05

​

t

t+1m

t+2m

t+3m

t+4m

http_requests_total{code=”200”,method=”GET”,job=”app”,instance=”foo:8080”}

10

20

40

50

58

http_requests_total{code=”400”,method=”GET”,job=”app”,instance=”foo:8080”}

0

0

0

1

2

http_requests_total{code=”500”,method=”GET”,job=”app”,instance=”foo:8080”}

0

2

2

5

8

17 of 49

Prometheus in a nutshell

  • Powerful query language.

18 of 49

Prometheus in a nutshell

  • Powerful query language.
  • Pull model (with service discovery).

19 of 49

Prometheus in a nutshell

  • Powerful query language.
  • Pull model (with service discovery).
  • Alerting based on timeseries.

20 of 49

Prometheus in a nutshell

  • Powerful query language.
  • Pull model (with service discovery).
  • Alerting based on timeseries.
  • Simple to operate.

21 of 49

Why instrument?

  • Measure customer satisfaction.

22 of 49

Why instrument?

  • Measure customer satisfaction.
  • Troubleshooting & debugging.

23 of 49

Why instrument?

  • Measure customer satisfaction.
  • Troubleshooting & debugging.
  • Capacity planning.

24 of 49

Why choose Prometheus?

  • Widely adopted exposition format.

25 of 49

Why choose Prometheus?

  • Widely adopted exposition format.
  • Low overhead.

26 of 49

Why choose Prometheus?

  • Widely adopted exposition format.
  • Low overhead.
  • Relevant for legacy and cloud-based architectures.

27 of 49

Example of code instrumentation

Server

Hello app

Alertmanager

Alerts

https://github.com/simonpasquier/simple_flask_application

28 of 49

Instrumentation methods

  • Request/response applications
    • Rate-Errors-Duration (RED)

29 of 49

Instrumentation methods

  • Request/response applications
    • Rate-Errors-Duration (RED)
  • Physical resources
    • Utilization-Saturation-Error (USE)

30 of 49

Instrumentation methods

  • Request/response applications
    • Rate-Errors-Duration (RED)
  • Physical resources
    • Utilization-Saturation-Error (USE)
  • Processing pipelines
    • Data freshness

31 of 49

Good practices of instrumentation

  • Use client libraries whenever possible.

32 of 49

Good practices of instrumentation

  • Use client libraries whenever possible.
  • Be consistent when naming of metrics.

33 of 49

Good practices of instrumentation

  • Use client libraries whenever possible.
  • Be consistent when naming of metrics.
    • duration ✗

34 of 49

Good practices of instrumentation

  • Use client libraries whenever possible.
  • Be consistent when naming of metrics.
    • duration ✗
    • http_requests_duration ✗

35 of 49

Good practices of instrumentation

  • Use client libraries whenever possible.
  • Be consistent when naming of metrics.
    • duration ✗
    • http_requests_duration ✗
    • http_requests_duration_milliseconds ✗

36 of 49

Good practices of instrumentation

  • Use client libraries whenever possible.
  • Be consistent when naming of metrics.
    • duration ✗
    • http_requests_duration ✗
    • http_requests_duration_milliseconds ✗
    • http_requests_duration_seconds ✓

37 of 49

Good practices of instrumentation

  • Use client libraries whenever possible.
  • Be consistent when naming of metrics.
  • Use labels with care.

38 of 49

Good practices of instrumentation

  • Use client libraries whenever possible.
  • Be consistent when naming of metrics.
  • Use labels with care.
  • Focus on metrics that are important for the business.

39 of 49

Good practices of instrumentation

  • Use client libraries whenever possible.
  • Be consistent when naming of metrics.
  • Use labels with care.
  • Focus on metrics that are important for the business.
  • Measure both client and server side.

40 of 49

Good practices of instrumentation

  • Use client libraries whenever possible.
  • Be consistent when naming of metrics.
  • Use labels with care.
  • Focus on metrics that are important for the business.
  • Measure both client and server side.
  • Add “informational” metrics.

41 of 49

Alerting

  • Regular PromQL queries.

42 of 49

Alerting

  • Regular PromQL queries.
  • Use symptom-based alerts.

43 of 49

Alerting

  • Regular PromQL queries.
  • Use symptom-based alerts.
  • Have metrics to identify the causes.

44 of 49

Alerting

  • Regular PromQL queries.
  • Use symptom-based alerts.
  • Have metrics to identify the causes.
  • Differentiate between urgent and routine alerts.

45 of 49

Conclusion

  • Software needs good instrumentation.

46 of 49

Conclusion

  • Software needs good instrumentation.
  • Metrics are cheap.

47 of 49

Conclusion

  • Software needs good instrumentation.
  • Metrics are cheap.
  • Prometheus is a great fit.

48 of 49

We’re hiring!

https://www.redhat.com/en/jobs

49 of 49