1 of 9

Celebrating Prometheus 3.0

All You Need To Know!

Bartek Płotka, Senior SWE at Google

bwplotka

2 of 9

Prometheus 3.0: UI and Remote Write 2.0

Prometheus UI Overhaul

Efficient & Feature rich Remote Write 2.0

https://prometheus.io/docs/specs/remote_write_spec_2_0/

bwplotka

3 of 9

Prometheus 3.0: OpenTelemetry and UTF-8

OTLP metrics receiving

https://prometheus.io/docs/guides/opentelemetry/

UTF-8 Support for metric and label names

Make sure to escape (quote) those names!

bwplotka

The next two notable Prometheus 3.0 features will be exciting for those who want to use Prometheus server to ingest metrics in OpenTelemetry format.

3.0 comes with an improved OTLP metric receiver endpoint that allows you to push OTLP metrics to Prometheus In 3.0 this endpoint works even better with Prometheus out-of-order feature, and we recommend enabling it for OTLP traffic given the nature of metric pushes. We also recommend a set of known resource attributes to attach to your metrics, while keeping verbose attributes in the special target_info metric which you can later join on using PromQL.

<click>

Now, to enable the best experience of using OpenTelemetry semantic conventions for metrics…

<click>

Prometheus 3.0 comes with full UTF-8 storage and UI support, by default. This allows users to query OpenTelemetry metrics in PromQL using exactly the same metric name and label names as defined in their Open Telemetry instrumentation.

<click>

Just keep in mind that all those metric and label names that contain previously invalid UTF-8 characters have to be quoted!

In general, UTF-8 support will be also incredibly helpful for those who don't use the English language.

4 of 9

Prometheus 3.0: Breaking Changes

Default UTF-8 supports means no more strict metric and label name character validation
Normalization of le and quantile labels (to float format e.g. le="1.0")
Prometheus structured logging is slightly different
Flags

Some feature flags removed; enabled by default
Agent mode set by --agent flag

Config

Remote Write defaults to HTTP/1.1
Renamed scrape_classic_histograms to always_scrape_classic_histograms

PromQL

The . pattern in regex now matches newline characters
Lookback and range selectors are now (start, end] previously [start, end]
Renamed (and experimental) holt_winters to double_exponential_smoothing

Scrape Protocols

More strict content-type check; error instead of fallback to text format

TSDB storage compatibility only with v2.55

TL;DR

For better use experience and efficiency we did a project configuration and behaviour cleanup for certain edge cases.

See the migration guide for the detailed list of breaking changes!

bwplotka

5 of 9

❤️ Optimizations since 2.0 (lower is better)

GKE node capacity: 8 CPU and 49 GB allocatable memory

v3.0 now!

v2.18 4y ago

v2.0 7y ago

v3.0 -4.4x

v2.18 -1.8x

OOM 🔥

v2.0 (base)

While it lasted…

v3.0 -3.5x

v2.0 (base)

While it lasted…

v2.18 -1.4x

bwplotka

Now we all love numbers and metrics, so I did some benchmarking yesterday to see what progress we made in terms of Prometheus efficiency, for exactly the same load of metrics and PromQL queries.

<click>

And I benchmarked and compared 3 Prometheus versions: 2.0, 2.18 from 4y ago and 3.0 today.

<click>

And it’s pretty amazing what the Prometheus community achieved here! In terms of memory usage, I ran out of memory for the 2.0 version, but nevertheless, we can see a massive decrease in memory usage, at least more than 4 times less for the 3.0 version than the first 2.0 version. Those numbers would likely be even better if we did not shrink our machines in our Prometheus benchmarking framework, given recent optimizations.

<click>

CPU use is improved significantly in similar fashion with 3.0 using almost 4 times less CPU seconds compared to 2.0.

6 of 9

Future work!

New, more inclusive governance
More Open Telemetry compatibility
Deliver Open Metrics 2.0, finally under Prometheus!
Native Histograms stability (also with custom buckets!)
More Optimizations!
More UTF-8 support coverage
…and more!

bwplotka

7 of 9

Thanks for listening!

Join Prometheus KubeCon NA sessions!

Check Prometheus blog!

Join us on #prometheus and #prometheus-dev Slack channels!

Join us on social media: @prometheus.io (Bsky) @PrometheusIO (X)

bwplotka

8 of 9

Resources (hidden)

bwplotka

9 of 9

bwplotka