OpenTelemetry Support in Nephio

OpenTelemetry

An observability framework and toolkit designed to facilitate the

• Generation
• Export
• Collection

of telemetry data such as traces, metrics, and logs.

It’s only concerned with the generation, modification and transmission pipelines of the system, and not with long-term storage.

Architecture layout

Nephio Management Cluster

Service Orchestration

GitOps/CICD

Infrastructure

Network Functions

Collector

Observability frontend

Trace DB

Metrics DB

Logs DB

​

• Collector collects telemetry info from various endpoints (i.e. service orchestration, GitOps, NF etc)
• Collector stores data into DB for later retrieval
• Observability frontend present info as a UI

OpenTelemetry ambitions in Nephio

1. Opentelemerty signals generation by components that are developed or used by Nephio
1. This includes things like metrics and traces for end-to-end flows for Nephio operators through to Porch and to the Git and Gitops engine to the NFDeployment operators
2. Opentelemetry data collection and visualisation
• This includes collection of NF and component data and end-to-end visualisation

​

See https://github.com/nephio-project/nephio/issues/513 for a more detailed breakdown

Status quo�Kubernetes Operator OTEL signal generation / Logging

​

None of the operator frameworks used in Nephio (controller-runtime, kopf) support emitting OTEL compatible logs.

The OTEL collector can be deployed as a sidecar or as a daemonset that allows for transformation of log entries into OTEL compatible logs (some enrichment can be done based on the k8s pod context).

What has been tested

Deployment of otel-collector configured to capture logs of porch-server and porch-controllers (but not limited to these) e.g. /var/log/pods/porch-system_porch-server-*/porch-server/*.log

Logs can be exported to other otel-collectors or any relevant databases (elasticsearch has been tested)

A kpt package for otel-collector is available here: https://github.com/constanton/nephio-otel-collector/tree/main

But, it is also possible to deploy otel-collectors and auto-instrumentation (if supported) using the OTEL Operator: https://opentelemetry.io/docs/platforms/kubernetes/operator/

Status quo �Kubernetes Operator Otel signal generation / Metrics

None of the operator frameworks used in Nephio (controller-runtime, kopf) support emitting OTEL compatible metrics. They support Prometheus compatible metrics.

Additionally kube-state-metrics can be used to gather metrics that are computed from status.

Comparison between OTEL and Prometheus metrics from a producer’s standpoint: 1. Otel metrics work on a push model, while Prometheus works on a pull model

2. OTEL allows for delta based metrics, while Prometheus doesn’t allow it by design

OTEL Collector can be deployed to scrape metrics from Prometheus compatible endpoints, and transmit them into an OTEL pipeline.

So far, the only attempt for porch performance metrics is an external go script that creates a Prometheus endpoint: https://github.com/nephio-project/porch/tree/main/test/performance

Status quo�Kubernetes Operator Otel signal generation / Traces

None of the operator frameworks used in Nephio (controller-runtime, kopf) support emitting Otel compatible traces.

Kube-apiserver and kubelet supports generating traces and spans synchronous requests. They do not accept traceparent header.

There was a KEP around distributed tracing for async parts of kubernetes core, but it was never implemented: https://github.com/kubernetes/enhancements/pull/650

There were some proposals from the CAPI project around this in 2020 to add tracing into controller-runtime based on the above KEP, but it was never finished: https://github.com/kubernetes-sigs/controller-runtime/pull/1211

​

​

Status quo �Gitops engine Otel signal generation / Traces

ArgoCD doesn’t support emiting traces.

FluxCD will emit traces and spans in the upcoming 2.7 release, but doesn’t support injecting a traceparent, and doesn’t push the traces or the spans downstream. See: https://github.com/fluxcd/flux2/pull/5321

​

Status quo �Conclusions

The Kubernetes operator ecosystem is capable of emiting signals that can be turned into Opentelemetry logs and metrics. Controller-runtime for example has a well established set of metrics to monitor. Additionally, metrics for Kubernetes resources can be collected via kube-state-metrics.

The Kubernetes operator ecosystem is not prepared to handle tracing correctly, and while there are some pockets of implementation for local tracing, it doesn’t help with end-to-end executions, and focused on the component’s own behavior.

There are some hard to solve problems in integrating the interaction model of Kubernetes that allows for multiple actors to change different parts of the same resource, making trace and span management quite complicated. For example if 2 flows modified the same resource, then does the reconciliation loop need to report actions to both of them? Do all subsequent reconcile loops need to report to the same trace? Even if the loop was a noop? What happens if a flow that initaially modified a resource has no active parameters in the .spec of a resource? Should the trace be not reported on anymore?

Proposed nephio impacts

1. Expose logs and metrics from all components to be in line with the other parts of the ecosystem. – Logs can be collected via sidecars, but metrics are not available from most components.
2. Create metrics that are component specific eg. porch git interaction timings
3. Create metrics that are usecase specific, eg. deployment time of a OAI function
4. For synchronous request handling, expose metrics, traces and be willing to receive and pass trace context. This is especially important with for adding instrumentation to the porch clients that manipulate packageRevisions and packageRevisionResources. This would allow for easier understanding of the operations of complex flows like nephio-controllers interactions during free5gc deployments.

​