Observability in Production
with
OpenTelemetry
presented by Ron Nathaniel
Apache 2.0 Licensed
Quick Intro
Name: Ron
Pronounced like: � Rhône
Saône
I love Python
github.com/ronnathaniel
Solutions Architect
Emerging Tech
& Incubation
OpenTelemetry
(Open Source CNCF Project)
Our Agenda
Our Agenda
What is Observability?
Our Agenda
What is OpenTelemetry?
Observability vs OpenTelemetry?
Our Agenda
How can we add OpenTelemetry
to your applications today?
Our Agenda
Where does all this data go?
Observability Basics
What is Observability?
Observability
is a measure of how well
internal states of a system�can be inferred from knowledge
of its external outputs
Observability
is a measure of how well
internal states of a system�can be inferred from knowledge
of its external outputs
Observability
is a measure of how well
internal states of a system�can be inferred from knowledge
of its external outputs
Can we observe applications?
Can we observe applications?
…What is an application?
Monoliths?
Monoliths?
Monoliths?
Monoliths?
Monoliths?
Microservices on the Cloud
Challenges
A
Challenges
Observe
B
Challenges
Troubleshoot
C
Challenges
Optimize
D
Telemetry Data
What is Telemetry Data?
Three Pillars of Observability
Metrics
Logs
Traces
Three Pillars of Observability
Metrics
(What?)
Logs
Traces
Three Pillars of Observability
Metrics
(What?)
Logs
(Why?)
Traces
Three Pillars of Observability
Metrics
(What?)
Logs
(Why?)
Traces
(Where?)
MeLT
Trace = Family
Trace = Family
Span = Person in
the Family
Let’s implement Observability
Let’s implement Observability
An Instrumentation framework.
Let’s implement Observability
An Instrumentation framework.
And a place to send the data.
Let’s implement Observability
An Instrumentation framework.
And a place to send the data.
And a way to visualize it.
Introducing…
OpenTelemetry
OpenTelemetry
A collection of tools, APIs, and SDKs.
Used to instrument, generate, collect, and export telemetry data.
OpenTelemetry
A collection of tools, APIs, and SDKs.
Used to instrument, generate, collect, and export telemetry data.
OpenTelemetry
A collection of tools, APIs, and SDKs.
Used to instrument, generate, collect, and export telemetry data.
Architecture
Source: grafana blog
*
Language SDKs
Two Methods
Language SDKs
Core Repo
(API and SDK)
Two Methods
Language SDKs
Contrib Repo
(automatic instrumentations)
Core Repo
(API and SDK)
Two Methods
Automatic
Instrumentation
Otel Collector Components
*
Do We Need a Collector?
OpenTelemetry is
just the Client?
OpenTelemetry is
just the Client?
Yes. For now…
Where can you send your Telemetry Data?
Vendor-Neutral Backends
Vendor-Neutral Backends�(Open Source)
Vendor-Neutral Backends�(Open Source)
TeleTrace
Vendor-Neutral Backends�(Open Source)
Jaeger
Vendor-Neutral Backends�(Open Source)
Zipkin
Vendor-Neutral Backends�(Open Source)
Prometheus
Vendor-Neutral Backends�(Open Source)
stdout / stderr streaming
What are we
looking at?
{
"name": "root",
"context": {
"trace_id": "0x7b21...”,
"span_id": "0x3784e...",
"trace_state": "[]"
},
"parent_id": null,
"start_time": "2022-07-11T20:37:37.498779Z",
"end_time": "2022-07-11T20:37:37.498849Z",
"status": {
"status_code": "UNSET"
},
"resource": {
"telemetry.sdk.name": "opentelemetry",
"telemetry.sdk.version": "1.11.1",
"service.name": "unknown_service"
}
}
Auto Instrument Flask
Hint: Google “OpenTelemetry Flask Instrumentation Python”
Answer: Use FlaskInstrumentor()
What are we
looking at?
{
"name": "/",
"context": {
"trace_id": "0xf06...",
"span_id": "0x2b3e...",
"trace_state": "[]"
},
"kind": "SpanKind.SERVER",
"parent_id": "0x334b5632f9967f45",
"start_time": "2022-07-04T09:51:33.578142Z",
"end_time": "2022-07-04T09:51:33.581726Z",
"status": {
"status_code": "UNSET"
},
"attributes": {
"http.method": "GET",
"http.server_name": "127.0.0.1",
"http.scheme": "http",
"net.host.port": 5000,
"http.host": "<app-name>.glitch.me",
"http.target": "/",
}
}
How automatic is automatic?
< 5 lines of code
< 5 lines of code
Sounds easy…
Too Difficult!
Why is in-code instrumentation too difficult?
Too many Applications (1000s).
Why is in-code instrumentation too difficult?
Might want to remove instrumentation in the future.
Why is in-code instrumentation too difficult?
Engineering doesn’t have the resources.
The Contributors of Otel Listened
No-Code
Automatic Instrumentation
No-Code
Automatic Instrumentation
Command Line
No-Code
Automatic Instrumentation
Kubernetes
No-Code
Automatic Instrumentation
Serverless Functions
No-Code
Automatic Instrumentation
And more every day…
Huge Community
Documentation
GitHub
Slack
SIG meetings
Joining the community
LICENSE
Copyright 2019, The OpenTelemetry Authors
Licensed under the Apache License, Version 2.0 (the "License");
you may not use this file except in compliance with the License.
You may obtain a copy of the License at
http://www.apache.org/licenses/LICENSE-2.0
Unless required by applicable law or agreed to in writing, software
distributed under the License is distributed on an "AS IS" BASIS,
WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
See the License for the specific language governing permissions and
limitations under the License.
Thank you very much.