1 of 40

Introduction to Distributed Tracing

Suraj Nath

Sr. Software Engineer, Grafana Labs

twitter: @electron0zero

2 of 40

Poll: Are you familiar with distributed tracing?

3 of 40

Poll: What are you using for tracing?

4 of 40

5 of 40

6 of 40

7 of 40

8 of 40

example call graph

9 of 40

What is Tracing?

10 of 40

Distributed Tracing is like cross service call-graph

11 of 40

Let’s learn some terms related to Tracing

12 of 40

Trace

13 of 40

Span

14 of 40

Context Propagation

15 of 40

16 of 40

Why do we need tracing?

17 of 40

Modern services are complex

Modern apps are powered by multiple microservices with many endpoints, databases, caches.

18 of 40

Logs and metrics are half the story

Hard to build the timeline of events with only logs and metrics

19 of 40

More detailed view into data transactions

Find dependencies, durations and bottlenecks across services.

20 of 40

So - how do I get started?

1

Instrument

Store

3

Query

4

Collect

2

21 of 40

Step 1 - Instrument your application

Many ways to instrument

Open Telemetry
Jaeger
Zipkin

Many Languages have auto Instrumentation in client SDK

Instrumentation can sometimes be the one the most complex parts of tracing. As mentioned before, the objective here is to measure processes and log the spans generated from them.

There are different ways of doing instrumentation. These are manual and auto instrumentation.

Manual instrumentation requires you to open the code of your application and inject additional code in order to log the spans. For this you can use many libraries for this, such as Jaeger and Zipkin, or you can use something more generic like OpenTelemetry, which is designed with the purpose of abstracting what tracing backend is used. OpenTelemetry has a number of libraries and SDKs that help with instrumenting applications in multiple different languages.

Also, you can use auto-instrumentation which automatically will inject the necessary tracing code to log spans. The usage and viability of auto-instrumentation is generally dependent on the ecosystem used. For example, auto-instrumentation is very common with Java ecosystems, because it relies on a VM to its execution, so the library can easily inject the tracing code.

22 of 40

Step 2 - Set up your tracing pipeline

A Pipeline / Agent / collector provides:

Buffering of spans for efficient data transmission
Ability to sample, enrich and control the flow of traces

23 of 40

Distributed Tracing (with Grafana Agent!)

So, let’s quickly go back to the our diagram of a service exporting trace spans to the tracing backend. We can use Grafana Agent instead to collect all of these spans, collate them together, pre-process them and apply additional functionality to them, and to remote write them onto the final tracing store (in this case, Grafana Cloud Traces which is running in the cloud).

Because we have a lightweight agent running here, we don’t need to setup the likes of Grafana Enterprise Traces or Tempo ourselves, and can concentrate on the running of our own services without having to run that as well.

As we’ve said, Grafana Agent isn’t just a data forwarder and gateway, as it includes a set of powerful integrations that allow manipulation of tracing and span data as part of a local pipeline.

In the case of tracing there is a lot of added functionality that can pre-process your traces, or create additional observability data based on these traces.

Grafana Agent can ingest data in all the formats that GET can
Remote writes tracing data compressed on the wire in OTLP format for fast, efficient transmission to the tracing backend
Grafana Agent uses OpenTelemetry based tail-sampling policies to only send traces and spans that adhere to these policies
Autologging can be enabled, so that service code doesn’t need to explicitly emit logs that include trace IDs
Prometheus metrics can also be sent to a GEM/Cloud Metrics instance, giving you RED signals for incoming trace spans
Prometheus style service discovery can also be used to find matching hosts and apply new key-value pairs to spans from those hosts
Grafana Agent also facilitates the building of GET service node graphs
Cluster load balancing is available, should multiple agents be operating, to ensure that spans are always attached to the right trace regardless of which agent receives them
And finally, there’s the ability to run multiple Tempo instances per agent, which can be useful for policy spreading

And of course, it also ingests metrics by scraping any Prometheus compatible service for GEM, and logs for GEL.

24 of 40

Step 3 - Set up your Tracing Store

Tracing Store / Database will store your Traces,

…and provide ways to query the traces

25 of 40

Step 4 - Visualize

26 of 40

Basics are covered, now what?

When do you need Distributed Tracing?

27 of 40

Distributed Tracing with Grafana Stack

Agent + Tempo + Grafana

28 of 40

Run it yourself with Open Source

29 of 40

Run it yourself with Enterprise

Grafana Enterprise Traces

Grafana Enterprise

30 of 40

Let us handle it with Grafana Cloud

Grafana Cloud

Traces

Grafana

Cloud

31 of 40

Running Tempo

32 of 40

Running in Scalable Single Binary mode

33 of 40

Running in microservices mode

34 of 40

Coming soon!

TraceQL +

35 of 40

TraceQL

36 of 40

37 of 40

https://github.com/grafana/intro-to-mlt

38 of 40

Recommended Session from ObservabilityCON 2022

Scale observability with Mimir, Loki, and Tempo

39 of 40

40 of 40

Thank you

Slides: https://suraj.dev/talks

Twitter: @electron0zero