1 of 34

Debugging with OpenTelemetry

August 18th 2023

Tracing - like Logging, just better

2 of 34

A few words about myself

Chief Architect at Torq.io

Love early stage startups 🫶 at my 4^th one atm

Love espressos - 17g in 34g out, 25 seconds.

I love reducing CI build times 😅

Windows Internals and Low Level background

Check out my blog at kostyay.com

3 of 34

Torq.io

Trusted by the the world’s best security teams

Our Background

Comprehensive low code/no code security automation platform that unifies and automates the entire security stack to deliver unparalleled protection and productivity

About Torq

Founded in 2020
Offices in US, Israel, UK, and Spain
Millions of Daily Security Automations

3

4 of 34

Tech Stack

100% GoLang
Microservice architecture running on GKE

20-30 services
10s of deployments per day to production

gRPC protocol internally, Frontend uses gRPC-Web (moving to connect.build)
Frontend VueJS
Databases:

Postgres
Redis
Cube for reporting

Heavy PubSub users
Light Cloud Functions users
Observability:

Metrics: Prometheus/Thanos
Logging: only errors (using zap)
Tracing – OpenTelemetry ❤️: Errors + Slow Requests

Feature Flags: LaunchDarkly

A few words about our tech stack,.

We use golang extensively.

Use the microservice architecture – with not too many nor too little services ☺

We have a full CI/CD pipeline, engineers are rolling out code to production 10s of times per day.

We have extensive testing in place featuring unit, integration, and end to end tests assuring nothing breaks on the way to production.

We use gRPC protocol as both our internal protocol and external protocol. Our frontend even uses gRPC-web to talk to the API Gateway.

Big fans of Postgres, Redis and use abit of Cube for our reporting views.

Since we are running on Google Cloud Platform we use a lot of the Google’s tech. Specifically Pubsub is our queue of choice as most of our flows are asyncroneous.

Pubsub gives us the peace of mind that no matter what the traffic is, it will scale and handle anything our customers can throw at it.

As far as observability goes we use Prometheus for metrics, For logging we try to log as little as possible as Tracing is ”debugging” technology of choice. We mostly log error logs.

Feel free to approach me after the talk if any of these technologies interest you or you are facing any challenges using any of them.

5 of 34

How do you debug applications?

6 of 34

We can debug with..

Locally, an interactive debugger…
Remotely, by using built in observability signals

In a properly instrumented application you should be able to understand a problem without having to add more instrumentation.

Normally we are faced have to debug in several scenarios

During feature development or testing when it’s simple to use an interactive debugger
When our application is running in production: This is when it gets more tricky. In that case ,its usually not possible to use an interactive debugger so we have to rely on other tools to understand the root cause. Those are signals that we added into the application in advance.
The more signals you have built into your application the easier time you will have when there is a production incident you have to investigate.

Often times you will not have enough signals.
What can you do then? Well if this problem is easily reproducible you may have to do some back-and-forth process of adding more signals and rolling them out to understand the issue.

Logs are the most common and the most popular means of “debugging” and can be used both during development and in production during incident.

7 of 34

Logs are great, but..

8 of 34

Shortcomings of logs

Never there when you need them 😅
Not ideal for understanding end to end processes

9 of 34

What is Distributed Tracing

10 of 34

Distributed Calculator

1

2

3

4

5

6

Service 1

Service 2

11 of 34

span

12 of 34

Distributed trace

A view on the lifespan of a request

A collection of spans

An excellent tool to debug production issues

13 of 34

OpenTelemetry

14 of 34

What is Open Telemetry?

CNCF project - 2nd popular and actively maintained
Observability framework - Tracing, Metrics, Logging
Current standard for Tracing
SDKs + Tools
Cloud centric

OpenTelemetry reference architecture

(source: Documentation)

OpenTelemetry is a CNCF observability framework designed to collect, process, and analyze telemetry data (traces, spans, metrics, and logs) from distributed systems.
It’s goal is to provide a set of standardized vendor-agnostic SDKs, APIs, and tools for ingesting, transforming, and sending data to an Observability back-end (i.e. open source or commercial vendor).
This is the current standard for distributed tracing.

App can be your app, or some other layers along the way, your cloud provider may add metadata or spans along the way.
OpenTelemetry doesn’t have the backend for ingesting the spans, just the tools to for collecting them and sending them to a storage engine supporting the protocol.

There are multiple backends supporting it both open source and also all the big cloud providers support it as well.

Tracing is the only stable and full implementation at the moment with metrics and logging lagging behind.

15 of 34

Tracing basics

Trace: End-to-End process in your application. Contains spans.
Spans: “call” in a trace

Attributes: key/value pairs; tags; metadata
Events: named strings
Parent: Previous span that encapsulates this one

Sampler: always, probabilistic, errors only, etc.
Exporter: OTLP, Jaeger, Prometheus, Cloud Provider.

16 of 34

Why should you trace?

Improve the observability of your software
End to end visibility across distributed systems
Understand service to service dependencies
Identify performance bottlenecks

17 of 34

1 - Initialize a Tracer in Go

otel.Tracer(“MyTracer”)

Rate

Service Metadata

Destination

18 of 34

2 - Create a Span

Create a span

End a span

Record Errors

19 of 34

Trace Visualization

20 of 34

Trace Viewer - Google Cloud

Traces

Spans

21 of 34

Span Attributes

Span Tree

Attributes

22 of 34

Linked Logs and Events

23 of 34

CloudSQL – Query Insignts Integration

24 of 34

25 of 34

Adopting Traces

26 of 34

Adopting traces in your team

Enable auto-instrumentation – free tracing
Add trace-id to structured logs
Add middleware to your common libraries (clients and servers)
Experiment with the SDK
Write the code: Start with service boundaries and 3rd party calls
Be an advocate in your team - Tracing makes solving production issues much easier

Some tools you are probably already using have tracing support built in out of the box.

It can be your Kong api gateway, if you have some services in other programming languages besides go there is usually instrumentation built in into frameworks like Spring or some Python libraries also support it out of the box.

Your next step is to try adding community maintained middleware to the go modules you are using. Start with service boundaries, put tracing MW in your HTTP server, in your database client, and so on. You will start seeing value right away.

Your next step after that is to experiment with the SDK. As you have seen on the previous slide the SDK is very simple to use. Its basically just 3 function calls that you need to remember.

Start adding spans in your applicative logic. Do you have an algorithmic calculation that may be taking a long time? Wrap it with a span, add some of its parameters to the span attributes.

Further integrate spans into your life cycle. If you are already a heavy logs user it may be hard to switch to traces right away. You can however link your existing logs to traces by adding a traceId to each log.

This way, with some small wiring your log collector should be able to integrate with the traces you are creaing. In GCP its works almost seamlessly.

If you are starting to see value from traces talk about them in your team, get them excited.

27 of 34

Using Middleware - The key to adoption

OpenTelemetry has a big community.

There is tracing support for many popular Go packages.

28 of 34

Example: Tracing in mux

> go get go.opentelemetry.io/contrib/instrumentation/github.com/gorilla/mux/otelmux

29 of 34

Use Traces in Tests

30 of 34

Trace based E2E tests – tracetest.io

Source: https://opentelemetry.io/blog/2023/testing-otel-demo/

31 of 34

32 of 34

Async Calculator

REST API

33 of 34

Presentation + Code Samples

34 of 34

Thank you

August 18th 2023