Daniel Dyla

Michael Beemer

​

​

Observable Feature Rollouts

With OpenTelemetry and OpenFeature

Who are we?

Michael Beemer

Senior Product Management, Dynatrace

​

• Experience as a Consultant, DevOps Engineer, Software Developer, and Product management
• Active open source contributor
• Co-founder of OpenFeature and member of Governance Committee

Daniel Dyla

Senior Product Architect, Dynatrace

​

• OpenTelemetry Governance Committee
• Maintainer of OpenTelemetry JS
• Contributor to OpenFeature

Standardizing Feature Flagging

An open specification that provides a vendor-agnostic, community-driven API for feature flagging that works with your favorite feature flag management tool or in-house solution.

What’s a feature flag?

Feature flags are a software development technique that allows teams to enable, disable or change the behavior of certain features or code paths in a product or service, without modifying the source code.

Feature flags have many benefits

Coordinate

Reduce risk

Experiment

… but introduce challenges

My environment so complex…

OpenTelemetry to the rescue

What is OpenTelemetry

Basic Telemetry Types

Traces

Events (Logs)

Metrics

Events (Logs)

• A point in time without duration
• Allow for arbitrary data and data types
• Powerful data analytics capabilities
• No collection processing requirements
• Logs are a particular type of event

Traces

• Collection of related spans
• Operation with a duration and timestamp
• Arbitrary data stored as attributes
• Spans linked together in a tree
• Requires propagation of span context
• Essentially 2 events - Start and End

Time

Spans

Metrics

• Numeric data aggregated from a series of events
• Usually original events are dropped
• Usually attributes are more restricted
• Requires keeping state on the client
• Usually requires strict control of cardinality
• Possible to generate later from events or traces

Telemetry Type Tradeoffs

Unprocessed Data

Aggregate Data

Events

Traces

Metrics

• Data stored in raw form and processed later
• More storage, transport, and server processing cost
• More available analysis options

• Data aggregated before transit and storage
• More efficient storage and transit
• More client processing cost
• Less flexible analysis options

Choosing Appropriate Signals

• How much data am I collecting?
• What types of analysis do I need to do later?
• Is my data structured, unstructured, or unknown?
• Am I collecting numeric data or some other type?

Instrumenting Your Application

Resource identifies your application

Exporters send telemetry to your backend

Instrumentations gather data from common libraries

Instrumenting Your Application

How about an example?

Sneaker shop architecture

Scenario

Response times are reasonable

Scenario

As load increases….

response time become worse.

Drilling into a trace

The database is the culprit.

The DB is the bottleneck

Let’s add read replicas

Here’s the plan

Put access to the read replica behind a feature flag

1

2

3

4

5

Enable the read replica for a small number of users

Analyze the impact

Enable the read replica for everyone

Remove the feature flag

Adding the feature flag

Identifies the feature flag

Controls the database connection

Evaluation context

Collect telemetry

Monitors flag evaluations with OpenTelemetry events

Enabling feature

Matching flag key

Enabling for 25% of sessions

Starting the rollout

Uh no! That shouldn’t happen.

Abort!

That was close…

Back to normal.

Let’s see what went wrong

Failure rate by flag variant

Aggregate error messages

Something is wrong with node 3

We can drill into a representative trace to confirm.

Confirming the diagnosis

It’s definitely node 3

We know what happened

Let’s try that again…

Phase: Read replica disabled

Phase: Read replica enabled for 25%

Phase: Read replica enabled for 50%

Phase: Read replica enabled for 75%

Phase: Read replica enabled

Recap

Rolled out an important performance fix

1

2

3

4

5

Controlled the impact of unforeseen problems

Validated assumptions

Rolled the new feature out for all users

Continuously monitor impact

Questions?

Feedback Appreciated