1 of 17

Splunk Office Hours

Splunk Application Performance Monitoring

January 14, 2025

Splunk Corporate Template | TMPLT-FY25-101

2 of 17

Forward-�looking statements

This presentation may contain forward-looking statements regarding future events, plans or the expected financial performance of our company, including our expectations regarding our products, technology, strategy, customers, markets, acquisitions and investments. These statements reflect management’s current expectations, estimates and assumptions based on the information currently available to us. These forward-looking statements are not guarantees of future performance and involve significant risks, uncertainties and other factors that may cause our actual results, performance or achievements to be materially different from results, performance or achievements expressed or implied by the forward-looking statements contained in this presentation.

For additional information about factors that could cause actual results to differ materially from those described in the forward-looking statements made in this presentation, please refer to our periodic reports and other filings with the SEC, including the risk factors identified in our most recent quarterly reports on Form 10-Q and annual reports on Form 10-K, copies of which may be obtained by visiting the Splunk Investor Relations website at www.investors.splunk.com or the SEC's website at www.sec.gov. The forward-looking statements made in this presentation are made as of the time and date of this presentation. If reviewed after the initial presentation, even if made available by us, on our website or otherwise, it may not contain current or accurate information. We disclaim any obligation to update or revise any forward-looking statement based on new information, future events or otherwise, except as required by applicable law.

In addition, any information about our roadmap outlines our general product direction and is subject to change at any time without notice. It is for informational purposes only and shall not be incorporated into any contract or other commitment. We undertake no obligation either to develop the features or functionalities described, in beta or in preview (used interchangeably), or to include any such feature or functionality in a future release.

Splunk, Splunk> and Turn Data Into Doing are trademarks and registered trademarks of Splunk Inc. in the United States and other countries. All other brand names, product names or trademarks belong to their respective owners. �© 2024 Splunk Inc. All rights reserved.

Splunk Corporate Template | TMPLT-FY25-101

3 of 17

Agenda

1. Introduction

2. Pre-submitted Questions
3. Open Q&A
4. Additional Resources

Don’t forget to join the #office-hours Slack channel!
Request access here: http://splk.it/slack

Splunk Corporate Template | TMPLT-FY25-101

4 of 17

Office Hours Intro

Safe and open space to ask questions (post in the chat or raise hand to come off mute).

Casual, informal discussion around best practices, tips & tricks.

These are all about YOU. Get help from Splunk experts.

Ask Questions

Discuss Ideas

Get Help

3 sessions per month, 1 per product area (Platform, Security, O11y).

Splunk Corporate Template | TMPLT-FY25-101

5 of 17

Your Splunk Experts

Brad Ishida

Jeremy Hicks

Sr. Solutions Architect

Staff Observability Strategist

Splunk Corporate Template | TMPLT-FY25-101

6 of 17

Do you have any tips for our APM beginners?

Splunk Corporate Template | TMPLT-FY25-101

7 of 17

Question #1

Can we get some tips to get started with mapping APM data to dashboards/alerts?

(Solution from 10:18-16:43 of the recording)

Solution

Yes, I'd recommend learning all about monitoring metricsets and the ones you get with APM.
Monitoring MetricSets are exactly the type of data used to build dashboards and alerts. If you have some APM data flowing in, your "APM Services" built-in dashboard will be populated. If you open up any of those charts in the "APM Services" dashboard, you'll see some good examples of monitoring MetricSet data being used and filtered for dashboards. You can also create an alert directly from any of those charts by clicking the "alarm bell" icon.

Documentation:

https://docs.splunk.com/observability/en/apm/span-tags/metricsets.html

Splunk Corporate Template | TMPLT-FY25-101

8 of 17

Question #2

What are AutoDetect detectors and how can I use them?

(Solution from 16:50-21:24 of the recording)

Solution

AutoDetect detectors are detectors that come out-of-the-box for common use-cases. You don't have to build them and you can simply subscribe to them to receive alerts from them.
A great example is to navigate to Infrastructure and then choose EC2 or Hosts and you'll see that the charts for CPU, Memory, and Disk all have colored borders indicating health. Those charts with colored borders have AutoDetect detectors linked to them.
You can also navigate to "Detectors & SLOs" -> Detectors to see a list of all detectors. There will be an icon that says "auto" next to the AutoDetect detectors and you can click through to "Manage Subscriptions" on each one that you want to subscribe to.

Documentation:

Splunk Corporate Template | TMPLT-FY25-101

9 of 17

Question #3

What are the differences between APM and AppDynamics and where should I use each?

(Solution from 21:45-27:05 of the recording)

Solution

The difference between Splunk APM and AppDynamics has a short answer and a longer answer. The short answer is that AppDynamics is really good for monitoring monolithic traditionally architected applications. And, Splunk APM is really good for monitoring applications that run as microservices.
The longer answer considers that there is nuance in the coverage that each product provides and the 2 products do technically provide some overlapping functionality. For example, it may be a thoughtful approach to use Splunk APM to monitor a traditional application if I know that application is on a modernization journey where it will be re-architected to use microservices in the near future. Exploring a right-fit solution is actually a great plug for the type of team I work on--the Solutions Architects. We can help you choose a path that makes the most sense and helps you avoid wasting time and money.

Resources:

Splunk Corporate Template | TMPLT-FY25-101

10 of 17

Question #4

What are some tips for setting up deployment environments?

(Solution from 27:07-30:38 of the recording)

Solution

Making use of deployment environments in Observability Cloud may be as simple as including the name of your environment with your telemetry.
For example, let's say I'm monitoring MyApp in both UAT and PROD. I will want to include a "deployment.environment" tag in my application instrumentation.
Then, in Observability Cloud, I can leverage that tag to breakdown views in places like the APM Service Map and Tag Spotlight. Breaking down MyApp by "deployment.environment" would allow me to view errors that are isolated to specific environments and hopefully avoid deploying bad code to PROD because I can easily see that MyApp is failing in UAT.

Documentation:

https://docs.splunk.com/observability/en/apm/set-up-apm/environments.html

Splunk Corporate Template | TMPLT-FY25-101

11 of 17

Question #5

What's a good way to introduce the idea of performance monitoring to my devops team?

(Solution from 30:50-38:08 of the recording)

Solution

Application Performance Monitoring (APM) can provide a number of benefits for a DevOps team.

APM tracing at its most basic provides your team with auditability of the calls and interactions happening within their own system. This is incredibly helpful when establishing where something broke or when an error happened in any given interaction without digging through logs to piece together an audit trail.
Additionally APM can provide you with 3 or more of your 4 Golden Signals (Latency, Errors, Traffic, and Saturation) or RED metrics (Rate, Errors, Duration) for a given service, endpoint, etc. and very quickly get monitoring enabled.

Documentation:

https://lantern.splunk.com/Observability/Product_Tips/Observability_Cloud/Adopting_monitoring_frameworks_-_LETS?_gl=

Splunk Corporate Template | TMPLT-FY25-101

12 of 17

Question #6

Can you give us a list of your top (i.e. most valuable) metrics to monitor with Splunk APM?

(Solution from 38:13-42:21 of the recording)

Solution

For valuable APM metrics, be sure to check out the doc below that explains Troubleshooting MetricSets and Monitoring MetricSets.

But if you want a specific example to get started, give some special attention to "service.request". If you navigate to any of the out-of-the-box APM dashboards and open the charts, you'll see variations of "service.request" used often and many of the charts are "service.request" metrics with variations of filters. For example, "service.request.count" with filters for a specific application (sf_service) that are flagged containing errors (sf_error is true) will give you the error rate for that application.

Documentation:

https://docs.splunk.com/observability/en/apm/span-tags/metricsets.html

Splunk Corporate Template | TMPLT-FY25-101

13 of 17

Question #7

How do you set up sending a VSAM File to Splunk as raw data?

Solution

This question is off topic, but we wanted to steer you in the right direction.

A VSAM file that you would get from a mainframe--this is a good use-case for Splunk Cloud or Splunk Enterprise--or what we refer to as the core logging platform. Splunk is able to ingest any human-readable logs so you would need to execute a few steps to ingest a VSAM log file. Generally speaking, you would export the VSAM data using a utility such as IDCAMS to export the data to a flat file. Then you would use a tool like FTP or IBM Connect:Direct to transport the flat file to an accessible location where it can be read by a Splunk Universal Forwarder and ingested into Splunk.

Splunk Corporate Template | TMPLT-FY25-101

14 of 17

Live Q&A

Raise your hand or post your question in the chat!

Q1: Does APM interact w/ ITSI?

A: The short answer is Yes! The longer answer was discussed in detail starting at 45:35 of the recording.

Q2: Do you have something similar to AWS Gameday event in Splunk Observability ?

A: Splunk used to run a “capture the flag” type experience called Boss of Ops and Observability (BOO), but it is not currently something we are doing. We do run workshops that you can set up through your account team and we are working on standing up some Observability workshops that are open to all customers. Stay tuned!

Q3: Do you have a demo on the Baggage and how would you use it (https://opentelemetry.io/docs/concepts/signals/baggage/). Do I even need to use it, if I use tags today ?

A: Solution starts at 49:42 of the recording.

Q4: When an issue is reported by an application team on a service, what do you feel is the best way to get started digging into an issue with APM?

A: Solution starts at 52:19 of the recording.

Splunk Corporate Template | TMPLT-FY25-101

15 of 17

Upcoming Observability Events

Tech Talk

Office Hours

Tech Talk

Come join us!

3-2-1 Go! How Fast Can You Debug Microservices with Observability Cloud?

Tuesday, February 25

11AM PT/2PM PT

Register here

Debugging Microservices

Tuesday, March 18

1PM PT/4PM ET

Registration coming soon!

AI Assistant in Observability Cloud

Tuesday, March 25

11AM PT/2PM PT

Registration coming soon!

Splunk Corporate Template | TMPLT-FY25-101

16 of 17

Additional Resources

When in doubt, check:

Splunk Answers: Observability Cloud: ask questions and get answers from our Community of Splunk enthusiasts.

Splunk Docs: APM: technical documentation on Platform capabilities and functionalities.

“What you can do.”

Lantern: how-to guidance and use cases for your specific industry, data source, or data type.

“What you should do.”

eLearning courses

Introduction to Splunk APM - A FREE 15-minute course for Developers, SRE's, and Architects to explore Splunk APM's key features, navigation and basic troubleshooting.
Using Splunk APM - This 4.5-hour virtual course, targeted to developers and DevOps, helps you use Splunk APM to analyze traces, and troubleshoot and monitor your microservices-based applications. $500.

OnDemand services (full APM catalog): recommended services

Create Custom Span Tags (5 credits): Assist with the creation of up to 5 Custom Span Tags to address 1 of the following: Query and filter Traces OR Provide additional information for each operation when inspecting the Spans of a Trace during troubleshooting
Assist with Automatic Instrumentation: (10 credits): Assist with auto-instrumentation of an application using 1 of the Splunk APM supported libraries, including: Java, Scale, Kotlin, Python, Ruby, Node.js, Golang, PHP, .NET Core and .NET Framework.

How-To Videos

Splunk Observability for Engineers YouTube Playlist

AppDynamics

AppDynamics resource homepage

.conf24 sessions (watch them OnDemand)

Splunk APM session list - check out the 5 great APM-focused sessions from .conf24, covering best practices, customer success stories, and more!

Lantern Guides

Splunk Corporate Template | TMPLT-FY25-101

17 of 17

Thank you

Register for upcoming Splunk Community Office Hour s

Splunk Corporate Template | TMPLT-FY25-101