Introduction to Application Performance Tuning

Sunny Chan

Agenda

Why should we measure application performance?

​

How do we measure?

​

Tools of the trade

​

Pitfalls

​

Demos

Why should We Measure?

• Can your system scale up for new business?
• Can you improve the time it takes for your system to response to client requests?
• Can the new improvement in infrastructure helps your system or save money?

Why should we measure

• For Financial Computing, time is money
• With Electronic Trading, the faster you can trade and response to the market, the more money you can make
• Milliseconds can be the difference between winning or losing millions of dollars

Throughput and Latency

Throughput is the amount of work (e.g. transactions, calculations, data transferred) your system can handle, in a period of time.

Latency is the time it takes for a system to receive the response for a request

​

High Throughput

DOES NOT necessarily means �Low Latency!

​

Other performance characteristics

• Availability
• Uptime
• Efficient use of resources
• Memory Access Efficiency
• Hardware
• Network bandwidth
• Disk / Database usage
• Scalability
• Horizontal Scaling vs Vertical Scaling

Performance Engineering for the system should be an integral part of software development process at all stages.

Benchmarking

Benchmark is the standard or reference program against which allow the system performance is being evaluated and compared.

A good benchmark should:

• Generate/simulate workload as close to production as possible
• Should be repeatable and verifiable against changes with the system
• Produce defined measurements, can be used to measure improvements
• Consider whether the measurement is meaningful
• The relationship between latency and load
• Average Latency vs. 90%, 95% and 99%ile

Highly Recommended talks from Gil Tene, CTO of Azul - he has plenty of talks on this topic

​

Beware of Microbenchmarking

Beware of Microbenchmarking

public class Mutlplication {� public static void main(String args[]) {� Random rand = new Random();� int numberToMultiply = 1000;� final double x = rand.nextLong();� final double y = rand.nextLong();� long start = System.nanoTime();� for (int i=0; i<numberToMultiply; i++) {� double v = x * y;� }� long timeTaken = System.nanoTime() - start;� System.out.println( "Multiplying"+numberToMultiply+" integers took "+timeTaken+"nanos");� }�}

​

Ways to make correct measurements

• Use a large number of samples
• Increase the number of the work cycles so that JIT compiler would kick in�
• Warm up your JVM before tests�
• If possible, use real world workload and input.
• Make it as unpredictable for the JVM as possible
• Replay production data�
• Use proper tools for your benchmarking
• If you must write microbenchmark, use JMH (https://github.com/openjdk/jmh)

How to find performance issue in your application

• Now that you have a benchmark, you can use it to exercise your code and find where the performance bottlenecks is
• You might have observed performance issue in production
• JVM has lots of performance tools built-in and third party tools
• Java Flight Recorder
• Async Profiler
• Instrumentation
• Logging
• Metrics
• Distributed Tracing

Types of profiler

Sampling Profiler

• Sample the state of your application at certain point in time
• Low overhead, inaccurate
• Ideal for general investigation and initial observation
• Examples
• Java Flight Recorder
• Async Profiler

Instrumentation Profiler

• Insert code (manually or automatically) into your application to generate measurement
• Higher Overhead, more accurate
• Ideal for in depth investigation for specific performance issue
• Examples:
• JProfiler
• Distributed Tracing
• Spring AOP profiling

​

Heisenberg Uncertainty Principle

The uncertainty principle, also known as Heisenberg's indeterminacy principle, is a fundamental concept in quantum mechanics. It states that there is a limit to the precision with which certain pairs of physical properties, such as position and momentum, can be simultaneously known. In other words, the more accurately one property is measured, the less accurately the other property can be known.

​

Java Flight Recorder

• Built into Java 8 and above�
• JVM has been modified to generate events
• E.g. GC, memory allocation, sampling execution�
• Events are stored in memory (“Flight recorder”)�
• Can selectively turned on different Events�
• Low overhead, can be used in production for a short period of time�
• Useful functionality:
• Low overhead sampling profiler
• Memory Profiling
• GC events collections
• Network and stack collection�
• Application can add custom events�
• Java Mission Control is the main UI for reading JFR recordings�
• Intellij also support JFR recordings

Async Profiler

• https://github.com/async-profiler/async-profiler
• Linux only
• Using perf profile subsystem
• Required extra kernel permissions - may not work well in Kubernetes or Docker
• Low overhead sampling profiler
• Can collect stack trace, heap allocation profiling
• Work around the Safepoint bias
• Can generate flame graph for your application

Logging, Metrics and Tracing

• Logging helps with post mortem investigation by recording states in the text statements
• Allow for searching and pattern
• Metrics allow you to insert instrumentations into your application
• Measure specific frequency / time used for specific part of code
• Allows for capacity planning and alerting
• Distributed Tracing allows you to track processing across multiple machines

Tools that you should be aware of:

Prometheus, Elastic, OpenTelemetry, Micrometer metrics, Zipkins

Three Key Takeaways

Performance Measurement allows you to make sense of the throughput and latency of your application

“Fast” is a loaded word in performance world - be careful!

A good benchmark is an important tool to help you fix performance issue before production

Be aware of the pitfall!

We have many Performance measuring tool built into Java platform - use them wisely!

Beware of the overhead of collecting measurement

Thank you

Linked-in