Debugging as a Science

CSE 598 – Applied Program Analysis and Debugging

Fall 2025

​

Fish Wang

Arizona State University

How I Solved The First Module

2

Debugging Your Program

• Observing an unexpected behavior
• “My program is giving me a wrong result.”
• Asking the right question
• “Why isn’t my program deleting the right file?”
• Making hypotheses
• “This variable should hold that value…”
• Examining the root cause
• “Oh this is why!”

3

4

The Very First Step

• Noticing the problem – something is wrong!
• Crashes
• Slowness
• Weird behaviors

Explorer.exe Handle Leaks

• Huh…

​

• Huh?

​

​

​

• Huh!

6

D:\> handle64.exe -p explorer.exe -s -nobanner

Handle type summary:

…

Key : 1780

…

Total handles: 9744

Explorer.exe Handle Leaks

• Huh!
• Hundreds of repeating entries

7

explorer.exe pid: 19428 type: Key 9C14: HKLM\SOFTWARE\Microsoft\Windows\CurrentVersion\Explorer\CommandStore

explorer.exe pid: 19428 type: Key 9C1C: HKLM\SOFTWARE\Microsoft\Windows\CurrentVersion\Explorer\CommandStore\shell\Windows.SortGroupsDescending

explorer.exe pid: 19428 type: Key 9C40: HKLM\SOFTWARE\Microsoft\Windows\CurrentVersion\Explorer\CommandStore

explorer.exe pid: 19428 type: Key 9C48: HKLM\SOFTWARE\Microsoft\Windows\CurrentVersion\Explorer\CommandStore\shell\Windows.Cancel

explorer.exe pid: 19428 type: Key 9C98: HKLM\SOFTWARE\Microsoft\Windows\CurrentVersion\Explorer\CommandStore\shell\Windows.SortGroupsAscending

...

What Tools Should You Use?

• printf() debugging
• A poor-man’s single step
• But it can be useful at times!

​

• Debuggers are not always available
• Debugging multithreaded programs is difficult
• Debugging asynchronous programs is difficult

8

What Tools Should You Use?

• printf() debugging (cont.)
• Q: Where should we printf?

​

• Binary search ftw
• Print at the location that a discrepancy is most likely observed

​

• Ughhhhhhhh now I need to decide between the two!
• I usually prefer binary search (when possible), because humans can be (overly) confident

9

What Tools Should You Use?

• Debuggers
• Many features for you to debug your programs in a fine-grained manner
• Single stepping
• Stepping over
• Conditional breakpoints
• Hardware watchpoints

​

• Q: Where do we set breakpoints?

10

What Tools Should You Use?

• Q: Are these tools good enough? Why or why not?

11

Requirements

• Printf() debugging
• The target program is changeable & recompilable
• The target can be run multiple times
• The bug is triggerable (somewhat) reliably
• Debugger
• The target program is debuggable (attachable, with the right permissions, etc.)
• The target can be run multiple times
• The bug is triggerable (somewhat) reliably

12

Requirements

• Printf() debugging
• The target program is recompilable
• The target can be run multiple times
• The bug is triggerable (somewhat) reliably
• Debugger
• The target program is debuggable (attachable, with right permissions, etc.)
• The target can be run multiple times
• The bug is triggerable (somewhat) reliably

13

Transient Bugs

• Debugging temporary and unreproducible bugs is usually very challenging
• You cannot reboot or rerun a long-running service
• You cannot restart a program since it only runs once

14

Transient Bugs (cont.)

• Debugging temporary and unreproducible bugs is usually very challenging
• The bug is only triggered one in N times
• The bug is difficult to measure or observe
• Slowness
• Multithreading-related issues
• Carefully crafted attacks

15

Other Tools

• Profilers
• Diagnosing performance issues
• Diagnosing bugs that don’t lead to crashes
• Long-running processes

​

• vTune Profiler, Luke Stackwalker, cProfile, vmprof, …

​

• Hot spots
• RPC delays
• Waits

16

Slow Duplicati

• Duplicati is a file backup utility written in .Net

17

https://github.com/duplicati/duplicati/issues/3167

Slow services.exe

• services.exe on Windows 10 would occupy a CPU core for ~20 seconds every N minutes

18

Slow services.exe

19

Slow GTA Online

• GTA Online has a slow JSON parser that counts the entire JSON string N times
• The JSON string is usually more than 10 MB in size
• N is the number of characters in the JSON string

20

https://nee.lv/2021/02/28/How-I-cut-GTA-Online-loading-times-by-70/

Slow GTA Online

21

https://nee.lv/2021/02/28/How-I-cut-GTA-Online-loading-times-by-70/

Slow UAC Dialog on Windows

• Every time the UAC dialog pops up, it would spend ~30 seconds in grey screen before the dialog shows up

22

Slow UAC Dialog on Windows

• Profiler (ETW + Windows Performance Analyzer) pinpoints the culprit for me
• Acronis True Image registers a custom shell overlay to display indicator icons on each file (e.g., is the file backed up)
• The callback function calls CreateToolhelp32Snapshot and Process32NextW to determine if another daemon process is running
• Enumerating processes is slow, and the callback is invoked tens of thousands of times…

23

Slow UAC Dialog on Windows

24

Screenshot credit: https://randomascii.wordpress.com/2025/05/26/acronis-true-image-costs-performance-when-not-used/

Side Note: Reporting The Bug

25

Side Note: Reporting The Bug

26

Other Tools (cont.)

• Logging
• Diagnosing issues on long-running processes
• Diagnosing issues without restarting the process

27

Broken Connections

• The server and the client couldn’t connect to each other all of a sudden
• Realizing that they are connected using OpenVPN
• OpenVPN has logging support
• Did OpenVPN establish connections properly?
• No. Why?

​

28

Broken Connections (cont.)

• Client side
• Incoming TLS packets …
• Connection failed. Retry in 30 seconds…

​

• Server side
• Incoming TLS request …
• Request failure. Reason: CA expired.

​

29

Poor Man’s Logger (Profiler)

• Tracers
• Syscall tracers: strace
• Library call tracers: ltrace, APISpy
• Execution tracers: Intel Pin, Valgrind, DynamoRIO

30

31

Debugging Others’ Programs

• Observing an unexpected behavior
• “Fish’s program is giving me a wrong result.”
• Asking the right question
• “Why isn’t his program deleting the right file?”
• Making hypotheses
• Sadfyawpefspeaf9uqparjw/sKC?S:Ad;jfdsa
• Examining the root cause
• “Oh this is why!”

32

An Implicit Requirement

• When debugging a program, you know
• What the expected behavior is
• Right vs wrong behaviors

​

• Your understanding of the debuggee guides your debugging choices
• Where to printf()
• Where to set breakpoints
• What to profile
• What logs to read

33

An Implicit Requirement

• When debugging a program, you know
• What the expected behavior is
• Right vs wrong behaviors

​

• Your understanding of the debuggee guides your debugging choices
• Where to printf()
• Where to set breakpoints
• What to profile
• What logs to read

34

Understanding A Program

• Q: How?

35

Understanding A Program

• What not to do
• Don’t start by reading source code
• Don’t start from an arbitrary version of source code
• Don’t start from line 0 (or the entry point)
• Don’t strictly follow the code line by line

36

Understanding A Program

• What you should do
• Start by reading documentation, articles, blog posts, issues, user complaints, mailing lists, logging output, etc.
• Make sure you are working on the right target and the right code base
• Read source code by following data flows and logic flows

37

Understanding A Program

• Your top priority
• Solving a bug that you observed
• Remember your goal when you are understanding the program, and always ask yourself, “is this part of logic relevant to my goal?”

38

Educated Guesses

• While all truths are in the source code, understanding a large code base is still extremely difficult and time consuming
• This is when we need to guess

​

• Q: Guess what?

39

Educated Guesses

• High-level logic of the program
• Instead of reading the source code, you can come up with a hypothesis on the program logic, and then verify it by reading or debugging the source code

​

• Low-level implementation details
• Instead of reading the source code, you can come up with a hypothesis on each implementation detail, and then verify it

40

Educated Guesses

• A rough expectation of the execution time or memory usage, and how they should grow when the size of input grows
• It does not make any sense for Microsoft Word to use 10 GB of memory
• It does not make sense for Unicorn Engine to run slower and slower on the same input
• It does not make sense for ls to take more than 10 minutes to list files under /tmp/ … or does it?

41

Educated Guesses

• Ultimately, the performance of a program is determined by its algorithms
• Algorithmic complexity
• Does the execution time or memory usage grow as expected?
• Is it using the most optimal algorithm?

42

Variance Reduction

• Only changing one variable (or one set of variables) and keeping all other variables constant and unchanged throughout one iteration of investigation
• Changes in multiple variables may all influence the result. Which one is the culprit?

43

Binary Search

• Multiple factors are potentially causing the problem. How can we quickly identify which one is the culprit?

44

Git Bisect

40b15f3164cca196f794bb2c4d47b1d0a7a7bf9b

ab187ac80deeeaf2e4e0d60562169556cfa1bacf

838cc653e4af003df00a4abf0c1ca60d063661dd

b0b456b794ed1b42c0292b2db8dd80820c8cd5d4

dc499fea726781a4e609d3429536fce7e7a902e2

515ab0d0bcb150939584f09959d4e279f2d8d956

d5391f96394a9415da1d50caa07e8dc99103050d

​

45

Broken

Works

?

ls /tmp/ is taking forever

• There are many files under /tmp/. Listing them takes time (welp), but why don’t we see any intermediate output?

​

• Guess: What does ls do?
• Opens /tmp/ (opendir)
• Loads file entries under /tmp/ (readdir)
• Formats the output … with colors and columns

​

• Q: What is the complexity of each step?

46

Complexity: Hard drives

• Spinning drives and SSDs have different reading speed (and complexity)
• Especially when reading small, fragmented files
• Spinning drives: O(N)
• SSDs: O(1)

​

• Cache matters!

47

Complexity: File systems

• Not all file systems are born equal
• XFS is slow when dealing with many small files
• NFS is extremely slow when dealing with many files (why?)
• Fuse is slow, and you may be using it implicitly
• ntfs-3g
• Mapping directories or partitions into a virtual machine (although they are on the same host machine) can lead to extremely bad performance
• Have you ever looked at their underlying implementations?

48

Complexity: Hidden behaviors

• Antivirus
• Stupid antivirus solutions would scan files that you are deleting
• Failing hard drives
• Reads may fail (taking too long, or returning erroneous data)
• Writes may fail and block reads

49

Complexity: Formatting Output

• It seems to be O(1)…
• But in column output mode, ls must know the longest file name to determine column sizes!
• This makes intermediate output impossible

50

Complexity: The Actual Output

• Outputting to a console is not fast
• cmd.exe or Windows Terminal are even slower
• Outputting to a console over SSH or RDP is even slower

51

Where is the bottleneck?

• Q: How do we determine the actual bottleneck(s)?

​

​

• Q: What is the solution?

​

52

Key Debugging Steps

• Making the bug reproducible
• Eliminating indeterminism
• Isolating the problem

Which process? Which executable? Which operation? Which input?

• Finding control variables
• Ideally, only one variable is changed each time!

53

Slow Docker

• A Dockerized service is taking much much longer to finish on a server
• What the service does: building software packages

​

​

54

Slow Docker

• Analysis steps
• The service itself?
• Docker?
• OS?
• File system?
• Hardware?

​

• The final truth
• SSDs were failing, leading to extremely low writing speed

55

Unicorn Engine Running Slower

• Unicorn Engine is a user-mode machine code emulator based on Qemu
• It runs machine code in an emulated environment
• Problem: The more code we ran, the slower each run became
• This is definitely not the expected behavior (does your CPU work like this?)

56

Unicorn Engine Running Slower

• What happened?
• Educated guess: Unicorn Engine is pretty much just an emulator. It does not really wait long for any syscalls.
• Educated guess: No multithreading
• Educated guess: It’s probably caused by some sort of data structure with an O(N) lookup complexity (e.g., a linked list).
• Hypothesis: Certain functions must be taking longer and longer each time we run code inside

57

Unicorn Engine Running Slower

• What happened?
• Making the problem reproducible: Writing a minimal test case to reproduce the problem reliably.�I developed a small timer function in C to time arbitrary functions.
• Isolating the issue: Binary search in all functions that are called, following the call tree. Observe which function takes longer and longer.
• Control variable: The number of machine code snippets that we execute in Unicorn Engine.

58

Unicorn Engine Running Slower

• Finally
• Unicorn Engine abuses the device mapping mechanism in Qemu to map memory regions and memory pages
• Qemu does not seem to support freeing device mapping regions, which is usually not a problem since you rarely plug and unplug devices when a system runs
• In Unicorn Engine, emulated programs may allocate or deallocate memory regions all the time!

59

Unicorn Engine Running Slower

• Finally (cont.)
• All memory regions are put together in a linked list. Without removing regions out of this linked list ever, allocating new regions becomes slower and slower.
• A fast-lookup view (FlatView) is supposed to be offering fast lookups. But it’s never reconstructed after a memory region is freed…

60

https://github.com/unicorn-engine/unicorn/pull/655

Questions?

61