Advanced Debugging Techniques and Strategies

CSE 598 – Applied Program Analysis and Debugging

Fall 2025

​

Fish Wang

Arizona State University

Becoming a Debugging Pro

• Efficient debugging is
• Using right tools
• Following the optimal strategy
• Having enough experience�Practice makes perfect

2

A Step-by-step Guide

• Noticing a problem
• Making the problem reproducible
• Ideally, in a controlled and minimal environment with controlled and minimal variables
• Making hypotheses
• What hypotheses to make depend on your intuition
• Finding proper tools and artifacts
• Logs? Debuggers? Tracers? Profilers?
• Source code? What libraries? Which version? Debug symbols?
• Can I modify source and rebuild?

​

3

A Step-by-step Guide (Cont.)

• Using tools to help verify your hypotheses, one at a time!
• (Optional) Fixing the bug

4

Making the problem reproducible

• What common solutions are there?
• Run the buggy line/function/program many times
• Eliminating unrelated variables in the environment, e.g., networking, files, human operations, etc.�Automation! Scripting!
• Eliminating unrelated threads
• Snapshotting the entire environment

5

Random exceptions in Python

... Snip ...

File "f:\angr\angr\angr\analyses\forward_analysis.py", line 557, in _analyze

self._analysis_core_graph()

File "f:\angr\angr\angr\analyses\forward_analysis.py", line 580, in _analysis_core_graph

changed, output_state = self._run_on_node(n, job_state)

File "f:\angr\angr\angr\analyses\variable_recovery\variable_recovery_fast.py", line 712, in _run_on_node

input_state = prev_state.merge(input_state, successor=node.addr)

File "f:\angr\angr\angr\analyses\variable_recovery\variable_recovery_fast.py", line 488, in merge

merged_register_region = self.register_region.copy().replace(replacements).merge(other.register_region,

File "f:\angr\angr\angr\keyed_region.py", line 159, in copy

kr._object_mapping = self._object_mapping.copy()

File "D:\My Program Files\Python37\lib\weakref.py", line 174, in copy

for key, wr in self.data.items():

RuntimeError: dictionary changed size during iteration

6

Random exceptions in Python

• When did it happen?
• On our CI: Once every three weeks (about 100 full CI runs)
• Locally: When using angr management (the GUI for angr); happened once on my machine
• Worse…
• I could not trigger this exception locally by running the “broken” test case on CI!

7

Random exceptions in Python

• What to do?

8

Random exceptions in Python

• What I tried
• Acquired an SSH shell into the box that ran angr CI tasks
• Run the “broken” test case 1000 times consecutively
• … did not trigger that exception

9

Random exceptions in Python

• What I tried
• Realized from the backtrace that the exception occurred during a variable recovery pass in angr
• Wrote a Python script that performs the same variable recovery pass 1000 times on the same function in the same binary
• … did not trigger that exception

10

Random exceptions in Python

• What I tried
• A key realization: angr management uses multithreading during the variable recovery analysis pass!
• Then I directly ran the same pass on the same binary and the same function 1000 times inside angr management
• It raised that exception reliably

11

https://bugs.python.org/issue35615

https://github.com/python/cpython/pull/11384

Making hypotheses

• This is where experience matters

12

Weird Network Jitters

• macOS Monterey (circa 2021) causing constant video and audio lags when using Parsec or Steam Play
• What might be happening?

13

Weird Network Jitters

• Jitters -> Inconsistent latencies

​

14

Weird Network Jitters

• Steps of diagnosis
• Logs showed nothing bizarre
• Traffic capturing revealed no strange traffic
• Terminating random running apps did not help
• Turning off Bluetooth did not help
• Reinstalling macOS did not help

15

A Key Observation

• “akin to what happens if you open the Wi-Fi dropdown on your menu bar and it starts scanning for networks”
• The behavior does not trigger when tethering over cellphones

​

​

16

Weird Network Jitters

• The culprit

​

​

• The AirPlay button on the Touch Bar caused macOS to constantly scan for AirPlay-enabled devices on the local network
• Unknown why it would cause jitters
• Fixed by Apple in macOS Montery 12.5

​

17

https://mnpn.dev/blog/airplay-network-disaster

Floats in Python

• angr is a program for analyzing binary code
• On some binaries with floating point arithmetic, angr’s emulation gives different results than a real CPU execution
• What’s going on?

18

xmm0

00 00 00 00 aa 9d f4 36

3a 26 00 44 aa 9d f4 36

angr’s emulation

Real execution

Floats in Python

• Knowing two facts
• IEEE 754
• Python’s floats are of double precision
• My hypothesis
• It is caused by emulating casting-to-float conversions as to-double (although it looks like casting-to-float) conversions in Python

19

xmm0

00 00 00 00 aa 9d f4 36

3a 26 00 44 aa 9d f4 36

angr’s emulation

Real execution

IEEE 754

• IEEE Standard for Floating-Point Arithmetic

​

20

Mantissa

-1^S x 2^E x M

Floats in Bytes

• Endianness: Little-endian
• Differences are within the mantissa
• Guess: angr’s emulated result is more precise than the one from real execution
• Educated guess: It’s probably because of float(a) somewhere in angr’s Python code.

21

xmm0

00 00 00 00 aa 9d f4 36

3a 26 00 44 aa 9d f4 36

angr’s emulation

Real execution

How does breakpoint work?

• The debugger launches or attaches to the debuggee (via a syscall)
• The debugger modifies the instruction to execute (changes the page permission, modifies a byte, and changes the page permission back, via syscalls)
• When the debuggee executes the instruction, it triggers a trap, and a signal is sent to the debugger

22

What instruction to use?

• Interrupts
• Kernel-only instructions
• Undefined instructions
• Store to an invalid address
• Read from an invalid address

​

​

23

On ARM…

• bkpt #0
• Breakpoint instruction
• f7f0a000
• Undefined instruction
• udf #16
• Breakpoint instruction
• Store to an invalid address

​

• Read from an invalid address

​

24

https://www.jwhitham.org/2015/04/the-mystery-of-fifteen-millisecond.html

14.8 milliseconds

9 microseconds

10 microseconds

10 microseconds

9 microseconds

One is slower than others?

• What might be going on?

25

One is slower than others?

• Observing the CPU usage while executing bkpt #0 in a loop
• 92.5% CPU time used by the executable, 3.9% used by rsyslogd
• Looking into the syslog
• Apr 4 22:50:59 heating kernel: [14572.940927] Unhandled prefetch abort: breakpoint debug exception (0x002) at 0x00008438
• Writing to syslog accounts for ~1ms

26

What’s next?

• Linux kernel handles several types of breakpoints

27

Fallback code

• The fallback logic for unhandled reasons calls printk to generate the “Unhandled prefetch abort” message
• printk is synchronized – it writes characters (character by character) to the console!

28

Proper tools and artifacts

• While printf debugging is possible, a good debugger can come extremely handy
• Windows: x64dbg, WinDBG, Visual Studio, VS Code
• Linux: GDB
• MacOS: LLDB, Xcode

29

Tips for using debuggers

• Conditional breakpoints
• Hardware watchpoints
• Scripting them
• GDB scripts
• Plugins
• Gef for GDB

30

Understanding The Bug

• Access violation/segmentation fault

31

Understanding The Bug

• Access violation/segmentation fault
• Caused by memory reads or writes to invalid addresses
• Why do they happen?
• Corrupted pointers
• Corrupted array offsets
• Why do they happen?
• Corrupted internal data structure (e.g., the heap)
• Calling memory read/write functions on corrupted arguments
• Type confusion
• Why do they happen?

32

Differential Debugging

• The execution result of a program is fully determined by its code (and the order of execution) and the data each instruction consumes

​

• If we have a buggy execution and a normal execution of the same program, differential debugging will help us find where their executions differ

33

Differential Debugging (Cont.)

• The comparison can be performed on various levels (with their own caveats)
• Full execution traces with data
• Full execution traces, instruction only
• Function call traces
• Library call traces
• Syscall traces
• Logging output

34

Poor Man’s Differential Debugging

• Demo!

35

Time Travel Debugging

• Traditional debuggers only debug one execution in sequence – you cannot go back in time
• If you set new breakpoints in code that has been executed, you will have to re-execute the program
• If a debugger logs effects of each instruction, it will allow us to perform time travel debugging

36

Time Travel Debugging

• Common TTD debuggers
• Mozilla rr
• WinDBG
• Qira

​

• How do they work?
• Record precise timing information of context switches between threads
• Record effects of syscalls
• (Optionally) record effects of checkpoints (certain instructions or program points)

37

Core dumps

• Reproducing crashes is not always possible…
• Core dumps provides a snapshot of registers and memory at the point of crashing
• Analysts need to use limited information in a core dump to reason about the root cause
• Doing it manually is tedious and sometimes impossible
• Can we do better?
• REtracer

38

A big problem in debugging

• We lack good tools
• What problems have you encountered during debugging?
• What tools can make our life easier?
• Discuss with me!

39

Questions?

40