Advanced Malware Analysis & Reverse Engineering (Digital Forensics Context)

By:

Dr. Mohammad Shoab

Week 4

Why Malware Analysis in Digital Forensics?

• Incident Response: How did the breach happen? What was the initial vector (e.g., phishing email, exploit)?
• Impact Assessment: What data was exfiltrated? What systems were compromised? (IOCs - Indicators of Compromise)
• Attribution: Linking malware to a specific threat actor (APT group, cybercriminal) via TTPs (Tactics, Techniques, and Procedures).
• Evidence for Legal Proceedings: Understanding the malware's functionality is crucial for building a case.
• Goal: Move from "we found a malicious file" to "this is what it did, how it did it, and who might be behind it."

​

2

Digital Forensics 2

Department of Computer Science

Malware Fundamentals: Types & Definitions

• Virus: Attaches to a clean file and spreads.
• Worm: Self-replicating across networks.
• Trojan: Disguised as legitimate software.
• Ransomware: Encrypts files for ransom.
• Spyware / InfoStealer: Logs keystrokes, steals credentials.
• Rootkit: Hides its existence and other malware.
• Botnet Agent: Allows remote control of the system (zombie).
• Forensic Question: "What type of malware is this, and what is its primary mission?"

3

Digital Forensics 2

Department of Computer Science

The Forensic Analysis Environment

• The Sandbox: Isolated, virtualized environment (e.g., VM Ware, VirtualBox).
• Air-Gapped Network: Physically disconnected from the internet and internal networks.
• Remnux: A Linux toolkit for reverse-engineering and analyzing malware.
• FlareVM: A Windows-based security distribution for malware analysis.
• Host-Only Networking: Allows VMs to talk to each other but not to the host's physical network.
• Snapshot Usage: Revert to a clean state after each analysis run.

4

Digital Forensics 2

Department of Computer Science

The Malware Analysis Process

• A Triage Approach:
• Static Analysis (Without Executing) -> Quick wins, initial IOCs.
• Dynamic Analysis (Behavioral Analysis) -> Observe real-time actions.
• Memory Analysis -> Analyze the malware's footprint in RAM.
• Code Reverse Engineering -> Deepest understanding.

​

• Forensic Principle: Start with the least invasive techniques to preserve evidence integrity.

​

5

Digital Forensics 2

Department of Computer Science

Static Analysis: Part 1 - Fingerprinting

"What is it?“

​

• File Hashing: Generate MD5, SHA1, SHA256 hashes. (Use for VirusTotal, indexing IOCs).
• PE Header Analysis: (For Windows binaries) Examine compile timestamps, sections, imports, exports.
• Strings Analysis: Search for human-readable text: IP addresses, URLs, registry keys, function names (e.g., CreateRemoteThread, RegSetValue).
• Tool: strings.exe, PE-bear, ExeinfoPe

​

6

Digital Forensics 2

Department of Computer Science

Static Analysis: Part 2 - Examining the Code Structure

"What might it do?“

​

• Import Address Table (IAT): Lists functions the malware imports from DLLs (e.g., kernel32.dll, advapi32.dll). A roadmap of its capabilities (network, file, registry access).
• Packing & Obfuscation: Malware is often compressed/encrypted ("packed") to evade signature detection. The first few instructions unpack the real code.
• Tool: Dependency Walker, PEview, IDA Pro (freeware version).

7

Digital Forensics 2

Department of Computer Science

Dynamic Analysis: Part 1 - System Monitoring

"What does it do when running?“

​

• File System Changes: What files does it create, delete, or modify?
• Registry Modifications: What auto-start persistence mechanisms does it set? (e.g., Run, RunOnce keys).
• Process Activity: What child processes does it spawn?
• Tools: Process Monitor (Procmon) from Sysinternals is the gold standard.

​

8

Digital Forensics 2

Department of Computer Science

Dynamic Analysis: Part 2 - Network Monitoring

"Who does it talk to?“

​

• DNS Requests: What domains does it try to resolve?
• HTTP/HTTPS Calls: What data is being sent out? (Exfiltration).
• C2 Communication: Callbacks to Command & Control servers.
• Tools: Wireshark (packet analysis), FakeNet-NG (simulates a network to trick malware into revealing C2).

​

9

Digital Forensics 2

Department of Computer Science

Memory Forensics: Why It's Crucial

• Malware, especially rootkits, often leaves no trace on the disk but must reside in RAM to execute.
• Memory contains plaintext passwords, encryption keys, decrypted code, and active network connections.
• Forensic Process: Acquire a memory dump (dumpit.exe, BelkaLiveRAM), then analyze it offline.
• Tool: Volatility Framework (the industry standard).

​

​

​

10

Digital Forensics 2

Department of Computer Science

Memory Forensics with Volatility

Common Volatility Commands:

• imageinfo: Identify the OS profile.
• pslist: List running processes (find the malware).
• dlllist: List DLLs loaded by a process.
• connscan: Scan for network connections.
• malfind: Find hidden/injected code.
• dumpfiles: Extract artifacts from memory.

​

• Forensic Goal: Correlate memory artifacts with disk and network findings.

11

Digital Forensics 2

Department of Computer Science

Introduction to Reverse Engineering

• Goal: Translate machine code (binary) back into assembly language to understand the program's logic.
• Disassembler: Static analysis tool that converts binary to assembly (e.g., IDA Pro, Ghidra).
• Debugger: Dynamic analysis tool that allows step-by-step execution of the code, inspecting registers and memory (e.g., x64dbg, OllyDbg).
• Prerequisite: Basic understanding of x86/x64 Assembly Language.

​

12

Digital Forensics 2

Department of Computer Science

Core RE Concepts: CPU Registers

• EIP/RIP: Instruction Pointer - holds the address of the next instruction to execute. The most important register for controlling flow.
• EAX/RAX: Accumulator - often used for function return values.
• ESP/RSP: Stack Pointer - points to the top of the stack.
• EBP/RBP: Base Pointer - points to the base of the current stack frame.
• Understanding data flow through registers is key.

​

13

Digital Forensics 2

Department of Computer Science

Core RE Concepts: The Stack

• A region of memory used for temporary storage.
• LIFO (Last-In, First-Out) data structure.
• Used for:
• Storing function parameters and return addresses.
• Saving register states.
• Allocating space for local variables.
• Stack Overflow: A common exploitation technique.

​

14

Digital Forensics 2

Department of Computer Science

Core RE Concepts: Assembly Instructions

• mov eax, 1 ; Move the value 1 into the EAX register (data transfer).
• add ebx, eax ; Add EAX to EBX (arithmetic).
• cmp eax, ebx ; Compare EAX and EBX (sets flags for...).
• jz 0x401000 ; ...Jump if Zero (control flow).
• call 0x401050 ; Call a function (pushes return address onto stack).
• push eax / pop ebx ; Manipulate the stack.

15

Digital Forensics 2

Department of Computer Science

Tools: Disassemblers - IDA Pro & Ghidra

• IDA Pro: Industry-standard disassembler. Creates interactive graphs of code flow. (Expensive).
• Ghidra: NSA's open-source alternative. Powerful, free, and includes decompilation (shows pseudo-C code).
• Key Features: Graph view, cross-references (XREFs), renaming variables, adding comments.

​

16

Digital Forensics 2

Department of Computer Science

Tools: Debuggers - x64dbg

• x64dbg: The modern, open-source debugger for Windows (32-bit and 64-bit).
• Features:
• Breakpoints: Pause execution at a specific instruction.
• Stepping: Execute one instruction at a time  (F7 for step into, F8 for step over).
• Viewing: Inspect registers, memory dump, and the stack in real-time.
• Use Case: Bypass anti-analysis checks, decrypt strings, understand complex logic.

17

Digital Forensics 2

Department of Computer Science

Common Malware Capabilities in Code

What to look for when reversing?

​

• Persistence: Calls to RegSetValueEx, CreateService.
• Code Injection: Calls to VirtualAllocEx, WriteProcessMemory, CreateRemoteThread.
• Network Communication: Calls to socket, connect, send.
• Anti-Debugging: Calls to IsDebuggerPresent, OutputDebugString.
• String Encryption: Loops with XOR operations or complex algorithms.

18

Digital Forensics 2

Department of Computer Science

Anti-Analysis Techniques

How Malware Fights Back?

​

• Packing/Obfuscation: Hides the real code.
• Anti-Debugging: Detects debuggers and alters behavior.
• Anti-VM: Detects virtualized environments (e.g., checks for VM-specific drivers).
• Code Armoring: Polymorphic and metamorphic code that changes itself.
• Forensic Countermeasure: Use a well-concealed environment and patching techniques to bypass these checks.

19

Digital Forensics 2

Department of Computer Science

Extracting Indicators of Compromise (IOCs)

The Forensic Deliverable

​

• Hashes: File hashes (SHA256).
• Network IOCs: C2 IP addresses, domains, URLs.
• Host-based IOCs: File paths, registry keys, mutex names.
• YARA Rules: Write a custom signature to detect this malware family in the future. "The pattern that defines the malware."
• STIX/TAXII: Standard formats for sharing IOCs.

​

20

Digital Forensics 2

Department of Computer Science

Building a Malware Analysis Report

Communicating Findings

​

• Executive Summary: Brief overview of the threat and impact.
• Technical Analysis: Detailed breakdown of static, dynamic, and code analysis.
• IOCs: Complete list in a machine-readable format.
• Mitigation Steps: Recommendations for containment and eradication.
• Attribution Assessment: Likely threat actor based on TTPs.

​

21

Digital Forensics 2

Department of Computer Science

Case Study: A Ransomware Attack

Applying the Process

​

• Scenario: A company is hit by ransomware. You have a sample from the incident response team.
• 1. Static Analysis: Find encrypted strings, check imports for crypto functions.
• 2. Dynamic Analysis: Watch it encrypt files, note the ransom note filename.
• 3. Reverse Engineering: Find the encryption algorithm and see if the key is recoverable.
• 4. Memory Forensics: Look for the encryption key in the memory dump of the encrypted process.
• Outcome: Provide IOCs to hunt for other infections and determine if decryption is possible.

22

Digital Forensics 2

Department of Computer Science

Legal and Ethical Considerations

• Authorized Access Only: Only analyze malware you own or are explicitly authorized to investigate.

​

• Containment is Critical: Prevent the malware from escaping your lab.

​

• Chain of Custody: Document how the sample was handled if it's part of an official investigation.

​

• Reporting: Findings may be subject to attorney-client privilege or used in court.

​

23

Digital Forensics 2

Department of Computer Science

The End

24

Digital Forensics 2

Department of Computer Science