Dynamic Analysis of Malware

Sareena K P

Secure Systems Engineering (CS6570)

IIT Madras

Mar 10, 2022

Static Analysis: What can go Wrong?

• Examines the malware binary (e.g. Strings) 

       To Infer maliciousness

• Can be evaded!

2

Polymorphic Malware!!

Dynamic Analysis

3

1

Dynamic Analysis

• Execute malware    -  Observe Impact on System
• Trails on the system stack (Network, OS, Hardware)
• Can identify run-time behavior
• Immune to
• Packers
• Obfuscation
• ZERO day malware

​

​

​

​

​

​

4

Executing Malware - Challenges

5

2

#1 Isolation and Containment

• Executing Malware -> Repercussions Costly!!!
• System and Network
• Data Loss, sabotage, DDoS

​

• Rootkits affects BIOS as well!
• Isolation and Containment necessary  
• Obvious approach – Virtual Machines, Sandboxes

​

6

#2 Sensitive Nature of Malware

• Malware can detect analysis environments
• Determine if running in a VM  (amount of RAM) , is a network of machines present?
• Can decide not to RUN
• Looks for trigger typically present in Real-World
• Network of devices , Heterogeneity

​

• Malware depends on Command and Control  (C&C) Server
• Ideal Requirement – Real-World Testbed, Internet Connectivity

​

​

7

Is this an analysis environment?

​

#4 State Reset

• Malware affects the System state –  Filesystem, Registry etc
• Each new sample must start from clean state of the system.
• Stateless Evaluations!
• Software available to reset OS! 
• e.g. RebootRx
• Rootkits that affect BIOS???

8

#4 Active Nature of Malware

​

• C&C servers added to public blacklist after some time >> BLOCKED
• If C&C servers are down, malware does not function as intended  - Stale Data
• Analyze promptly before C&C servers are pulled down.

​

9

Analysis Frameworks

10

2

Analysis Framework

• Malware Sample
• Hardware & OS
• Analysis Tools
• Monitors the code or the system
• Eg – Procmon that capture the system calls
• Wireshark to capture network packets

​

​

11

Requirements

• Trusted—Should not be compromised by the malware
• Undetectable – To prevent evasion
• Collect most information possible
• Meet requirements for the malware to execute
• Isolation and containments

​

​

12

#1 Bare Metal (Real-World)

• Executing malware on a physical machine
• Challenge is  to reset the clean baselines
• Only suitable for user mode malware
• Rootkits?

​

13

Virtualizations – In Guest Analysis

Virtual Machines

​

​

• Separation between host OS and guest OS
• Can run kernel mode malware
• Resetting the VMs – easier
• Can affect the host OS if VM implementation has vulnerabilities

14

Outside Guest Analysis

15

Secures the analysis tool

Limits the guest OS's access to hardware resources

​

Challenges Summary

Real World Testbed 

Virtualized Analysis Environments

16

Complete State Reset

(e.g. Rootkits)

Precise

Low

Containment

High

Containment

Safe but can be evaded

Precise but risky

Sophistication

Analysis Techniques

17

3

Analysis Techniques

Passive (Tracing)

• Passively observe the trails/behaviour
• Side Channels
• Network
• OS
• Hardware
• Memory
• Use the data to infer maliciousness.
• Likely to be not detected by malware

​

​

​

Active

• Control the execution and observe
• Function call analysis
• Execution Control
• Flow tracking
• Malware can identify debugging environments and evade

18

Passive Techniques (Tracing)

19

A

Network Behaviour

• Malware communications to external servers
• To its command and control
• Network packets
• Cannot be evaded.

20

Network

Hardware

OS

Operating System

• All System calls
• File system (Read/Write)
• Registry, Process
• And even  network!
• Functionality
• Achieve persistence,
• Remain stealthy

​

21

Network

OS

Hardware

Hardware - Counters

• Side channel 
• Micro-architectural Events
• Eg. Cache misses, 
• Observable using special registers called hardware performance counters.

22

Network

OS

Hardware

Hardware - Volatile Memory Acquisition

• Copying RAM contents to a memory dump file
• Acquisition
• Software
• Hardware
• Difficult to evade
• Malware has no way of telling that a dump was taken 
• Suitable for ? Guesses?  -

​

​

23

Network

OS

Hardware

Hardware - Volatile Memory Acquisition

• Copying RAM contents to a memory dump file
• Acquisition
• Software
• Hardware
• Difficult to evade
• Malware has no way of telling that a dump was taken 
• Suitable for ? Guesses?  -
• Fileless malware, Rootkits, and Malicious Firmware

​

24

Network

OS

Hardware

Active Techniques

25

B

Function Call Analysis

• Function calls / System calls
• Hooking – getting notification when a function is called
• - Windows OS hooking mechanism
• Code injection techniques
• E.g -  CWSandbox, Capture, MalTrack

​

​

​

26

​

Target Function

Injected

Hook

Analysis Function

Access process stack, paramters

Can explore only one control path that was executed!

Execution Control 

​

​

​

27

• Interrupt the execution and check system state
• Debugging
• Binary Instrumentation
• Forward Symbolic Execution
• Multiple Path Exploration

​

​

#1 Debugging

​

​

​

28

• Single stepping using CPU Trap flag
• Interrupt after each opcode
• Analysis process can check the state of process and OS
• Cons
• Resource-consuming
• Malware can easily detect and evade (check for PUSHF)

​

​

​

#2 Binary Instrumentation

​

​

​

29

• Add analysis code to original malware code
• Decompile the malware 
• Look for control transfer opcodes (CALL, RET, JMP)
• Return address is altered so that it jumps to analysis tool which executes the next block.

​

​

#3 Forward Symbolic Execution

​

​

​

30

• Identifying required conditions required to execute 
• Evaluate using different inputs and environments
• Keeps track of the input and its effects on the malware.
• Multiple-path exploration – Explores all paths

​

​

​

Path1

αi < 10

Path2

αi >= 10

If i < 10

Input i

Eg – For Query timestamp​, replace with symbols

Flow Tracking – Data Tainting

​

​

​

31

   x   =   input ( )

   …

   y    =  x   +   42

   …

   goto y

​

• Tracking the information flow – to understand how malware interacts with OS
• Taint data of suspicion (e.g. user input)
• When opcode manipulates x, the memory is also tainted, 
• E.g. Buffer Overflow Exploit
• Eg – Vigilante, Panorama

​

​

​

Flow Tracking – Data Tainting

​

​

​

32

   x   =   input ( )

   …

   y    =  x   +   42

   …

   goto y

​

Input is tainted

• Tracking the information flow – to understand how malware interacts with OS
• Taint data of suspicion (e.g. user input)
• When opcode manipulates x, the memory is also tainted, 
• E.g. Buffer Overflow Exploit
• Eg – Vigilante, Panorama

​

​

​

Flow Tracking – Data Tainting

​

​

​

33

   x   =   input ( )

   …

   y    =  x   +   42

   …

   goto y

​

Input is tainted

X

• Tracking the information flow – to understand how malware interacts with OS
• Taint data of suspicion (e.g. user input)
• When opcode manipulates x, the memory is also tainted, 
• E.g. Buffer Overflow Exploit
• Eg – Vigilante, Panorama

​

​

​

Flow Tracking – Data Tainting

​

​

​

34

   x   =   input ( )

   …

   y    =  x   +   42

   …

   goto y

​

Input is tainted

X

X

42

• Tracking the information flow – to understand how malware interacts with OS
• Taint data of suspicion (e.g. user input)
• When opcode manipulates x, the memory is also tainted, 
• E.g. Buffer Overflow Exploit
• Eg – Vigilante, Panorama

​

​

​

Flow Tracking – Data Tainting

​

​

​

35

   x   =   input ( )

   …

   y    =  x   +   42

   …

   goto y

​

Input is tainted

X

X

y

42

• Tracking the information flow – to understand how malware interacts with OS
• Taint data of suspicion (e.g. user input)
• When opcode manipulates x, the memory is also tainted, 
• E.g. Buffer Overflow Exploit
• Eg – Vigilante, Panorama

​

​

​

Flow Tracking – Data Tainting

​

​

​

36

• Tracking the information flow – to understand how malware interacts with OS
• Taint data of suspicion (e.g. user input)
• When opcode manipulates x, the memory is also tainted, 
• E.g. Buffer Overflow Exploit
• Eg – Vigilante, Panorama

​

​

​

   x   =   input ( )

   …

   y    =  x   +   42

   …

   goto y

​

Input is tainted

X

X

y

y

42

Techniques versus Frameworks

37

4

Mapping Layout and Techniques

38

Cannot be detected by malware

Easily detected by malware

Malware may detect

Mapping Layout and Techniques

39

Cannot be detected by malware

Easily detected by malware

Malware may detect

Analysis instruments the malware binary

Mapping Layout and Techniques

40

Cannot be detected by malware

Easily detected by malware

Malware may detect

Analysis tool is a different process

Analysis instruments the malware binary

Mapping Layout and Techniques

41

Cannot be detected by malware

Easily detected by malware

Malware may detect

Analysis tool is a different process

Analysis instruments the malware binary

Mapping Layout and Techniques

42

Cannot be detected by malware

Easily detected by malware

Malware may detect

Analysis tool is a different process

Analysis instruments the malware binary

Mapping Layout and Techniques

43

Cannot be detected by malware

Easily detected by malware

Malware may detect

Analysis tool is a different process

Analysis instruments the malware binary

Limitations

• Only executed code is observable.
• No triggers –-> Not analyzed

​

• Requires computational overhead, which may slow down execution. 

​

​

44

Virtualizations

Virtual Machines

​

Emulation

​

45

Emulator – hardware level implementation of a system

Does not execute malware on hardware.