Analysis of a Compromised System  
Part 1
: Online Analysis and Data Collection

License

This work by Z. Cliffe Schreuders at Leeds Beckett University is licensed under a Creative Commons Attribution-ShareAlike 3.0 Unported License.

Contents

Preparation

Configuring the environment

Note IP addresses

Reminder to save

Introduction

Live analysis

Collecting live state manually

Comparing process lists

Gathering live state using statically compiled programs

Collecting live state using scripts

Checking for rootkits

Offline analysis of live data collection

Windows and live incident response

What’s next ...

Introduction

This lab has been designed and tested with openSUSE and Kali Linux, but could be completed on most Linux systems with tcpdump, Wireshark, and Snort installed.

Preparation

If you are working on campus in the IMS labs using the oVirt online labs, click here for instructions on how to login on campus in the IMS labs and create VMs from templates.

If you are working remotely using the oVirt online labs, click here for instructions on how to login via VPN and create VMs from templates.

The oVirt system is a new online lab infrastructure hosted here at Leeds Beckett. This infrastructure is currently on trial, as a pilot. There is a good chance there will be some technical issues, and not every lab will be available via this system. However, if you are happy to benefit from this experiment, please keep in mind that you may need to fall back to one of the above methods.

If you are working remotely having downloaded our VMs or by copying them when you were on campus, click here for instructions on how to download VMware Player and configure the VMs to run remotely.

If you are on campus using the IMS system, click here for instructions on how to use the IMS system and VM download scripts.

Create a new VM from the Redhat7.2_Template2 template and give it a name that begins with your “c” number. DO NOT START the new VM.

The Red Hat 7.2 template is a copy of a server that has been compromised  (courtesy of the Honeynet Project[1]).

Once oVirt has finished creating your new VM, right click on it and choose Edit as shown below:

In the edit dialogue box shown below (Help: if you do not see all these options, click the "Show Advanced Options" button at the bottom left-hand side of the window):

Select CD-ROM as the Second Device.

Tick the Attach CD box.

Select the “fire-0.3.5b.iso” file from the dropdown box.

Click OK to save the selections.

Start these VMs:

Note: The root password on the openSUSE_Leap_42.1-- which should NOT be used to log in graphically -- is “tiaspbiqe2r” (this is a secure password but is quite easy 2 remember). Again, never log in to the desktop environment using the root account -- that is bad practice, and should always be avoided.

Start both VMs and log in as the users specified above.  

Note: if you are using VMware VMs, you will have to download (from the University Network drive) and attach a copy of the fire-0.3.5b.iso file as a CD.

Note IP addresses

Login to the Red Hat 7.2 VM with root and toor.

Run “dhcpcd” on the VM to renew its IP address.

On your the openSUSE system run /sbin/ifconfig, and also run ifconfig on the compromised (Red Hat) VM. Make a note of the two IP addresses, which should be on the same subnet (starting the same): for example, if both systems include IP addresses starting with 192.168.201 they are on the same network.  You will need these IP addresses later.

Reminder to save

Remember to save any evidence you collect (on your Google Drive or webmail it to yourself), as this will be used as the basis for the next lab. You cannot complete the following lab (part 2) without saving the evidence you collect during this lab.

Introduction

So you have reason to believe one of your servers has experienced a security compromise... What next? For this lab you investigate a real Linux server that was connected to the Internet, and attacked and compromised by unknown remote attackers.

The investigation of a potential security compromise is closely related to digital forensics topics. As with forensic investigations, we also aim to maintain the integrity of our “evidence”, wherever possible not modifying access times or other information. However, in a business incident response setting maintaining a chain of evidence may not be our highest priority, since we may be more concerned with other business objectives, such as assuring the confidentiality, integrity, and availability of data and services.

During analysis, it is good practice to follow the order of volatility (OOV): collecting the most volatile evidence first (such as the contents of RAM, details of processes running, and so on) from a live system, then collecting less volatile evidence (such as data stored on disk) using offline analysis.

Live analysis

After suspecting a compromise, before powering down the server for offline analysis, the first step is typically to perform some initial investigation of the live system. Live analysis aims to investigate suspicions that a compromise has occurred, and gather volatile information, which may potentially include evidence that would be lost by powering down the computer.

On the compromised VM (Redhat7.2): To keep a record of what we are doing on the system, start the script command:

mkdir /tmp/evid

script -f /tmp/evid/invst.log

Note: if you do this lab over multiple sessions, be sure to save the log of your progress (/tmp/evid/invst.log), and restart script.

Make a note of the risks and benefits associated with storing a record of what we are doing locally on the computer that we are investigating:

Consider the advantages of handwritten documentation of what investigators are doing.

Many of the commands used to investigate what is happening on a system are standard Unix commands. However, it is advisable to run these from a read-only source, since software on your system may have been tampered with. Also, using read-only media minimises the changes made to your local filesystem, such as executable file access times.

During preparation, you configured the compromised VM to have access to the FIRE (Forensic and Incident Response Environment) CD/DVD ISO (which is equivalent to inserting the optical disk into your server’s DVD-tray). FIRE is an example of a Linux Live Disk that includes tools for forensic investigation. In addition to being able to boot to a version of Linux for offline investigation of evidence, the disk contains Linux tools for live analysis.

On the compromised VM (Redhat7.2), mount the disk, so that we can access its contents:

mount /dev/hdc /mnt/cdrom/

Help: On VMware VMs you may need to use mount /mnt/cdrom only if the above command does not work.

On a typical system, many binary executables are dynamically linked; that is, these programs do not physically contain all the libraries (shared code) they use, and instead load that code from shared library files when the program runs. On Unix systems the shared code is typically contained in “.so” files, while on Windows “.dll” files contain shared code. The risks associated with using dynamically linked executables to investigate security breaches is that access times on the shared objects will be updated, and the shared code may also have been tampered with. For this reason it is safest to use programs that are statically linked; that is, have been compiled to physically contain a copy of all of the shared code that it uses.

On your openSUSE VM look at which libraries are dynamically loaded when you run a typical command:

ldd /bin/ls

Examine the output, and determine how many external libraries are involved.

On the compromised VM (Redhat7.2): The FIRE disk contains a number of statically compiled programs to be used for investigations. Check that these are indeed statically linked:

ldd /mnt/cdrom/statbins/linux2.2_x86/ls

Compare the output to the previous command run on your own system. The output will be distinctly different, stating that the program is not dynamically compiled.

Note that, although an improvement, using statically linked programs such as these still do not guarantee that you can trust the output of the programs you run. Consider why, and make a note of this:

Collecting live state manually

The next step is to use tools to capture information about the live system, for later analysis. One approach to storing the resulting evidence is to send results over the network via Netcat or SSH, without storing them locally. This has the advantage of not changing local files, and is less likely to tip off an attacker, rather than storing the evidence on the compromised machine.

Comparing process lists

On your openSUSE VM, check whether the local SSH server (sshd) is running on your system:

sudo systemctl status sshd

If the sshd service is not running, you can start it by running:

sudo systemctl start sshd

Check to make sure the SSH server has started:

        sudo systemctl status sshd

Use Yast to make sure that the Secure Shell Server is on the list of Allowed Services in the Firewall settings.

Click   (KDEMenu) → System → Yast

Click Firewall in the Security and Users section

Click “Allowed Services”

Choose “Secure Shell Server” from the “Service to Allow” drop down list

Click Add to add ssh to the list of allowed services.

Click Next → Finish and close Yast

On the compromised VM (Redhat7.2), test sending the results of some commands (process lists using ps) over SSH to your openSUSE VM:

Note: if the VM is not using a UK keyboard layout, the @ and " symbols may be reversed, and the | symbol is located at the ~. Alternatively, run "loadkeys uk" in the RedHat VM to swap to a UK keyboard layout

ssh student@openSUSE-IP-address "mkdir evidence"

ps aux | ssh student@openSUSE-IP-address "cat > evidence/ps_out"

(Where openSUSE-IP-address  is the IP address of your openSUSE NVM, which should be on the same subnet as your compromised VM)

Each time you are prompted for a password, if you are on the openSUSE VM, type “student” (otherwise use a different username/password pair, available on your own system).

Why might it not be a good idea to ssh to your own account (if you had one on the openSUSE VM in real life) and type your own password from the compromised system?

On your openSUSE VM, find the newly created files and view the contents.

Hint: you may wish to use the Dolphin graphical file browser, then navigate to “/home/student/evidence”.

On the compromised VM (Redhat7.2), run the statically compiled version of ls from the incident response disk to list the contents of /proc (this is provided dynamically by the kernel: a directory exists for every process on the system), and once again send the results to your openSUSE VM...

First, to save yourself from having to type “/mnt/cdrom/statbins/linux2.2_x86/” over and over, save that value in a Bash variable:

export static="/mnt/cdrom/statbins/linux2.2_x86/"

Now, to run the statically compiled version of ls, you can run:

$static/ls

Run the command:

$static/ls /proc | ssh student@openSUSE-IP-address "cat > evidence/proc_ls_static"

On your openSUSE VM, find the newly created files and compare the list of pids (numbers representing processes) output from the previous commands. This is the second column of output in the ps_out, with the numbers in proc_ls_static.

Hint: you can do the comparison manually, or using commands such as “cut” (or awk), “sort”, and “diff”. For example, cat ps_out | awk '{ print $4 }' will pipe the contents of the file ps_out into the awk command, which will split on spaces, and only display the fourth field. Ensure this is displaying the list of pids, if not try selecting a different field. You could pipe this through to “sort”. Then save that to a file (by appending “ > pids_ps_out”). We have covered how to use diff previously. Remember “man awk”, “man sort”, and “man diff” will tell you about how to use the commands (and Google may also come in handy).

Are the same processes shown each time? If not, that is very suspicious, and likely indicates a break-in, and that we probably shouldn’t trust the output of local commands.

Gathering live state using statically compiled programs

On the compromised VM (Redhat7.2), save a copy of a list of List only inodes of removed files that are still open or executing:

$static/ils -o /dev/hda1 | ssh student@openSUSE-IP-address "cat > evidence/deleted_out"

Tip: on VMware VMs only, you may need to replace “hda1” with “sda1”.

Save a list of the files currently being accessed by programs:

$static/lsof | ssh student@openSUSE-IP-address "cat > evidence/lsof_out"

On your openSUSE VM, open evidence/lsof_out.

Are any of these marked as “(deleted)”. How does this compare to the ils output? What does this indicate?

On the compromised VM (Redhat7.2),

Save a list of network connections:

$static/netstat -a | ssh student@openSUSE-IP-address "cat > evidence/netstat_out"

(Some commands such as this one may take awhile to run, wait until the Bash prompt returns)

Save a list of the network resources currently being accessed by programs:

$static/lsof -P -i -n | ssh student@openSUSE-IP-address "cat > evidence/lsof_net_out"

Save a copy of the routing table:

$static/route | ssh student@openSUSE-IP-address "cat > evidence/route_out"

Save a copy of the ARP cache:

$static/arp -a | ssh student@openSUSE-IP-address "cat > evidence/arp_out"

Save a list of the kernel modules currently loaded (as reported by the kernel):

$static/cat /proc/modules | ssh student@openSUSE-IP-address "cat > evidence/lsmod_out"

Creating images of the system state

We can take a snapshot of the live state of the computer by dumping the entire contents of memory (what is in RAM/swap) into a file. On a Linux system /proc/kcore contains an ELF-formatted core dump of the kernel. Save a snapshot of the kernel state:

$static/dd if=/proc/kcore conv=noerror,sync | ssh student@openSUSE-IP-address "cat > evidence/kcore"

Next, we can copy entire partitions to our other system, to preserve the exact state of stored data, and so that we can conduct offline analysis without modifying the filesystem.

Start by identifying the device files for the partitions on the compromised system (Redhat7.2):

df

Note that on this system the root partition (mounted on “/”), is /dev/hda1.

Help: on VMware VMs only, you may need to replace “hda1” with “sda1”.

Then, copy byte-for-byte the contents of the root (“/”) partition (where /dev/hda1 was identified from the previous command:

$static/dd if=/dev/hda1 conv=noerror,sync | ssh student@openSUSE-IP-address "cat > evidence/hda1.img"

Help: on VMware VMs only, you may need to replace “hda1” with “sda1”.

Tip: Running this will take some time, so you may wish to continue with the next step while the copying runs.

This command could be repeated for each partition including swap partitions. For now, let’s accept that we have all we need.

On your openSUSE VM, list all the files you have created:

ls -la /home/student/evidence

At this stage look through some of the information you have collected. For example:

less /home/student/evidence/lsof_net_out

Examine the contents of the various output files and identify anything that may indicate that the computer has been compromised by an attacker. Hint: does the network usage seem suspicious?

Collecting live state using scripts

As you may have concluded from the previous tasks, manually collecting all this information from a live system can be a fairly time consuming process. Incident response data collection scripts can automate much of this process. A common data collection script “linux-ir.sh”, is included on the FIRE disk, and is also found on the popular Helix IR disk.

On the compromised VM (Redhat7.2), have a look through the script:

less /mnt/cdrom/statbins/linux-ir.sh

Note that this is a Bash script, and each line contains commands that you could type into the Bash shell. Bash provides the command prompt on most Unix systems, and a Bash script is an automated way of running commands. This script is quite simple, with a series of commands (similar to some of those you have already run) to display information about the running system.

Identify some commands within the script that collect information you have not already collected above.

Exit viewing the script (press q).

Run the data collection script, redirecting output to your openSUSE VM:

cd /mnt/cdrom/statbins/

./linux-ir.sh | ssh student@openSUSE-IP-address "cat > evidence/ir_out"

On your openSUSE VM, have a look at the output from the script:

less /home/student/evidence/ir_out

Use what you have learnt to spot some evidence of a security compromise.

Checking for rootkits

An important concern when investigating an incident, is that the system (including user-space programs, libraries, and possibly even the OS kernel) may have been modified to hide the presence of changes made by an attacker. For example, the ps and ls commands may be modified, so that certain processes and files (respectively) are not displayed. The libraries used by various commands may have been modified, so that any programs using those libraries are provided with deceptive information. If the kernel has been modified, it can essentially change the behaviour of any program on the system, by changing the kernel’s response to instructions from processes. For example, if a program attempts to open a file for viewing, the kernel could provide one set of content, while an attempt to execute the file may result in a completely different program running.

Detecting the presence of rootkits is tricky, and prone to error. However, there are a number of techniques that, while not foolproof, can detect a number of rootkits. Methods of detection include: looking for inconsistencies between different ways of gathering data about the system state, and looking for known instances of malicious files.

Chkrootkit is a Bash script that performs a number of tests for the presence of various rootkits.

On the compromised VM (Redhat7.2), have a quick look through the script, it is much more complex than the previous linux-ir.sh script:

less /mnt/cdrom/statbins/chkrootkit-linux/chkrootkit

Exit less

Confirm that if we were to run ls, we would be running the local (dynamic) version, probably /bin/ls:

which ls

To understand why, look at the value of the environment variable $PATH, which tells Bash where to look for programs:

echo $PATH

Set the $PATH environment variable to use our static binaries wherever possible, so that when chkrootkit calls external programs it will (wherever possible) use the ones stored on the IR disk:

export PATH=$static:$PATH

Confirm that now if we were to run less, we would be running the static version:

which ls

This should report the path to our static binary on the FIRE disk.

It is now safe to run chkrootkit[3]:

./chkrootkit-linux/chkrootkit | ssh student@openSUSE-IP-address "cat > evidence/chkrootkit_out"

Help: you may get a message in the terminal before you type the password.  You should still type the password for the script to run.  The srcipt should not take long to run.

On your openSUSE VM, have a look at the output:

less /home/student/evidence/chkrootkit_out

From the output, identify files or directories reported as “INFECTED”, or suspicious.

Also, note that the .bash_history is reportedly linked to another file.

On the compromised VM (Redhat7.2), investigate the Bash history:

$static/ls -la /root/.bash_history

What does the output mean? What does this mean for the logging of the commands run by root?

At this stage you should be convinced that this system is definitely compromised, and infected with some form of rootkit.

Save a record of your activity to your openSUSE VM:

cat /tmp/evid/invst.log | ssh student@openSUSE-IP-address "cat > evidence/script_log"

Power down the compromised system (Redhat7.2), so that we can continue analysis offline:

$static/sync; $static/sync

If you do not know what the sync command does, on your openSUSE VM, run “info coreutils 'sync invocation'” for more information.

Tell the oVirt Virtualization Manager to force a Power Off .

(In VMware: “Virtual Machine”, “Power”, “Power off”, “Yes”).

Why might we want to force a power off (effectively “pulling the plug”), rather than going through the normal shutdown process (by running “halt” or “shutdown”)?

Offline analysis of live data collection

Note that even though the bash_history was not saved (as we discovered above), we can still recover commands that were run the last time the computer was running. This is possible by searching through the saved RAM (the kcore ELF dump we saved earlier).

On your openSUSE VM, run:

sudo -u student bash -c "strings -n 10 /home/student/evidence/kcore > /home/student/evidence/kcore_strings"

The above “strings” command extracts ASCII text from the binary core dump.

Open the extracted strings, and look for evidence of the commands you ran before saving the kernel core dump:

less /home/student/evidence/kcore_strings

Now press the ‘/’ key, and type a regex to search for commands you previously ran to collect information about the system. For example, try searching for “ssh student” (press ‘n’ for next).

Windows and live incident response

The theory you have applied in this lab applies similarly to other operating systems such as Windows. Many incident response live disks (such as FIRE and Helix) also include Windows programs that can collect information about the state of a system.

Unguided task: use an incident response disk (such as Helix) to obtain information from a running Windows computer. You should collect a dump of the contents of RAM, and a list of running processes and network connections.


Take screenshots on Windows of information about processes and network connections being captured and displayed using an IR disk, as evidence that you have completed this part of the task.

Label it or save it as “LiveIR-A1”.


What’s next ...

In the next lab you will analyse the artifacts you have collected, to determine what has happened on the system.

Important: save the evidence you have collected (save /home/student/evidence to Google Drive and leave it on your openSUSE VM), as this will be used as the basis for the next lab.


[1] Note that the password for root has been reset for these exercises.

[2] In reality, if we knew the system was compromised, we would likely leave it powered off, and move straight to offline analysis.

[3] Note that it would be better to not have to include $PATH, and only use static versions. Unfortunately, FIRE does not include statically compiled versions of all of the commands that chkrootkit requires.