We are not living in a perfect world, and it is possible for a VPS/Servers to get hacked. However, we can find the culprit and make sure it won’t happen again. In this article we will learn some techniques and tools we can use to investigate our VPS/Server if we suspect they’ve been compromised.
Introduction
VPS/Servers can be compromised as a result of various factors: weak passwords, weak iptables rules, older versions of software with known exploits, and more. If your VPS/Server has been compromised, do not panic. Panic will lead to poor decisions, and then the situation could become worse.
Instead, try to understand what happened and make sure your VPS/Server will not get compromised again in the same manner. The main idea of this article is simple: learn from your mistakes and don’t make the same mistakes twice.
This article is the first in our “VPS/Server Investigation” series. At the end of this series, we’ll learn how to track intruders on a compromised VPS/Server, and to backup our data and packages.
In this first article we will cover the things we can do before going into rescue mode (which is covered in the second article).
The VPS/Server used for this article series was running Ubuntu 8.10. However, the steps demonstrated will be similar for other Linux distributions.
Important Warning
Before you do anything, you need to make an important decision—do you plan to involve law enforcement and prosecute the attacker? If the answer is yes, you should leave the compromised system alone and make no changes to it.
Any changes you make post-attack could complicate and taint the evidence. Because of that, a common policy is to power-off a system once a compromise is detected, and then to leave it off until law enforcement is ready to investigate.
Checking Network Connections
Let’s start our investigation by checking our VPS/Server’s network connections…
Usage
netstat -an
This command helps you check for any backdoors which have been opened on your VPS/Server.
netstat -an Active Internet connections (servers and established) Proto Recv-Q Send-Q Local Address Foreign Address State tcp 0 0 0.0.0.0:22 0.0.0.0:* LISTEN tcp 0 0 0.0.0.0:80 0.0.0.0:* LISTEN tcp 0 0 0.0.0.0:25 0.0.0.0:* LISTEN tcp 0 284 1.2.3.4:6697 5.6.7.8:34506 ESTABLISHED
In this case we see port 6697 is open — that port is commonly used by IRC servers. That’s not a good sign, unless we’re running our own chat server. We can sniff any connections to that port using tcpdump. For more info on tcpdump, check here.
tcpdump src port 6697
This will capture all the packets with destination port 6697.
Using lsof
lsof is a command line utility which stands for “list open files”. It is used in many Unix-like systems to report a list of all open files and the processes that opened them. By default Linux treats everything, including devices, as a file. This makes lsof a very powerful tool.
For example, we can use lsof to see what user has a particular file open:
sudo lsof /etc/passwd
If we discover the user name under the intruder’s control, lsof can be used to display all his running processes:
sudo lsof -u hisUserName
lsof also helps us check our network connections. Investigating various aspects of our VPS/Server with multiple tools is important — if we suspect the system is compromised, we can’t be sure which commands will provide reliable results. Also, lsof provides some options which netstat does not.
To list all the open IP sockets associated with your VPS/Server’s SSH server run the following command:
sudo lsof -i:22
Summary
In this article we learned some techniques that can be used to discover backdoors and track intruders on our VPS/Server.
This will help us avoid a repeat of whatever situation or mistake led up to the compromise, so we’re less likely to get hacked again in the same way.
In the second part of this series we will learn how to investigate our VPS/Server in rescue mode.
Author: Ismail @ VPS/ServerHost