On 8/8/06, David Carlson <thecubic at thecubic.net> wrote: > 0) Absolutely distrust the server in question. If it appears that users > aren't logged in, don't believe it. That goes for most admin utilities > (w,users,uptime). Don't think you can delete things and restart it - you > want to reimage the OS. > > 1) Ask the ISP if they detect promiscuous mode (meaning suspicious ARP) > coming from the server > > 2) nmap or have the ISP nmap the server (from a nearby host) > > 3) Check for strange traffic with tcpdump/tshark (exclude the login > traffic port with [(tshark) -f] 'not port 22', etc). This is probably > only useful from another machine that sees all the traffic from that > machine. > > 4) Check for rootkits. http://www.chkrootkit.org > This isn't totally reliable though. If you go through this process, realize that *if* there is a compromise, the output from any of these programs run on the local system is suspect. What's more, by going over everything with a fine-tooth comb, you may have eliminated any timestamp evidence that may have been on the system. Unless of course, you mount your [reiser-only??] partitions noatime, in which case there are no access timestamps anyway. > 5) Sniff (or, better, have the ISP sniff and deliver) some outgoing > traffic and analyze it with wireshark GUI. This should be your first step in the case you've described. Use a separate system and sniff both incoming and outgoing traffic between your server and the ISP using tcpdump or wireshark (formerly ethereal) in promiscuous mode. Examine this traffic for any connections to overseas IP blocks (whois 1.2.3.4), IRC traffic, and suspicious traffic that should be there. For instance, if you did not have FTP running at all, and you see FTP commands over the wire, destined for port 34343, you likely have a compromised server. Ideally, you find nothing and find something like an errant process, as mentioned by another poster. > If any of the tests show something wrong, have the ISP cut power (don't > run 'halt') forcefully. Save the hard drive image somewhere for forensics > (don't boot off of it). You will likely have to rebuild the server - the > only thing you should copy over are user files that have been examined. If you find nothing, rpm and apt both have functionality to verify the validity of installed software packages, ensuring MD5 checksums match up with the original installation. When run from a bootable CD, this can serve to validate your system's integrity. In any case, once you rebuild/restore "normal" service, investigate using tools like AIDE, OSSIM, or Samhain, which are all host-based intrusion detection systems. You may have heard of tripwire, another partially-free example of an HIDS.