My experience with being cracked
Tyler
About 9 months ago I started doing security consulting work for my ISP. In exchange for free internet access, I would look over their machines and check for security flaws, cracked accounts, etc. They had a pretty typical setup for an ISP. The webservers were running Red Hat and Apache, the email server was an NT machine running Imail and their two DNS servers were running Red Hat and OpenBSD.
The first machine I had access to was a webserver/production machine running Red Hat 5.0. The machine was used to write CGI programs for their clients and to test the programs out.
When I first got on the machine, everything looked normal except for the logs. For some reason, utmp and wtmp looked like they had been corrupted. Whenever I would issue a 'who' or 'last' command, I would get garbage back. I suspected a trojan program, especially after the files were zeroed and the output from the programs was still corrupted.
Not knowing which trojan programs or rootkits could have been used at this point, I poked around the system some more. When I got to the passwd file, unshadowed of course, I saw the following entry: moof::0:0::/root:/bin/bash.
The machine had been hacked.
I looked through the file system some more and found a "..." directory in /root, root's home directory. In it were three files. One was a sniffer, one was a backdoor program and the third was a text file full of usernames and passwords that had been sniffed. After examining the time stamps of the files, it looked like they had been last modified almost a year before.
The programs looked alot like the ones that come with Linux Rootkit IV, so I downloaded the source and examined the code to see how it worked and which programs it trojaned. Of course, one of the programs is trojaned was /bin/login. I compiled the trojan for /bin/login on the hacked machine and ran a hex editor to see exactly where the backdoor password was kept. After finding it, I ran the hex editor on a copy of the /bin/login that was in use and found what I thought was the backdoor password. To test my findings, I logged onto another machine, ran a telnet session to the hacked machine and logged in as root with the backdoor password. Bingo! I was now root. I tried this again with rsh and was able to execute commands as root.
I emailed my findings to the systems admin and the owner of the ISP, including the backdoor password and how to use it, with the suggestion that they should backup everything, wipe the machine, and load a current version of Red Hat (6.0 at the time) with the latest patches. They replied that they would look into it.
The next machine that I had access to was the main web server. This was another Red Hat 5.0 box running Apache. As soon as I logged on I checked the passwd file, but did not find a problem. However, I got the same garbage whenever I ran 'last' or 'who'. I looked in the same spots as before but did not see any files from a rootkit. When I downloaded /bin/login though, I found a backdoor password (different than the other machine) and was able to login as root again.
It looked like it was a different cracker that had broken into this machine than the last one. Everything was installed cleaner and nothing was left lying around. I also couldn't find any files from the rootkit, with the exception of the trojaned programs. On a hunch, I went into /dev and looked for any unusual
files. I didn't see anything unusual with 'ls', but when I ran 'file *', I saw a few text files that had not shown up with ls. These were the configuration files for the rootkit which showed which processes, IP addresses, directories and files that the trojaned programs should hide. With this information I was able to discover where the rootkit source and other files were and where the cracker was coming from.
Of course, after examining the timestamps on the source files left on the machine I found that it had been broken into a few months earlier. I emailed what I had found, including the backdoor password, to the systems admin and owner. I also sent the same suggestion as before, to back everything up, wipe the machine and load the current version and patches. I got the same response: "Thanks, we'll look into it."
The ISP never fixed the machines. Even though I had shown them what had happened and had given them exact instructions on how root could be compromised, they still decided not to do anything. Their reasoning was that since the machines had been broken into so long ago and that it didn't look like the cracker had been back in awhile, it wasn't worth the time, effort and possible loss of production time to fix the machines.
This was frustrating for a number of reasons. First, I had spent a number of hours researching what exploits and rootkit had been used to break into the machines, finding the backdoor passwords and documenting what had occured and how to reproduce it.
Second, as a security person, this was going against all that I had learned. Whenever you are hacked, the first thing you do is back everything up and reload the operating system with the latest patches. This is the only way to be totally sure that you are safe.
The last reason I was frustrated was that this was the ISP I used. I bought things online over them, I had email accounts through them and I used them for day to day things on the internet. They had been compromised and did not want to do anything about it. There was already evidence that the cracker had grabbed passwords and chances are that he had sniffed other things, like email, too. My confidence that the ISP was doing what they could to provide a reliable service, including keeping my information private, had just fallen. As far as I know, the ISP has still not fixed any of the problems on the machines and they are still running the trojaned programs.
| 