I found a trojan running on my server today - rootedoor (http://vil.mcafeesecurity.com/vil/content/v_128116.htm).
Couldn't find much information about it on the web. If you know anything specific about it (not general speculation on rootkits, trojans, etc.), I would appreciate you sharing it with the list.
I detected it running as one of the last two processes listed by 'ps ax' just before a reboot to install a new kernel; the reboot seems to have elimiated all traces of it from the system.
I found a number of missing or corrupt perl modules on the system, but that may not be related, it's an old system and I've had some perl issues before.
This _is_ the system that suddenly lost all it's files Sunday night, so it could have been hacked before.
On Fri, 25 Feb 2005, Jonathan Hutchins wrote:
I found a trojan running on my server today - rootedoor (http://vil.mcafeesecurity.com/vil/content/v_128116.htm).
I detected it running as one of the last two processes listed by 'ps ax' just before a reboot to install a new kernel; the reboot seems to have elimiated all traces of it from the system.
Any decent rootkit will overwrite ps, ls, less, more, netstat and a cast of thousands. If you don't re-install these, your system tests have little value.
Regards,
-Don
On Friday 25 February 2005 06:56 pm, Don Erickson wrote:
Any decent rootkit will overwrite ps, ls, less, more, netstat and a cast of thousands. If you don't re-install these, your system tests have little value.
You should, perhaps, follow the link I provided and read about this particular trojan.
I did a CRC check against ALL of the system files. They're fine. I checked RPM before I used it to check the rest of the system.
RPM's a great tool for a lot of things, including verifying system integrity. People who don't understand it and have been frustrated by the fact that it, itself, doesn't say resolve dependencies or download files tend to talk a lot of ignorant trash about it, but it does what it does quite well.
It's VERY hard to hack an RPM system in such a way as to conceal tampering with files within the packages. Not impossible, but hard in a way that the low-level simplicity of rootedoor tends to contraindicate.
I've dealt with a couple of systems that got rooted several years ago. "login" is one of the favorite files to replace, or move.
As I mentioned, I also have an image of the system, taken when it was running normally. I know what files I can trust, and I have used them to establish a chain of verification that gives me good confidence in the system's current state.
On Fri, 25 Feb 2005, Jonathan Hutchins wrote:
I did a CRC check against ALL of the system files. They're fine. I checked RPM before I used it to check the rest of the system.
RPM's a great tool for a lot of things, including verifying system integrity. People who don't understand it and have been frustrated by the fact that it, itself, doesn't say resolve dependencies or download files tend to talk a lot of ignorant trash about it, but it does what it does quite well.
It's VERY hard to hack an RPM system in such a way as to conceal tampering with files within the packages. Not impossible, but hard in a way that the low-level simplicity of rootedoor tends to contraindicate.
The problem with said philosophy is that the system had to be hacked before the rootkit was installed. You are working under the assumption that this is all they did ...
Also, you should really read the rpm code sometime. It is a tool to verify, but if it is the only tool you are using ... you are only fooling yourself. If you had a tripwire database to also compare with I might be inclined to agree that you have a little bit of time before you need a re-install. Still, you need to re-install.
Paranoid sysadmins are good sysadmins. Yes, they are out to crack your systems.
//========================================================\ || D. Hageman dhageman@dracken.com || \========================================================//
On Saturday 26 February 2005 02:31 am, D. Hageman wrote:
The problem with said philosophy is that the system had to be hacked before the rootkit was installed.
It's hardly a rootkit. It's a back door. That's all.
You are working under the assumption that this is all they did ...
No, I have worked toward the conclusion that was all they did, given the evidence of the condition of the sytem.
I belive I've said rather clearly, and more than once, that I have taken multiple approaches to fixing this. RPM, comprehensive as it is, has not been the only tool.
If you had a tripwire database...
Annoying as it is, I do.
Still, you need to re-install.
Well, I have reinstalled a good deal of the system, as I clearly said in earlier messages. I think the blind "oh ya gotta wipe the system" response is unjustified in this case, and I had hoped to avoid a lot of useless speculation toward that end.
I have dealt with hacked systems where that was necessary, I know what they look like. Rather early in the age of the internet I dealt with an open FTP server that actually _had_ been rootkitted. Wipe and restore was not necessarily the only answer in that case either, but it was the best.
What I'd really like is some extra eyeballs looking for information on rootedoor and what exploits are typically used to install it.
On Sat, 26 Feb 2005, Jonathan Hutchins wrote:
On Saturday 26 February 2005 02:31 am, D. Hageman wrote:
The problem with said philosophy is that the system had to be hacked before the rootkit was installed.
It's hardly a rootkit. It's a back door. That's all.
Well, if it wasn't installed by root, then what user DID install it? That would be a pretty healthy clue.
What I'd really like is some extra eyeballs looking for information on rootedoor and what exploits are typically used to install it.
What kernel version were you running? You're assuming that rootedoor doesn't give a root shell, which looks optimistic from this chair. And, in my opinion, the assumption that a specific cracker tool has "typical" exploits used to install it is flawed. It's all a moving target.
Regards,
-Don
On Saturday 26 February 2005 09:56 am, Don Erickson wrote:
What kernel version were you running?
2.4.20-37.7.legacy. Now on 2.4.20-42.7.legacy.
You keep accusing me of making assumptions. I believe you are, in fact, making rather a lot of assumptions given the fact that you haven't examined the system. I have. I have made conclusions, not assumptions.
On Sat, 26 Feb 2005, Jonathan Hutchins wrote:
2.4.20-37.7.legacy. Now on 2.4.20-42.7.legacy.
Okay. If I am reading the info correctly, 2.4.20-37.7.legacy was compiled in Sept 2004, before the uselib() root kernel hole was discovered in January, 2005. http://www.isec.pl/vulnerabilites/isec-0021-uselib.txt.
There was also an awstats exploit patched on Feb 16 or so, that allowed remote execution of commands by the apache-owner, "nobody" on Redhat, I believe.
Now, it may be presumptuous of me to assume your box has been rooted, but, given the above information, that's how it looks to me. I did ask you what user had installed the backdoor that you're asking about, but didn't receive an answer. From your examination, you have concluded that your box was not rooted. Given the known security holes that were present in your system and the information that you have given us, my conclusion is that root access was absolutely possible, and it would be foolish, IMO, to ignore this probability. Also, you told us that this box had all of its files deleted, and nobody but root can do that.
I realize that having crackers rooting (figuratively if not literally) around on your system isn't exactly a prescription for a pleasant demeanor. It is not my intention to annoy you, I'm only trying to help. Maybe your box wasn't rooted. But, if not, which user did install the backdoor? And how were the files deleted?
You keep accusing me of making assumptions. I believe you are, in fact, making rather a lot of assumptions given the fact that you haven't examined the system. I have. I have made conclusions, not assumptions.
Okay. You have more information than I, but I'm still drawing a different conclusion than you from the information that I have seen.
Regards,
-Don
At the risk of prolonging the speculation:
On Saturday 26 February 2005 11:16 am, Don Erickson wrote:
Okay. If I am reading the info correctly, 2.4.20-37.7.legacy was compiled in Sept 2004, before the uselib() root kernel hole was discovered in January, 2005. http://www.isec.pl/vulnerabilites/isec-0021-uselib.txt.
There was also an awstats exploit patched on Feb 16 or so, that allowed remote execution of commands by the apache-owner, "nobody" on Redhat, I believe.
Clearly, there was some vulnerability, and I agree that those are two very likely candidates. Of course, most vulnerabilities are at most potential DoS potenitals in that someone can, at most, crash the system. A true back-door vulnerability is rarer, and I will continue looking for possibilities.
Now, it may be presumptuous of me to assume your box has been rooted...
While it's clear somebody took advantage of a vulnerability to install a trojan, I think that "rooted" implies a lot more to you than what actually happened to this system.
I did ask you what user had installed the backdoor that you're asking about, but didn't receive an answer.
That's because I don't have one. The only thing that was "installed" appears to have been a running process (rootedoor), and when I killed it with a reboot it doesn't appear to have left any traces.
We'll probably never know what happened to the system the day it was "cleared". I have good reason to believe that except for vulnerabilities, the restored image was uncompromised, and through exhaustive examination I belive that the system is currently uncompromised. Thirty-six hours of running with no suspicious behavior confirm this.
I've seen compromised systems, I've wiped systems where I felt there was a possibility that I couldn't trust them. I have checked all of the specific items you suggested, and I don't feel there's any need to panic and wipe the thing.
If you can think of other items to check, I really do appreciate that, I'll check and let you know.
On Sat, 26 Feb 2005, Jonathan Hutchins wrote:
While it's clear somebody took advantage of a vulnerability to install a trojan, I think that "rooted" implies a lot more to you than what actually happened to this system.
"Rooted" to me means that an unauthorized person has gained root access. Since your disk was wiped, this frankly has _got_ to be the case. To me, "trojan" refers to the method of replication and has nothing to do with the function of a program. The question to me is, does the "rootedoor" "trojan" provide a root shell if running as root? The answer is certainly yes, although I can't really find any information about it.
We'll probably never know what happened to the system the day it was "cleared". I have good reason to believe that except for vulnerabilities, the restored image was uncompromised, and through exhaustive examination I belive that the system is currently uncompromised. Thirty-six hours of running with no suspicious behavior confirm this.
I suggest remote logging so that you will have some forensics if it ever happens again.
I don't feel there's any need to panic and wipe the thing.
I didn't suggest that you should. That's your call, not mine. Even if you do wipe it, there are always user programs that will be replicated. I just think that you should consider using known safe utilites to inspect the machine, like from a Knoppix CD. Make sure nothing is running using netstat that you can't explain, or isn't necessary. Check all your cron entries, sometimes backdoors only run on alternate Tuesdays from 4 till 4:15. I don't really do the RedHat thing, but update all the programs that have security advisories. Reinstall login, ssh, telnet, ftp, apache, anything that has a port open to the outside. Check all binaries in cgi-bin. Actually, reinstall everything if it's an old box. Run nessus against it from inside and outside. These rootkits are pretty damned sophisticated. Anybody who really knew what he was doing, you'd probably never find out that he was there in the first place.
Having said that, since whomever got in wiped the disk, which is not the act of a "professional", it's much more likely that your backup image is okay. My opinion doesn't make it so, however. Maybe it was a first-timer who got nervous about being able to cover his tracks. Who knows?
*If* you restored a pre-crack backup and *if* you plugged the hole(s) that were exploited, your system is safer now than before the crack. Those are two big ifs, since you don't know exactly when and how it was compromised. But the only way to have a perfectly secure box is to unplug it and padlock it in a bunker somewhere. Maybe with Dick Cheney.
I'm no "security expert". I subscribe to the debian-security mailing list and read the occasional article and treatise. I try to stay on top of possible root exploits, but I'm more like the mule who gets whacked by a two by four and gains some knowledge about the nature of lumber in the process.
Regards,
-Don
First, let me make it clear that the disk coming up blank happened first, and I found rootedoor running several days afterward.
On Sunday 27 February 2005 09:56 am, Don Erickson wrote:
"Rooted" to me means that an unauthorized person has gained root access. Since your disk was wiped, this frankly has _got_ to be the case.
Not necessarily. Hardware errors are possible, but as the hardware has run without errors since, that's unlikely. Client error is also not only possible, but likely. Finally, there are exploits that, like a buffer overflow, don't give crackers useful access to the machine, but merely destroy the system. Someone may have exploited a CGI weakness to delete all the files without having arbitrary access.
The question to me is, does the "rootedoor" "trojan" provide a root shell if running as root?
Read up. I posted the links to what information is available.
It appears to ONLY provide a doorway, and probably one that implants itself and then waits for the exploiter to come along and use it. I caught it right after it was launched, and I'm pretty sure I caught it before anybody used it.
I have already followed some of your suggestions. Log file analysis/extraction that is delivered to some other machine can be even more usfull than raw remote logging, as it reduces the data to a more useful set. It can also be used via email when remote logging isn't an option.
Since the machine's in Tucson, doing something like booting from a Knoppix CD isn't possible, but it's not actually necessary in order to have good confidence in the integrity of the system.
One of the concepts that doesn't seem to have been caught in this discussion is that you can take a hash of your system files just like a PGP signature, and the verify that hash, either against a stored hash or the hash of a known good copy. That allows you to do a fairly easy system sweep, looking for tampered files. If the hash verifies, you really can be reasonably certain that the file has not been tampered with and does not need to be replaced.
Check all binaries in cgi-bin.
How would you do that? Understand that most of the web site is writen by the client, so his selection of CGI programs is something I don't fully control. I've scrutinized some of them, but sometimes he puts test scripts on it that I know nothing about. Any good methods for screening them? Mark 1 eyeall of the code is the best I've got, and at a certain level of volume and complexity it becomes less reliable.
These rootkits are pretty damned sophisticated.
Again, (and again, and again), this doesn't seem to be a "rootkit". It's just a backdoor, not a kit. I have seen some of the moderately sophisticated ones, and this ain't.
Anybody who really knew what he was doing, you'd probably never find out that he was there in the first place.
Unless you happened to run "ps ax" at a critical moment...
On Sun, 27 Feb 2005, Jonathan Hutchins wrote:
Someone may have exploited a CGI weakness to delete all the files without having arbitrary access.
Write access to the entire disk is about as arbitrary as it gets, it seems to me. Do you have a link to an example of such an exploit? I've never heard of one, and can't imagine how it might work.
Since the machine's in Tucson, doing something like booting from a Knoppix CD isn't possible, but it's not actually necessary in order to have good confidence in the integrity of the system.
I was just trying to think of an easy way to check the system with "known clean" utilities.
One of the concepts that doesn't seem to have been caught in this discussion is that you can take a hash of your system files just like a PGP signature, and the verify that hash, either against a stored hash or the hash of a known good copy. That allows you to do a fairly easy system sweep, looking for tampered files. If the hash verifies, you really can be reasonably certain that the file has not been tampered with and does not need to be replaced.
I'd be interested in learning more about this.
Check all binaries in cgi-bin.
How would you do that? Understand that most of the web site is writen by the client, so his selection of CGI programs is something I don't fully control. I've scrutinized some of them, but sometimes he puts test scripts on it that I know nothing about. Any good methods for screening them? Mark 1 eyeall of the code is the best I've got, and at a certain level of volume and complexity it becomes less reliable.
Maybe not. Cgi-bin programs should run as the apache user, so it might not be as critical of an issue.
Again, (and again, and again), this doesn't seem to be a "rootkit". It's just a backdoor, not a kit. I have seen some of the moderately sophisticated ones, and this ain't.
I understand that there doesn't "seem" to be a rootkit. The entire mission of any rootkit is to obliterate the evidence of ingress to a hacked box and also hide it's own existence from the owner of the machine, allowing the perpetrator to utilize the installed remote access for whatever purposes he chooses. Obviously, this is the worst case scenario, but I have shown you that such security holes that _could have allowed remote root access_ existed on your box at the time(s) that the disk was wiped and the backdoor was installed. By somebody.
I'm not telling you that your machine has a rootkit installed, or even a backdoor still installed, but it's what you've got to look for. You say the box is clean and you trust the utilities, and maybe it is clean, and all the utilities are fine. I certainly can't disprove this.
I'm just saying what I would do if it were my box. I wouldn't base any decision as to the integrity of the box on the output of the utilites on the box itself.
Regards,
-Don
On Sun, Feb 27, 2005 at 09:28:36PM -0600, Don Erickson wrote:
I'm just saying what I would do if it were my box. I wouldn't base any decision as to the integrity of the box on the output of the utilites on the box itself.
Bingo.
I understand and appreciate the value of the hash information, both from the rpm database and from tripwire.
What neither covers (well, maybe tripwire would, I don't know, I guess it depends on how it's used) are changes to files that have been *added*. ie, you might be able to track changes in ls or ps or other known system binaries, but if an executable file were added elsewhere to the filesystem, would you know it just by looking at a list of checksums of known files? You wouldn't, because you wouldn't have a checksum of its predecessor file. You'd have to have a completely comprehensive look at *all* files, and the rpm verify doesn't give that.
So, have you looked for files set to be executable elsewhere in the filesystem, especially files owned by root or most especially files setuid root hidden in some out of the way directory? maybe you have, I don't recall reading that, though.
And I feel your pain about the lack of control you have over the cgi on the box, but a cgi program that gives user-level executable access to a setuid-root binary is all it would take.
I've started to put cgi behind proxies to try to give myself another layer between me and the bad guys.
Oh, and about booting the thing into KNOPPIX, why not? Mail the guys a CD and tell 'em to stick it in and fire it up. If they aren't up to configuring the network themselves, write yourself a script to do it and put it on the disk (from the knoppix-cheatcodes.txt file):
From Version 2.1 and up, a file called "knoppix.sh", if located in the toplevel KNOPPIX directory on CD, will also be executed at startup. This makes ist easier to create customized versions without having to change anything on the compressed filesystem KNOPPIX/KNOPPIX.
Put a copy of (one of) your ssh public key(s) on there and you're all set.
What are your favorite sources for realistic vulnerability warnings?
Don, you seem pretty well informed on potential risks. I find that the real difficulty is not in getting information, but in sorting the useful information from the chaff, FUD, and panic. Knowing that a potential back door for open code execution in a common utility has been publicised is a LOT more useful than knowing about an obscure potential for a buffer overflow that could knock out some obscure and rarely used service.
-
D. Hageman -
D. Joe -
Don Erickson -
Jonathan Hutchins