login
Search
Hi all, Andrew was looking for someone to run a linux-next tree that just contained the subsystem git and quilt trees for 2.6.x+1 and I (in a moment of madness) volunteered. So, this is to announce the creating of such a tree (it doesn't exist yet) which will require some (hopefully) small amount of work on the part of subsystem maintainers. Those interested in discussion about this are encouraged to join the linux-next@vger.kernel.org mailing list. The first things I need from the subsystem maintainers (you know who you are) are a contact address (a list address is fine) and at least one git branch or quilt series that contains all the things you want to see go into 2.6.26. I am happy for there to be multiple branches/series (in fact there probably will need to be in some cases where there are dependencies on others work). The tree will be based on the current version of Linus' tree but you may specify an earlier branch point if you need to (hopefully not - but more likely for quilters, I guess). I hope to recreate this tree every day automatically. In order to do this, any tree that has a conflict will be dropped from that days tree. The maintainer will be notified. I hope to provide some clue as to what the conflict is with, but probably not initially. I will attempt to build the tree between each merge (and a failed build will again cause the offending tree to be dropped). These builds will be necessarily restricted to probably one architecture/config. I will build the entire tree on as many architectures/configs as seem sensible and the results of that will be available on a web page (to be announced). This is just a start ... Comments? [I suspect that Andrew's dream includes actual time to dream :-)] --=20 Cheers, Stephen Rothwell sfr@canb.auug.org.au http://www.canb.auug.org.au/~sfr/ For reference - Andrew's original musings: I perceive the following problems with kernel development: - The 80-odd subsystem trees are ...
Actually, it sort of does. If you look here: http://www.kernel.org/pub/linux/kernel/people/jejb/merge-tree/ You'll find a merge candidate tree that builds nightly from everyone's git and quilt trees. I'm using it to track merge conflicts (so I only build the patch, I don't check it compiles). You're welcome to the scripts that do this: http://www.kernel.org/pub/linux/kernel/people/jejb/build.pl And the config file that runs it: http://www.kernel.org/pub/linux/kernel/people/jejb/merge-tree-build I don't plan to do much more than keep it building to check conflicts, Actually the experiment with the -mc tree shows that most of the conflicts are trivial in nature (usually docbook stuff or feature-removal.txt stuff), so you can do a trivial triage by hand. You can't automatically drop them (well, not unless you want to end up dropping half the trees). The other problem is that we actually maintain deliberate conflicts with a last person to merge fixes it type attitude. Again, it's usually in minor areas, and the fixups are fairly trivial, but it illustrates why conflicts can't be a reason to drop a tree, you have to maintain some sort of automatic fixup (at least I had to with the -mc tree). The reason we do this is that it would give the maintainers a nasty web of included trees (which is almost impossible for the quilt trees anyway) Yes, this is the bit I've never dared do ... principally because it's such a time sink. James --
Hi James, On Mon, 11 Feb 2008 19:36:49 -0600 James Bottomley <James.Bottomley@HansenP= Well I did a trial run with the 40 git trees in the latest -mm and got only 6 conflicts (which actually were not trivial, unfortunately). Of I was hoping to be able to automatically find the other tree involved in a conflict and point both maintainers at the problem. Also, there is I am hoping (one of Andrew's bugbears) that over time we will end up with several branches from git using trees (the vast majority) most of which are completely contained within their own subsystem and don't depend on anything but Linus' tree. The conflicting and dependent branches will be merged later in the sequence. Thus we will end up with a large amount of the tree becoming stable as the merge window approaches. (Yes, sometimes I have a couple of lackeys who already do this stuff :-) Thanks for the comments - I will keep them in mind as I sink into the abyss= :-) --=20 Cheers, Stephen Rothwell sfr@canb.auug.org.au http://www.canb.auug.org.au/~sfr/
Hmm, OK of the 46 I pull in, it's only xfs causing difficulties at the moment. Pre 2.6.24 being declared, we had about 10 fixups, all of which It's frequently just not possible. There are a lot of global API changes that sweep across subsystems. For instance, I had a trivial Right .. it's really not possible to work without an infrastructure that We can't really do this. We don't work in a utopian my area only environment. Almost every release cycle involves dealing with patches that cross subsystem boundaries. Often we try to minimise the problems by taking patches from different subsystems via our trees to keep the dependencies straight, but it does lead to conflicts. Most of them are ones we're aware of, and easily resolvable as the trees go in in the merge window. So any infrastructure that builds a unified tree has to be aware of them too. James --
Note that a lot of these are already in the MAINTAINERS file. But for the record, here's mine, in the order they need to be pulled from. Driver core: kernel.org/pub/linux/kernel/people/gregkh/gregkh-2.6/gregkh-01-driver/ PCI: kernel.org/pub/linux/kernel/people/gregkh/gregkh-2.6/gregkh-02-pci/ USB: kernel.org/pub/linux/kernel/people/gregkh/gregkh-2.6/gregkh-03-usb/ These are all quilt trees, with the series file in the directory for the order of the patches, and a README saying what kernel version they have This is going to get really interesting, especially when (not if) we do more global api changes. Look at the last round of kobject changes. That touched a lot of different places, and other trees ended up not building because of it, because I changed apis and they had added new code based on the old apis. I think the only way to fix this is not going to just "drop the tree" like you are suggesting, but to let both people know (the person who caused the change, and the person who's tree broke after the merge), and then either add a "fixup patch" for the build like Andrew has been doing, or disabling something from the build section. As I know I'm going to be changing more driver core apis[1] this week, I'm sure we will get a very good set of examples of this for you to see in action :) Good luck, greg k-h [1] Hopefully the "multiple drivers for a single device" feature people have been asking for for years will be landing soon, of course the number of odd places in the kernel that made the assumption that we could only have one driver per device is causing lots of fun... --
On Mon, 11 Feb 2008 20:21:33 -0800 in my experience, the only chance you have is doing API changes as first in the set of changes, and then hoping (making) all other trees use the new APIs. Any other order just turns into an impossible mismash. I can see the point of doing an LKML annouce of the new API after the series is done, so that all other maintainers have a chance to fix their trees (this of course is only for new occurances of the old api showing up; the api change series should convert all existing users) -- If you want to reach me at my work email, use arjan@linux.intel.com For development, discussion and tips for power savings, visit http://www.lesswatts.org --
It would be very nice to have a separate tree with _only_ API changes that could be frozen well before Linus' merge window opens. It should be a requirement that maintainers use this tree as a basis for testing API changes and even test that their own changesets were properly integrated with the changed APIs. As it is, we tend to get the order entirely wrong: API changes typically get merged _after_ the maintainer changes have been merged, and so no API changes are ever really integration tested. Cheers Trond --
The other way that might work in some circumstances would be if we tried a little harder to avoid API changes that don't involve an interface naming change. That is, instead of adding a new parameter to a function, and then having to sweep through all of the trees to catch all of the users of siad function, we could instead add a new a new interface, __deprecate the old one, and then give enough time for trees to adapt, you can avoid needing to do flag day transitions. If the old interface is __deprecated at the beginning of the merge window, and then disappears at the very end of the merge window, that's plenty of time for the subsystem maintainers to move to the new interface. This doesn't always work, of course (for example, if we make a fundamental change in how some critical low-level data structure is locked). But every little bit that we can do to avoid the tree integration pain would be a win. Regards, - Ted --
From: Theodore Tso <tytso@mit.edu> Deprecate is garbage, shit hangs around in the tree forever and people just turn off the warnings. Clean sweeps work much better, albeit with some merge pain, we'll cope. --
On Mon, 11 Feb 2008 22:02:22 -0800 (PST) I agree with that. There is maybe a middle ground in this -next idea; as very first part of the series, the new api gets added, current users converted and api marked __deprecated. Then there's a second part to the patch, which is a separate tree, which gets added at the very end, which removed the old api. Both will go in at the same merge window, and the next-meister needs to track that no new users show up... but the final tree allows this to be done somewhat more gentle. Doesn't work for API changes that just change the API rather than extending it, and doesn't solve the dependency issues. So I still think a cleansweep works best in general, but I suspect Andrew just disagrees with that. -- If you want to reach me at my work email, use arjan@linux.intel.com For development, discussion and tips for power savings, visit http://www.lesswatts.org --
Yes, that's exactly what I was suggesting. The __deprecate only lasts for the merge window, and we remove the old API at the end of the merge window. So it's only there for a very short time, and it's only there to make the cleen sweep a little less painful --- not one where "shit hangs around in the tree forever". - Ted --
yup, in many cases that's pretty easy to do. But people dive straight for the end result, which of course provides the cleanest outcome but isn't really very real-wordly. Another thing we could do when these things happen is all gang up on Linus and ask him to merge the API change into mainline. Because often the change can be done in two stages: 1) change the interface then 2) add the code in the callee which _uses_ that changed interface. Part 1 is an obviously-safe do-nothing change and fixes the merge problems. Part 2 is at a single site and can be merged in 2.6.x+1. There are lots of well-known things we can do to simplify Stephen's life. But first we have to have the motivation to do that thing. If linux-next proves to be useful and developers are seeing good testing and bug-reporting coming out of it then I expect people will be happy to make such accommodations. Let's see if we can get to that stage first. A fully running and successful linux-next won't change outcomes a lot, I expect. It will help to pick up on obvious bugs and build errors and bisection breakages prior to them hitting mainline. But we fix those things by -rc1 or -rc2 anyway. So it's just a time-shift, causing no great change in the final 2.6.x. linux-next will do little to solve our (IMO) largest problem: unfixed regressions and bugs. otoh it will free up a lot of my time and hair-tearing, so I can devote more attention to the bugs. (where "attention" usually= "sending irritiating emails") --
I have an alternative (with help from Paul Mackerras). We have a branch in linux-next that is stable and only progresses forward i.e. is never rebased. Global API changes/introductions that are of type 1 above go into this branch (after appropriate etc). Fixes to them go in as well. People can base development on this branch. But ... they can only really do that *if* we can guarantee that part of linux-next will be in Linus' tree very early in the next merge window. So here is the had part ... We need to ask Linus to promise that he will pull the stable branch from linux-next first in the merge window. For that to happen, I would expect that Linus would also review and sign off (or ack) these commits to the linux-next tree. The rest of linux-next would always be based on the stable part plus Linus' current tree. Will this fly? I am hoping that this time shift will allow more time for bug And may be you can get more sleep and dream up some more schemes :-) --=20 Cheers, Stephen Rothwell sfr@canb.auug.org.au http://www.canb.auug.org.au/~sfr/
I doubt it. See David's previous post about constantly rebasing his tree over the lifetime of development during a -rc series. I do the same thing, dropping things in the middle, re-arranging stuff, handling merge issues that came into Linus's tree, etc. I wouldn't want to change our current development flow to force such a thing on Linus, or us right now. My main point in all of this was just that you are going to have to help with some of these merges by hand, adding patches to get things to build, and not just blindly dropping trees when things go wrong, as you originally stated. After that, we can continue doing what we have been doing in the merges with Linus, as that is working very well (see our rate of change metrics for proof of that.) thanks, greg k-h --
Changing the commits in git in anyway changes their ID, which has the same effects as a rebase. With this idea, Linus only has two choices: 1. pull the entire set of linux-next changes whether he likes them or not, because he's going to get them either from the linux-next tree or someone elses tree which is based upon that. 2. don't pull the changes, nor anyone elses tree if he hates the changes in linux-next. So really, Linus needs to ack the changes _before_ they go into linux-next. -- Russell King Linux kernel 2.6 ARM Linux - http://www.arm.linux.org.uk/ maintainer of: --
Hi Russell, On Thu, 14 Feb 2008 08:14:05 +0000 Russell King <rmk+lkml@arm.linux.org.uk>= t. This is exactly what I suggested (or meant to) *except* that I want it to only apply to the stable branch. I intend that the stable branch of linux-next will never be rebased and so is suitable for others to base their trees off. The master branch will be continually rebased as the subsystem trees change over time. Originally, I assumed the stable branch would be for our "usual" API changes, but it appears we are not having any more of those. :-) However, I see an argument for attempting to stabilise possible conflicting changes get Linus' review/ack and add them to the stable branch. Linus suggested that such changes should go into an independent tree that everyone could pull into their trees with the full confidence that that tree would be merged into Linus' tree when the merge window opens. I am suggesting that that tree be the stable branch of linux-next. I know I haven't thought through all the consequences of this, so discussion is encouaged. --=20 Cheers, Stephen Rothwell sfr@canb.auug.org.au http://www.canb.auug.org.au/~sfr/
It's not that we should _never_ have them, it's that they shouldn't be "business as usual". I'm happy with them being a "a couple of times a year". I'm not happy with them being "once or twice for every release cycle". That's the big deal for me. If we have a big flag-day that affects a lot of drivers (or architectures) once or twice a year, I think everybody involved will be happy to stand up and say "ok, that fixes problem X, and the new thing really is better, so let's do it, it's worth it". But if it's something that happens essentially every single release, that is somethign else altogether. Then it's not a "ok, let's bite the bullet and make the kernel better" thing any more, but instead it devolves into "f*ck, the merge window is open again, now I have to fix up all the crap people pushed on me". See? That's a *huge* difference, even if it is "only" a mental one (and clearly it isn't - there's the actual real work of the "I have to fix things up" part too). So to recap: I have absolutely nothing against fixing up bad internal API's and breaking things. But 99% of the time that should be something we can do incrementally (ie introduce the new API, and simply accept the fact that removing the old API will take a few months). And the case when that _really_ doesn't work should be rare enough that it doesn't wear people down. Because if you listen to the tone of people in this discussion, much of it is about people being _tired_ of having to fix things up. It's not exactly been a "wow, the end result sure was nice!" kind of discussion, is it? And this is where "process" really matters. Making sure people don't get I absolutely have no problem with having a "this is the infrastrcture changes that will go into the next release". In fact, I can even *maintain* such a branch. I've not wanted to open up a second branch for "this is for next release", because quite frankly, one of the other problems we have is that people already ...
On Thursday 14 February 2008, Linus Torvalds wrote: One of the problems caused by this 'grind' is being locked out of using 3rd party closed drivers until the vendor decides its stable enough to make the effort to update their binary blobs to match the newer functions. Nvidia vs 2.6.25-rc1 being a case in point, and they (nvidia) are appearing to indicate its not a problem until some distro actually ships a kernel with the changes that broke it. That could be months or even a year plus. So you've lost one 'canary in the coal mine' tester at least until that happens as I don't have a spare box I can setup to run the nv driver, which itself seems to be suffering from bit rot recently and cannot run this screen at its native 1680x1050 resolution, reverting to something that resembles what I used to get from a timex 1000 in 1978 but with a few colors. I just recently had to install the nvidia driver on my milling machines kubuntu 6.06 box cuz an xorg update put it back to 640x400 if the nv driver was used. This is the real world, where politics aside, it just has to work... :-( But I'll still be lurking and building to test anyway even if I don't boot it for more than 10 minutes. :) -- Cheers, Gene "There are four boxes to be used in defense of liberty: soap, ballot, jury, and ammo. Please use in that order." -Ed Howdershelt (Author) I'm not a lawyer. I don't even play one on TV. - Linus Torvalds on the gcc mailing list --
How about "weeks". Both Fedora and openSUSE's next release is going to be based on 2.6.25, and the first round of -rc1 kernels should be showing up in their trees in a few days. So for this instance, I think you will be fine :) thanks, greg k-h --
-- Cheers, Gene "There are four boxes to be used in defense of liberty: soap, ballot, jury, and ammo. Please use in that order." -Ed Howdershelt (Author) There's small choice in rotten apples. -- William Shakespeare, "The Taming of the Shrew" --
a few days == *NOW* kernel-2.6.25-0.40.rc1.git2.fc9.x86_64 is in Fedora Rawhide already. --
Actually following the NVidia forums indicates otherwise: http://www.nvnews.net/vbulletin/showthread.php?t=107144 I expect Zander will be posting a patch rather soonish, for some value of soonish. And if you're running a -rc or -mm kernel, patching the 169.09 drivers should be well within your abilities.... --
Not so for the binaries, existing patches do make it compile but it still upchucks someplace in the binary, or was this time yesterday. -- Cheers, Gene "There are four boxes to be used in defense of liberty: soap, ballot, jury, and ammo. Please use in that order." -Ed Howdershelt (Author) It is not well to be thought of as one who meekly submits to insolence and intimidation. --
Umm.. if you actually *read* the mentioned thread, you'll see that "existing patches" are known to be incomplete, complete with the "upchucks in the binary" (mentioned at entry number 9 of the thread), and that Zander already knows about it (entry #11), and has apparently one remaining issue left to resolve (entries #32 and #36). And entry #36 is what the NVidia engineer doing the work was thinking about 20 hours ago. Interpret it as you will...
very much agreed. I've yet to see a _single_ wide-scale API change that broke stuff left and right where that breakage was technically justified. I have not seen a single one. All those cases were just plain old botched attempts. Either someone can do a large-scale API change like the irq_regs() cleanups with near-zero breakages, or someone cannot. In the latter case, gradual introduction and trickling it through subsystem trees is a _must_. and if it's _hard_ to do a particular large-scale change, then i think the right answer is to _not do it_ in a large-scale way, but do it gradually. I claim that there's just not a single valid case of doing wide-scale changes atomically and departing from the current to-be-stabilized kernel tree materially. _Every_ large-scale API change can be done in a staged way, with each subsystem adopting to the change at their own pace, it just has to be planned well and tested well enough and has to be executed persistently. And the moment we trickle things through subsystem trees, there's no integration pain, as subsystem trees are largely disjunct anyway. i also fear that having an API-changes-only tree will dillute our testing effort of the current to-be-stabilized upstream tree, as it materially disrupts the flow of patches. Most maintainers should concentrate on stabilizing current -git with only one serial queue of fixes and enhancements ontop of that tree. I dont see how having a second queue would help - it clearly splits attention. widescale API changes should be discouraged, and forcing them through the harder, "gradual, per subsystem" route is _exactly_ such a strong force that discourages people from doing them. Ingo --
Hi Linus, On Thu, 14 Feb 2008 10:01:14 -0800 (PST) Linus Torvalds <torvalds@linux-fou= So, will you open such a branch? If so, what would be the mechanics of having patches applied to it? I assume people would have to suggest such changes explicitly and have them reviewed (hopefully more thoroughly than usual) in that light. I guess one place these "infrastructure" changes may be noticed would be when subsystem maintainers stray outside their subsystem in what they submit to the linux-next tree (or break it). Then I assume most people would start working on a merge of this "next" branch and your "master" branch, right? Consequently, each linux-next would also be based on that merge. I suppose I am stating the obvious (or asking the dumb questions), but I always find it easier to have explicit answers to these sorts of things. --=20 Cheers, Stephen Rothwell sfr@canb.auug.org.au
Two things may largely eliminate the need for parallel branches. 1. Do infrastructure changes and whole tree wide refactoring etc. in a compatible manner with a brief but nonzero transition period. 2. Insert a second merge window right after the usual merge window for changes which cannot be well done with a transition period. (I probably missed a number of points why these two things are not always feasible, because I am just a downstream person.) -- Stefan Richter -=====-==--- --=- =-=-- http://arcgraph.de/sr/ --
A third option would be if people add new functions (with no users) in -rc2 or -rc3 timeframes as long as it is part of a fully reviewed patch with users that will use those new features in various kernel development trees. Since there wouldn't be any users in Linus's tree, there's no risk in making those functions available in mainline ahead of time. - Ted --
I don't like suggestions based on unrealistic assumptions like
"a fully reviewed patch".
E.g. userspace ABI's are much more stable and everyone is aware that
they must be gotten right with the first try since they are then cast in
cu
Adrian
--
"Is there not promise of rain?" Ling Tan asked suddenly out
of the darkness. There had been need of rain for many days.
"Only a promise," Lao Er said.
Pearl S. Buck - Dragon Seed
--
I'm talking about kernel interfaces, not userspace API's. And we can change them if they are wrong, since they *are* kernel interfaces; but if they correct, they ease the cross-tree merge pain. - Ted --
I agree, and that's what I do. The problem is, the API change is still in my tree. So, if for example, the IB tree goes and adds some new functionality before my API changes have landed, they need to use the "old" API in order for them to be able to test and build things on their own. Then, when the -next tree merges everything together, the IB tree breaks the build, not my driver tree. It's those "who goes first" type things that end up being the cause of a lot of Andrew's headaches I think :) thanks, greg k-h --
On Mon, 11 Feb 2008 20:43:14 -0800 this is why you need specific trees for just the API change, and these need to EXPLICITLY go first before EVERYTHING ELSE. Yes this needs a bit of coordination, but it's the only way. --
From: Arjan van de Ven <arjan@infradead.org> API trees don't work, just like other changes they will have interdependencies on things like fixups, cleanups, etc. This is why, with the networking, we've just tossed all of the network driver stuff in there too. I can rebase freely, remove changesets, rework them, etc. and this causes a very low amount of pain for Jeff Garzik and John Linville. This may or not be a legitimate thing to do with other heavily dependant subsystems. --
I was hoping that eventually certain pieces will stop moving and will become a base that we can ask people to rebase on top of. Perhaps even asking that stuff destined for 2.6.x+1 go into a -next branch and the more devel quality stuff go into a -mm branch that is based off of their -next branch. Gradually stuff would move from -mm to -next, getting constantly integrated against other peoples' stuff. More disruptive changes could go in and gradually as people rebase their trees the conflicts would hopefully get less painful. But, let's see how things go. Harvey --
s/very low/not low/ Rebasing is always a pain, and John and I both agreed the other day that you do it too often. I've complained about this before, too... but figured this was just another thing I was getting ignored on, and so life moved on. But don't try to sell rebasing as "low pain". Rebasing makes the history all nice and pretty, but by totalling flattening the history, trashing all the commit ids (and rewriting associated metadata), you create obvious downstream problems. Rebasing is low impact only if you don't have git downstream people. Otherwise, you're just treating it as a useful quilt clone, really. Jeff --
I used to be in this camp too: seeing rebasing as the ultimate evil. However, what I found was that as I tried to preserve my merge tree's history, I was picking up merge points with non trivial fixups. You can argue that this is how git is supposed to operate, but the problem is that these merge points become immovable objects in the tree history ... I can't rebase the tree at all any more and I can't simply remove a commit (which gets messy for commits duplicated by other trees) They also effectively separate the fix up from the patch causing it. Plus, it became difficult for people to go to gitweb to see the tree history, because everything before the merge point got buried. I finally concluded that I needed to rebase my trees to resolve conflicts rather than merging them. The advantages are I know the exact patch with the problem and I can keep the history annotated with what had to be done to get it to apply. Plus with git (and this is a huge advantage over quilt) knowing the base of the patch, I can still use the merge algorithms to help me do the fixup. I'm not saying rebase constantly ... I'm still mindful of the disruption it causes, so I only do it if I pick up an actual conflict. However, I do think it does becomes necessary at times. The problem rebasing causes is that git is pretty horrible at rebasing a tree which is based on your now rebased tree (try saying that after a few martinis). Git really needs to acquire better rebasing tools for this one case, but if that were solved, I think a lot of the objections to rebasing could be answered. James --
I do think that some people rebase too often (and that David is in that
number), often with no real discernible reason. I think rebasing is great
when you are doing active development (ie you're really acting in "quilt
mode") and I actually think git could and should integrate more of the
queues modes, but I don't think it should be a default action for an
up-stream developer.
I also don't think rebasing helps the particular problem under discussion
(ie conflicts due to having to sort out dependencies between different
trees), and in some ways hurts it.
One thing that I personally react to is that
- I think sysfs and the device model layer has had too much churn, and
I'm unhappy that people seem to expect that to continue.
[ NOTE!! I'm picking on the device model/sysfs stuff here not because
it's the only one, but because it's the obvious and good example. I
do think we have other cases of the same thing. ]
Really. I do agree that we need to fix up bad designs, but I disagree
violently with the notion that this should be seen as some ongoing
thing. The API churn should absolutely *not* be seen as a constant
pain, and if it is (and it clearly is) then I think the people involved
should start off not by asking "how can we synchronize", but looking a
bit deeper and saying "what are we doing wrong?"
It may well be that part of the problem is that the people causing the
churn don't realize the downsides of the pain they are causing, because
THEY aren't the generally ones that see it!
For example, it's easy for Greg to change his driver core, and he can
obviously synchronize with himself in the other trees (because his left
hand is hopefully somewhat aware of what his right hand is doing), so I
suspect Greg simply doesn't see the pain that much. So Greg thinks that
the solution is to just have me merge his changes early, and the pain
is all gone as far as he is concerned.
- That ...The problem is how do we use a? Usually we need to track your -rc tree as our fixes go in ... some of which affect our development trees. If we stick with (a) as the base, we don't get to pull in the fixes in your tree. If we use your tree we have to pull in (a) creating n Yes, this is effectively what I did with the post merge SCSI tree. However, if you do this rebasing becomes a fact of life because you need to rebase out all the dependencies you have before you merge (in fact, it's a good way of checking whether your dependencies have been merged yet or not, seeing what survives a rebase). James --
IMHO, this base tree should typically be based off of linus' tree and kept rebased on top of it. This way you get the mainline fixes through the integration base tree. Benny --
Hell no! No rebasing! If people rebase, then it's useless as a base. That base tree needs to be something people can *depend* on. It contains the API changes, and not anything else. Otherwise I will never ever pull the resulting mess, and you all end up with tons of extra work. Just say *no* to rebasing. Rebasing is fine for maintaining *your* own patch-set, ie it is an alternative to using quilt. But it is absolutely not acceptable for *anythign* else. In particular, people who rebase other peoples trees should just be shot (*). It's simply not acceptable behaviour. It screws up the sign-off procedure, it screws up the people whose code was merged, and it's just WRONG. Linus (*) The exception being if there is something seriously wrong with the tree. I think I've had trees which I just refused to pull, and while most of the time I just say "I refuse to pull", early on in git development I actually ended up fixing some of those trees up because my refusal was due to people mis-using git in the first place. So I have actually effectively rebased a maintainer tree at least once. But I still think it is seriously screwed up. --
Linus, what you're saying is pretty extreme. Our experience with pnfs development taught us that rebase was actually the only method that worked on the long run because it allows us to maintain a *clean* series of patchsets over an extended period of time (order of two years). In the past, when we merged your tree into ours rather than using git-remote update and rebase we ended up with a cluttered mess that buried all the resolved merge conflicts in merge commits and maintaining it became an increasingly harder nightmare. Nowadays, I rebase the linux-pnfs git tree almost on a daily basis, keeping tags on historical refs (to be cleaned up when a branch stabilizes) and the other contributors simply use git-remote update to refresh their tracking branches and rebase their stuff onto the new heads. It took the team a couple weeks to get used to this methodology but it really works great for us now. It seems to me that merging works well one (your :) way when patches are merged upstream and become immutable at this point, while on the other way, keeping a patchset fresh and comprehensible is feasible only with rebase when there are occasional conflicts with stuff coming down from upstream. The domino- effect caused by rebase is manageable and not worse than merge if people just keep track of the original bases of their development branch before updating their tracking branches. Maybe the missing link is a tool to help do rebasing more easily. I wrote my own tool for that: git-rebase-tree (available on git://git.bhalevy.com/git-tools.git) and using it makes the mechanics of rebasing our patchsets almost trivial. I really encourage folks to try it out and give me feedback about it. So what's the solution for maintaining a series of patchsets that depend --
I don't understand what you mean. This is true whether you pulled (a) or not. If you have any changes what-so-ever in your tree, if you pull in fixes from my tree, you'll get a merge. But if you mean that you cannot rebase (a), then yes. That was what I said. Rebases *do*not*work* (and fundamentally cannot work) in a distributed environment. But why would you merge with my tree in the first place? My tree won't normally have any conflicts or anything like that anyway. With a "Linux-next" tree, you'll see the conflicts if they occur (since *that* tree would merge!), and in that case you would say "now I need to merge Linus' tree just to resolve the conflicts!" But before that, merging my tree (or rebasing on top of it) is simply I don't see the logic. You shouldn't need to rebase at all. I don't see why you claim that this makes rebasing more of a fact. It doesn't. It has no impact at all, except making rebasing _less_ possible! Linus --
Yes, that's what I meant ... no rebasing of (a) once it becomes the Hm ... I think net is a counter example to this. Rebases certainly work for them. The issue, I thought, was around the policy of rebasing and how often. I see the question as being one of who creates the history. When something goes into your tree, that's an obvious history point and we can't rewrite it. However, when something goes into my trees, I don't think it's as obviously a history point that has to be preserved, so I can rebase obviously, rebasing causes disruption, so I don't do it This normally happens when we have bug fixes to a driver that is simultaneously being worked on for the merge window ... and it actually happens quite a lot. Sometimes the conflicts actually don't come via my tree (usually a -mm docbook update, include file removal or something else that looked trivial to the submitter). The worst examples of this are the "run lindent on the driver" type of Right ... that's exactly why I created the -mc tree ... it warns me that there's a looming problem I need to fix (and not just in your tree). For that reason, a -next tree that does exactly the same thing will be Yes ... I don't do that ... Like I said, I only rebase for an actual Well, it came at me because Jens was rebasing the block tree as he worked through issues in the two branches I was based on. If the block tree hadn't rebased, I wouldn't have rebased that often. However, we did come across other things that had to move from my tree to Jens' as we developed this, mainly bug fixes moving closer to the source to preserve bisectability, and in that case rebases were really necessary. James --
They consider themselves to be "one tree" and are thus largely a totally different issue than the one discussed here. Also, I actually flamed David a lot last round over his rebasing of netfilter etc. It was no acceptably done. That network tree was all screwed up, with the git committer information not matching the signed-off path etc. If you do cross-tree rebasing, you need to consider it 100% equivalent to Yes, and I am in no way saying that the core driver model has been the only problem spot. And also, I do not like "hard rules". Every rule always has an exception, and sometimes a rebase-based strategy can be the right thing even across trees. But you're all ignoring my fundamental objection: you're talking as if cross-tree fundamental API changes should be the "norm", and that we should try to solve the workflow issues that stem from that. And I'm saying that I think we should try to FIX the issue, and make sure that it simply *isn't* the norm. In other words, I'm perfectly happy to be an a*hole and tell people that I simply won't merge things that cause undue API churn at all, and that were not thought out sufficiently. We've had too many issues like that (SG chaining, iommu, driver core, not to mention the upheavals in x86) lately, but realistically, which subsystem remains a problem for the future? And maybe the correct thing to do is to just say "enough!". I'm perfectly happy being hardnosed and saying "nope, that's crap, it doesn't matter if the code is slightly better if you cause those kinds of issues". The thing is, sometimes the answer really *is* "Don't do that then!". If our model becomes bogged up by cross-subsystem serialization issues, that means that we (a) spend too much time handling the fall-out from that and (b) too little time just refining the details. And quite frankly, while "good core architecture" matters a lot, in the end, "getting all the boring small things right" matters even more! Big ...
.. btw: I'd need to know this in advance. I usually don't see the problem until it's too late. And this is very much an area where "Linux-next" can help: if some subsystem causes problems in Linux-next for other maintainers, I really think it shouldn't just be a matter of "drop the git tree that didn't merge cleanly", but it should literally be a question of "maybe we should drop the _earlier_ git tree that caused the later one not to merge cleanly". In other words, maybe things like core block layer changes or device model changes should be *last* in the merge-list (or if first, also be first to be dropped if they cause merge errors downstream!). That way, infrastructure changes that screw up others can only happen if the maintainer actively works with the others to make sure it works even before it would ever merge into Linux-next successfully. That may sound odd, but it actually matches what I personally believe in: we have more driver code and other "outlying" things than we have core things, and most of our problems come from that - so we should prioritize *those* things, not the "fundmantal core changes". So how about making that the default situation: drivers and other outliers merge first. If fundamental API changes happen, they merge last, and if their maintainers can't make it in time in the merge window, they just get dropped. That sure as hell would put the pain on API changes solidly where it belongs. Linus --
Sure, I have no objection to that at all. thanks, greg k-h --
We don't always - and I'd say in terms of ARM it would be extremely rare. The sysfs API changes at the start of the last merge window is one example of this. I had everything nicely prepared in the ARM tree for merging shortly after the window opened, and then wham bam thankyou mam, the merge window opened and a lot ended up breaking. We know that the -mm tree is pretty much useless in terms of code coverage for ARM, and it's getting increasingly unlikely that anything short of a build of all ARM defconfigs will pick up on merge issues - which is a lot of CPU cycles, and I'm not going to insist its something that should be done. -- Russell King Linux kernel 2.6 ARM Linux - http://www.arm.linux.org.uk/ maintainer of: --
If you need help in that area (build-testing ARM), surely we can find some fast machines that can do builds for you. Any interest? Jeff --
It would probably make some sense - it would help to get hold of the infrastructure that armlinux.simtec.co.uk uses to run their test builds of Linus' tree and apply it (on a separate machine) to -mm (or indeed whatever other) kernels. -- Russell King Linux kernel 2.6 ARM Linux - http://www.arm.linux.org.uk/ maintainer of: --
We could simply decide that API changes affecting more than one subsystem Must Be Serialized(tm). Explicitly. As in "any such change is posted and discussed in advance, order of merges decided upon and we have merge window for one decided set of API changes + fallout *ONLY*". With merge window in question normally taking a few days. --
Welcome to dreamland. The only way I can get serial changes done is to wait months and eventually simply persuade Andrew to ignore the "maintainers" who are basically not responding. Maybe it works for vfs and mm internals but out here in driverspace forget it - and if I simply break drivers people get cross instead of fixing them nowdays. --
On Tue, 12 Feb 2008 21:51:52 +0000 "maintainers"? I think the best way to get the serial drivers maintained would be to cat them all onto the end of synclink.c and hope that Paul thinks he did it. --
Well I've already broken the buffering so he'd fix it ;) We have a pile of old ISA drivers that are going to break soon with the locking changes and a pile of USB drivers that when I looked at the locking were so terminally broken I couldn't be bothered to fix them. So it'll be fun, lots of && BROKENs to add --
Hrm... How badly is pl2303 broken? I actually use that sucker, so if it needs help - count me in. --
2303 is pretty good (in fact by usb serial standards outstanding). It has all the internal locking needed for now and right down to killing lock_kernel entirely outside of open/close (which is going to hit everything). Only fixme I have tagged for it is - if you set an unsupported baud rate it reports it set rather than the one you got Which is a trivial mend for someone who has suitable docs (its marked FIXME: at the right spot) --
I'll see what I can do about fixing this. thanks, greg k-h --
Let me know which USB ones are broken, I'll work to fix them. thanks, greg k-h --
On Tue, 12 Feb 2008 14:55:31 -0800 That I noticed doing an audit for unlocking the mctrl functions: ir-usb: global variables without locking used in per port operations iuu_phoenix: no locking on internal data structures mos7840: ditto option: ditto kobil_sct: ditto These drivers do interesting things (where interesting is probably not too evil on a PC - except ir-usb) involving playing with data structures without locks. It seems there was some kind of evolution along the way as some drivers do have a carefully used port private data structure lock (or two) but many do not. --
Ok, I'll look into them this week. I'm currently working on revamping the option driver as it had the mistake of being based on the keyspan drivers, which weren't the nicest to start with... thanks, greg k-h --
... because we didn't realise that the pre-requisits for those patches had already gone in. If that had been the case, I'd have applied the ARM changes for the new tty ioctls immediately. It was a communication problem, not a process problem. -- Russell King Linux kernel 2.6 ARM Linux - http://www.arm.linux.org.uk/ maintainer of: --
From: Linus Torvalds <torvalds@linux-foundation.org> If a person does a driver API change and does all the work to sweep the entire tree updating all the drivers, doesn't it penalize that person a bit much to stick a new driver in front of that work? People write code on top of infrastructure, both new and old, not the other way around. At least to me, that seems how the merging ought to work too. --
If that API change doesn't conflict with the work that hundreds of other people do, it's obviously not a problem whichever way it goes. And if the API change *does* cause conflicts, then yes, the onus of fixing those conflicts (again) goes to the person who changed the API. Everybody You think that infrastructure is more important than outlying code. But you do that only because you write the infrastructure, not because you have any logical reason to think so. The fact is, that "outlying code" is where we have all the bulk of the code, and it's also where we have all those developers who aren't on the "inside track". So we should help the outliers, not the core code. And very fundamentally, API changes are to be discouraged. If we make them harder to do and make people think twice (and occasionally say "not worth it"), that sounds like a damn good thing to me. Linus --
From: Linus Torvalds <torvalds@linux-foundation.org> Good point. For the record I write drivers that use infrastructure too and have to deal with the API changes just as equally :-) --
OK, I'll address it then (or try to). Just don't eat me please ... I'm
I can sort of agree, but I don't think you can say enough == no more API
changes. There's the request_irq argument removal patch floating around
somewhere for instance ...but as someone said "Every rule always has an
exception".
I think what's really needed is a balance where it's painful (on the
originator) to make an API change. Part of what -mc or -next trees can
I think this is exaggerated. By the end of the -rc cycle for 2.6.24 I
had six merge fixup patches because of trivial cross subsystem API
changes and two dropped trees because of intractable problems (and they
were notified and later fixed themselves up). Given that I was
including about 50 git trees and 4 quilt ones, that's not too bad,
I agree with this too. When I introduce an API, I usually try to make
sure it will last the test of time ... the trouble is, I'm not always
omniscient.
So the point, I think we've reached is that:
1. we want to try to enforce good design for APIs to try to ensure
they're as future proof as foreseeable.
2. We have to recognise that 1. is impossible in practice and
provide a mechanism for correcting APIs
So, I think our current system is good in that it enforces pain on
people who change APIs. However, perhaps we are missing the bit where
we reflect longer before introducing APIs in the first place. (hey,
better reviewing ... I've heard that one before ...)
James
--
From: Linus Torvalds <torvalds@linux-foundation.org> And if conflicts happen every day, what should someone do? This isn't fantasy, that's exactly how things got with the networking tree near the end of 2.6.24. --
Hmm ... as far as I can see, Jeff and John (i.e. net tree downstreams) are pretty loudly unhappy with Dave rebasing too often, right? -- Jiri Kosina --
That's true ... but irrelevant to the argument of whether rebasing does or doesn't work. I don't think anyone's arguing that rebasing doesn't cause real problems to downstream users ... when I was based on block for the scsi post merge tree, I was just such a user ... James --
From: James Bottomley <James.Bottomley@HansenPartnership.com> And I realize that regrettably I end up rebasing a lot. Let's say that today I merge a TCP bug fix into Linus's 2.6.24-rcX tree. When I have the net-2.6.25 tree going I know that this is going to create merge conflicts with the 80 or so odd TCP patches I have in there. Nobody can pull net-2.6.25 into Linus upstream without having to sift through the merging of that stuff. It never merges cleanly using the automated mechanisms, because since the changes are split up nicely there are long changeset dependency chains. So I rebase, and do the merging work by hand. Next, let's say Jeff merges some net driver bug fixes into upstream, resulting in potential conflicts with the several hundred or so driver changes that are in the net-2.6.25 tree too. In fact near the end of 2.6.24 development, there was a new merge conflict created on a daily basis with the net-2.6.25 tree. You simply cannot avoid this when you're managing 1500+ changes. I even had to rebase the net-2.6.25 tree once or twice in Australia as the merge window was openning up because I could push something cleanly to Linus. There were conflicts created by stuff that got in before the net-2.6.25 tree, mostly in files like the feature removal schedule, Kconfig files, and whatnot. At times I even felt the urge to avoid merging a bug fix upstream because of all the merge conflicts it would create, but I of course can't and won't do that. 8) It actually turns out that things simplify once a tree gets into the -stable folks hands. I pick out bug fixes as they go upstream, and toss it to them once Linus sucks it in and I have an upstream changeset ID to give them. I don't have to worry about -stable changesets causing development merge grief later on. And I've also yet to be shown how to completely remove a changeset from a GIT tree without rebasing :-) --
That's a matter of opinion. I'm working on cleaning up the libertas driver as and when I have time, and the constant rebasing of the git trees effectively means that I can't just use git as it was intended; I feel that I'm being forced back into the 1990s by having to deal with sequences of patches instead. It's such a pain that I'm seriously tempted to push my changes directly to Linus instead of through the subsystem maintainers. -- dwmw2 --
I may be a bit defensive here, but I hope that all of the recent kobject/kset/driver core changes have been done with the thought of "what are we doing wrong". For the kset rework, we went back and looked at how people were trying to use this code, realized that it was way too complex, and reworked it all, making both the implementation simpler, the kernel usage model simpler, and documented the whole thing so that everyone knows exactly what is now going on. To quote the original developer of that code when hearing of the rewrite, "thank you for doing it, I have no idea what I was thinking when I wrote that code originally." The end result caused more code to be removed from the kernel than was added, always a nice thing. The rework went through many iteratations, reviews, rebases, and touched many portions of the kernel. In the end, there was only 1 merge issue, in a new IB driver, and Roland and I handled that after Andrew pointed out. That kind of dependency was what I was trying to warn the -next No, I understand the issues here, and am working hard to resolve them. Now that the kobject underlying layer looks very good, I'm moving a bit higher up, into the driver core to be able to handle a long-standing requirement by a lot of hardware vendor and driver authors. And yes, that is going to cause a few problems in a few places in the bus-specific logic (the PCI core looks to be the biggest issue, as it does some nasty things with some internal device lists), but for the rest of the kernel, and individual driver authors, it will not be an issue at all. But I understand your main point here, a lot of time I might come across as wanting to constantly change this chunk of code, but I'm only doing it because it's necessary. I'd much rather be off just writing new drivers, and not having to touch this stuff at all. If people see changes in which they think I and Kay are unnecessarily causing churn, please call us out on it, I have no problem with that. A ...
.. but are we expecting it to be finished? That's the point. This whole "Linux-next" discussion so far has almost been predicated on the whole assumption that this is an on-going concern. And it really should NOT be. If it's an on-going concern, we need to tackle *that* issue, not the issue that cross-subsystem merges are hard. They simply seem to happen too much. In other words, I'm not AT ALL interested in the merges we've already done. That's over and done with, and we'll never ever do those merges again. Who cares? I don't. I'm purely and _only_ interested in the merges of the future. You don't need to be defensive about the things that led up to this discussion, I'm more hoping that we can aim at fixing the problem at the source, rather than trying to work around it. We simply shouldn't have all that many conflicts. We've had *way* too many of them lately, and I think it's because people have felt it wasn't too painful. Put another way: back when we worked with just patches, we avoided renames like hell, and we also tried to simply even re-architect the whole tree so that you didn't have so many patch conflicts. One main reason as far as I was concerned for things like per-directory Kconfig files and the whole initcall() stuff was the fact that the old single Kconfig file and the old crazy init/main.c file were total *nightmares* when it came to conflict resolution. So we split things up more, and we didn't do renames (or were very careful about it). We avoided the things that caused pain. I think we need to remember that: yes, we'll always have to have ways to fix the pain that does happen, but even more importantly, we should strive for models where it doesn't happen in the first place! And simply avoiding cross-subsystem API changes unless there is a major *MAJOR* reason for them is the obvious thing to do. Simply face the fact that even in open source there are major reasons to stay with an old interface even if it's not ...
Not it isn't. To quote you a number of years ago: "Linux is evolution, not intelligent design" We need to constantly be able to make these kinds of changes in order to Oh, it's been painful at times, but they are, overall, very rare. If you look at the rate of change we are currently running at, it's amazing that we do not get _more_ of these kinds of problems. Remember, we are currently clocking along at the steady rate of: 4000 lines added every day 1900 lines removed every day 1300 lines modified every day And the rate of change in each major portion of the kernel (drivers, arch, core, network, etc) is exactly proportional to the amount of the kernel that that portion takes up (I have detailed numbers if people really want to see them.) Because of this rate of change, we are surviving, and evolving into what is exactly needed at this point in time from a kernel. To try to quench this change, and force us to stick with things that are no longer optimal, is going to cause us to make the same mistakes that AIX and We have done very good at that. The problem comes in when one subsystem depends on another one. Like the kobject core, or driver core. Or network core. Changes there trickle out into the other portions of the kernel that depend on them. And we fix them up, and move on. And so far, we're doing this very I strongly disagree here. We lived with that kset/ktype crap for years, and I finally broke down and cleaned it up, simplifying things, removing code, making the kernel smaller, leaner, and easier for others to change and use in the future. With your statement, such a change should have never taken place as it what we had at the time was "not optimal", but They usually don't, by virtue of our current development model and how we have the kernel structured. But they do happen about once or twice a kernel release, just by virtue of the way things need to happen. And since the last kernel had 10353 different commits, only 2 conflicts ...
???? lines reviewed every day. AFAICS, we are in situation when review bandwidth is where the bottleneck is. Not the merge one... --
I can run the numbers, but almost every one of those changes has at least 2 signed-off-by: on them, so they should all be being reviewed Are there still large numbers of posted patches, not reviewed or picked up by anyone laying around somewhere? I thought Andrew-the-patch-vacuum had been doing a great job of keeping that from happening lately. thanks, greg k-h --
Er... Surely you can think of examples of patches that went through -mm without serious review? Not to mention that 2 S-o-B normally means only one thing: "went to Linus through some tree". Which does not guarantee any real review, as you damn well know. --
Yes, I agree, there are lots of examples of this, but the overall majority are reviewed by 2 people at least (or sure as hell should be, maybe we need to bring into existance the "reviewed-by" marking to ensure this.) Do you know of a way to help this get better? thanks, greg k-h --
Well, I don't really "review" any patches that come through Andrew. What I do is: - global search-and-replace Andrew's "acked-by:" with one that is both him and me (that way I make sure that I _only_ sign off on patches that he has signed off on!) - look through all the commit *messages* (but not patches). This sometimes involves also editing up grammar etc - some of those messages just make me wince - but it also tends to include things like adding commit one-liner information if only a git commit ID is mentioned etc. - and only for areas that I feel competent in, I look at the patches too. So, to take an example, when Andrew passes on uml patches that only touch arch/um and include/asm-um, my sign-off does not mean *any* kind of review at all. It's purely a sign that it's passed the sign-off requirements properly. When it comes to VM issues or other things, things are different, and I actually review the patch (and occasionally send it back with "nope, I'm not applying this"). But for stuff that comes through Andrew, that's probably less than a quarter of the patches. And I don't mark the ones I've reviewed specially in any way. And I suspect I'm not at all alone in this. People simply have maintainers they trust (and _need_ to trust in order to not become a bottleneck). Linus --
I wrote in one of the many bug reports I sent for compile errors
introduced in Linus' tree during this merge window:
An architecture specific patch that breaks the one architecture it
touches at the first file being compiled is even for kernel standards
cu
Adrian
--
"Is there not promise of rain?" Ling Tan asked suddenly out
of the darkness. There had been need of rain for many days.
"Only a promise," Lao Er said.
Pearl S. Buck - Dragon Seed
--
Umm. Have you read a lot of books on evolution? It doesn't sound like you have. The fact is, evolution often does odd (and "suboptimal") things exactly because it does incremental changes that DO NOT BREAK at any point. The examples are legion. The mammalian eye has the retina "backwards", with the blind spot appearing because the fundmanetal infrastructure (the optical nerves) actually being in *front* of the light sensor and needing a hole in the retina to get the information (and blood flow) to go to the brain! In other words, exactly *because* evolution requires "bisectability" (any non-viable point in between is a dead end by definition) and does things incrementally, it doesn't do big flips. It fixes the problems on an incremental scale both when it comes to the details and when it comes to both "details" (actual protein-coding genes that code directly for some expression) and "infrastructure" (homeobox and non-coding genes). So quite frankly, you're the "intelligent designer" here. You're the one No, overall, they have *not* been rare lately. We've had them all over. I don't think that's a valid argument. Sure, we have lots of changes, but 99.9% of them have no cross-subsystem You didn't listen at all. I said that the threshold should be high, not that it should be impossible. I also said that we should strive for making it unnecessary to have the painful total synchronization points. The fact is, we *have* been able to do things like this gradually and well, without introducing breakage. Take the VM changes, for example: those were pretty damn fundamental, where we've changed the calling convention totally for fault handling. But that thing was done without at any point really seriously breaking code. It involved adding the new interface, and letting the old one live in parallel. The last remnant of the old "nopage()" interface still exists, but I think right now it's only used by DRM. Did it require the drivers ...
No, I might have mistyped previously, but I really do not believe in I agree with this. But I was reacting to your "there are major reasons to stay with an old interface..." portion above, while I feel we should No, not at all. I have tried, and successfully done this many times in the past. The kobject change was one example: add a new function, migrate all users of a direct pointer over to that function, after that work is all done and in, change the structure and do the needed work afterward. All is bisectable completly, with no big "flag day" needed. Same goes for a lot of USB changes. Hell, we've been slowly moving toward changing the USB urb callback for _years_ yet, not quite getting there, but the groundwork is done, making it much simpler and easy for it to happen. Eventually things will flip over and the eventual patch for all callback functions will be trivial and easily reviewed and merged. In short, I agree that flag days are bad and should be avoided at all costs, and that slow incremental, bisectable changes are how we evolve properly. And that we need to always be able to do this kind of work, and never be stuck with the "can't change an old, internal api" model. To bring it around to the original topic, -next should help us in finding these issues out, exactly like -mm is. If we want to make a new rule of "only merging api or cross-subsystem changes after everything else", that's wonderful with me, I have no objection to it at all. thanks, greg k-h --
That takes care of bisectability, but doesn't take care of the problem of new stuff being merged which used the old API and is now broken. Maybe the last step (removal of old API) needs to be taken a lot later than it currently is (maybe a second merge window between -rc1 and -rc2 for removal of interfaces that became unused during the first merge window before -rc1?) -- Intel are signing my paycheques ... these opinions are still mine "Bill, look, we understand that you're interested in selling us this operating system, but compare it to ours. We can't possibly take such a retrograde step." --
I think this statement has been used unfortunately as a hard and fast rule (which we all know how much Linus loves :-) to mean, in its most extreme form, that we should *never* try to do some careful reflection about careful API design, and that the extremes of "no interface without an in-tree user" applies to (a) parameters in a function call (heck, we can always sweep through all the in-tree users to add that extra parameter later, and thats a *good* thing because it breaks those evil out-of-tree drivers) and (b) to not even thinking if some particular interface (that is not needed now but which reasonably will be needed later) is even *possible* without doing a sweep of all of the in-tree users of the interface. Related to this syndrome is the assumption that measuring the rate of changes in lines of code changed per second implies that any development process which causes the number of lines of code changed second, including frequent sweeps through the tree changing all interfaces, is a *good* thing. Yes, this is an extreme position, and I'm not accusing anyone of holding the above in its entirety --- but I've seen aspects of all of these from one developer or another. We come to it from the attacking another strawman, which assumes that *all* interfaces which don't have an in-tree are evil, and that keeping old __deprecated interfaces for a long time is an evil which causes intolerable pain, and that it's never worthwhile to try to anticipate future expandibility into an interface because you will inevitably get it wrong. Clearly, we are right to mock Solaris for making claims that they will never, ever, *ever* change an interface, not even one that goes back sixteen years to Solaris 2.3. But it doesn't follow the opposite point of view, that constant mutability of kernel interfaces to make sure that things are always perfect and pure and clean is the right Also, evolution also means that things like vestigal organs (like our appendix) are tolerated. So are ...
Incorrect - because this all happened far too quickly. This is one of the reasons that I ended up having to redo various parts of the ARM tree because stuff broke - set_kset_name() completely vanished introducing compile errors, and iirc some merge issues as well. I had patches introducing new system objects which use that, and modifications extremely close to other uses in the PXA code. The end result (through rebuilding the affected parts of my git tree, and asking people for replacement patches) was something that is bisectable - but had I tried to merge stuff as is, it would've been an utter mess, and _was_ unbuildable. -- Russell King Linux kernel 2.6 ARM Linux - http://www.arm.linux.org.uk/ maintainer of: --
On Fri, 15 Feb 2008 23:23:08 +0000 I wonder why I didn't see any of this - I build arm allmodconfig at least once a week, usually more frequently. So either the offending patches weren't in my pile or arm allmodconfig is worse than I thought :( It really is in arch maintainers' best interest to keep their allmodconfig in good shape, for this reason. arm's _isn't_ in good shape: the compile fails for several long-standing reasons (eg: no hope of building DRM) and I don't think the coverage is very broad either. --
I think that Russell has said that allmodconfig isn't very realistic for ARM, with its 70+ config files. Nevertheless, having a usable allmodconfig would be very helpful. --- ~Randy --
On Fri, 15 Feb 2008 15:47:24 -0800 You'd need to pick one board support and enable everything else you possibly can. --
Basically, you don't build any of the PXA platforms, which is what was Which is quite impossible. I've said all along that all*config is bad news for ARM and folk haven't listened. allmodconfig can (and does) work on some platform configurations such as the one Andrew builds - which will be based on the Versatile platform. However, that doesn't get _any_ of the PXA SoC drivers scattered throughout the tree, the PXA SoC support in arch/arm/mach-pxa, none of the PXA platform support files. If you built an allmodconfig PXA configuration, you'd get those, but you wouldn't get the OMAP SoC drivers, nor the ARM Primecell drivers found on ARMs Integrator, Versatile and Realview platforms and the Cirrus EP93xx SoCs. Nor the Atmel AT91 drivers... and so the list goes on. Each family of platforms are, unfortunately, quite distinct from each other. -- Russell King Linux kernel 2.6 ARM Linux - http://www.arm.linux.org.uk/ maintainer of: --
Does that mean that an artificial allarmconfig can't be made to build? We clearly don't care whether it can boot or work, but we would just as clearly like to be able to build more ARM stuff without N (N > 10) configs. -- ~Randy --
This is not entirely true if the pressure for changes are removed. For instance in mammals the bones in the ear are what used to be gills in fish. When fish became amphibians the gills weren't needed as much and evolution In nature there is a lot of duplication: several copies of genes can exist and different copies may have a distinct evolution. There is also a lot of 'junk' DNA that doesn't code for anything (although it may have regulating functions). In there some copies of genes may remain that are inactivated, as well as parts of virusses, slowly obtaining random mutations because there is no pressure on the evolution of them. Some may eventually become active again and have different functions. The duplication also often ensures there is fallback when random mutations are acquired and a protein is knocked out. Besides the two chromosomes several proteins also can have overlapping functions. The result is more like a balance. Evolution in nature and changes in code are different because in code junk and bugs are constantly removed. In biology junk is allowed and may provide a pool for future development. Linux development is intended and not survival. --
This is true of very complex animals, but much less so when looking at things like bacteria (and arguably, any current sw project is closer to bacteria in complexity than anything mammalian). In bacteria (and viruses), duplication of DNA/RNA is a big cost of living in general, and as a result there is *much* less junk DNA. So in an evolutionary sense, it's much closer to what the kernel should have (with occasional duplication of code and interfaces to allow new functionality, but rather aggressive pruning of the excess baggage). In other words, all of these choices are a matter of "balance". In some areas, excess code is not a sufficient downside, and we keep even broken source code around with no actual function, "just because" (or rather, because the cost of carrying it around is so small that nobody cares). That's true in the kernel as in biology: check out not just deprecated code, but the drivers and other odds-and-ends that are explicitly marked as non-coding DNA (we just happen to call them BROKEN in our Kconfig). Linus --
I like the comparison, and while I wrote my comment I have to admit I was also thinking of bacteria and virusses as an exception: There the speed of replication can be an important factor for survival. Less DNA means faster replication and therefore the pressure is on removal of junk DNA. It can be disputed however that removal of 'junk sourcecode' is a survival factor Maybe we can elaborate a bit on this comparison, just for fun: I think not the linux kernel alone, but rather the entire Linux OS could be compared with a cell. The kernel source encodes vital software parts including the interactions with hardware - the environment for the computer. Gcc can be compared with the (transcription and) translation machinery. DNA can be seen as a language that encodes proteins, with biological functions: Some are vital, including ones that allow interactions with the environment: The cellular environment is beyond the membrane. Interactions occur through membrane receptors, channels, etc. Interaction between proteins can be compared with functions selectively calling other functions. Activation of certain proteins can cause a cascade of protein interactions, comparable with function calls in a loop: the activated protein activates particular protein(s) several times. Some proteins influence intracellular messengers: cellular global variables. Transmembrane receptors responding to extracellular signals transmit this through conformational changes across the membrane to the intracellular region: These structural changes may allow interactions with new proteins. Maybe comparable with a combination of hardware interrupts, signals and the userspace? The response to extracellular signals often depends on several sequential interactions between proteins. This provides a protective layer that can be compared with the kernelspace layer. This is where the comparison probably becomes too biased to continue. --
I would be interested to see any evidence (rather than intuition) to support that given that both appear to be the same kind of structure and that structure is nowdays fairly well understood. Alan --
What do you mean with structure, the evolution? that both are a language? --
On Sat, 16 Feb 2008 00:05:59 +0100 No that they show the same mathematical structure and behaviour - both are scale free networks. Alan --
From: Greg KH <greg@kroah.com>
BTW, there are some things we can do to with the kernel structure
to help a lot of really idiotic cases. Here are some odd-the-cuff
suggestions:
1) Make feature-removal-schedule a directory with files in it.
Everyone touches that file, creating merge issues.
2) Let's move away from some/dir/{Kconfig,Makefile} schemes and
instead have each "thing" have it's own Kconfig.foo or
Makefile.foo that gets automatically sucked into the main
directory Makefile or Kconfig using file globs or similar.
Even better, encode the building of things into the *.[ch]
files themselves, and have the Kconfig/Makefile machinery
automatically extract this information when you build.
Little things like this would go a long way to eliminating merge
hassles.
For example, with #2, when a driver is added it would only
every add files never edit existing files. Merge conflicts
are impossible unless two new drivers try to use the same
file names. :-)
--
On Tue, 12 Feb 2008 16:41:49 -0800 (PST) 3) teach people that you don't always have to add new includes right at the end of the list 4) teach people that you don't have to add Makefile rules right at the end of the list 5) ditto feature-removal 6) ditto lots of other things. My usual way of fixing these things when they pop up is to just move the offending addition to some random position other than end-of-list. I must have done this hundreds of times and as yet I don't think anyone has noticed ;) --
From: Andrew Morton <akpm@linux-foundation.org> That sounds like a real useful usage of your time. I think my proposal saves everyone, including you, this time and the effort necessary to implement it would only need to be expended once instead of this ever present time and learning sink you seem to be advocating. --
If the list is already sorted, I insert it at the right position.
If not, well...
Gr{oetje,eeting}s,
Geert
--
Geert Uytterhoeven -- There's lots of Linux beyond ia32 -- geert@linux-m68k.org
In personal conversations with technical people, I call myself a hacker. But
when I'm talking to journalists I just say "programmer" or something like that.
-- Linus Torvalds
--
So we could do: config foo tristate "do you want foo?" depends on USB && BAR module obj-$(CONFIG_FOO) += foo.o foo-y := file1.o file2.o help foo will allow you to explode your PC The [module] part would go unedited into a kbuild file located in same dir as the Kconfig file and included by kbuild. That would solve the Makefile issue. And we could introduce support for source "drivers/net/Kconfig.*" But then we would have to make the kconfig step mandatory for each build as we would otherwise not know if there were added any Kconfig files. Does this fit what you had in mind? Sam --
From: Sam Ravnborg <sam@ravnborg.org> Yes it does. Now I'll ask if you think embedding this information in one of the C files for a module would be even nicer? Also, we need to make sure we can properly handle top-level container-like items. For example, where would menuconfigs like NETDEV_10000 go if we adopt this kind of scheme? --
I have no good idea for the syntax and I and not sure what is gained by reducing a driver with one file. If it makes sense to group stuff inside a menuconfig it would also make sense to put the same modules in a subdirectory. And then we would have the menuconfig in the Kconfig file that would source the others. So I do not see this as an issue for the 'embedded' syntax described above. Sam --
From: Sam Ravnborg <sam@ravnborg.org> Sounds good to me. --
I like the Sam proposal, but maybe we can simplify the rules on
"module" segment: some informations are often redundant, dependencies
are sometime calculated by config part and sometime by Makefile
(and sometime in the Makefile there are some wrong hacks).
I would really like a good section like:
module
foo : file1.o file2.o
and let the complex rules in the normal Makefile
(which is also good because the complex rules are often not
specific to a single driver).
But I don't like merging all info in a single file:
- not so clean in case of multiple source-file driver
- it would be more complex the "copy and paste"
from other drivers: most developers are not
comfortable with Kconfig and Makefile, so easy to grep
others Kconfig/Makefile could help developers not do
do strange/wrong hacks.
ciao
cate
--
I kind of like to be able to grep over just Kconfig files, to find out what is going on... -- (english) http://www.livejournal.com/~pavelmachek (cesky, pictures) http://atrey.karlin.mff.cuni.cz/~pavel/picture/horses/blog.html --
Hi, I'm more thinking about something like this: module foo [FOO] tristate "do you want foo?" depends on USB && BAR source file1.c source file2.c if BAZ Avoiding direct Makefile fragments would give us far more flexibility in the That's a real problem and it would be a step back of what we have right now, so I'm not exactly comfortable with it. bye, Roman --
And we can finally distinguish between
config bar
bool "do you want bar?"
for boolean options and
module baz
bool "do you want baz?"
for modules that cannot be modular?
And we can make `depends on' do the right thing for dependencies on modules
that are modular (cfr. e.g. commit e11a6c236b3070ed05b079f91a9b3defa48b54d3,
[VIDEO]: XVR500 and XVR2500 require FB=y)?
Gr{oetje,eeting}s,
Geert
--
Geert Uytterhoeven -- There's lots of Linux beyond ia32 -- geert@linux-m68k.org
In personal conversations with technical people, I call myself a hacker. But
when I'm talking to journalists I just say "programmer" or something like that.
-- Linus Torvalds
--
Much better and now I see it I recall you posted something along these lines before. Is this something that you plan to look into implementing? I can do the kbuild bits but I need you to do the kconfig stuff (which is by far the biggest effort too). It would be much appreciated to get this. Sam --
From: Linus Torvalds <torvalds@linux-foundation.org> Perhaps not, but self-conflicts are the bigger issue for the networking. If I (or Jeff or John) push a bug fix to you outside of the merge window, that blows the entire development tree because at 1500+ changesets I can guarentee you there will be a merge conflict somewhere. Near the end of 2.6.24 every bug fix I merged to you created a conflict with my net-2.6.25 tree. Every time I rebased there were on the order of 10 to 20 patches with merge conflicts to resolve. --
Yep. I do that with the powerpc tree sometimes, too: I'll ACK a drivers/net change that is obviously PPC-specific, and that drivers/net change will go in via paulus. Like you said, it's done on a per-patch not git tree basis at the moment. That sort of cooperation has become common: in areas of shared responsibility, request an ack, and then the "most obvious" person (generally the one with most merge dependencies) merges that shared-responsibility change along with some other changes specific to their own tree. Jeff --
> The other is that once somebody says "ok, I *really* need to cause this
> breakage, because there's a major bug or we need it for fundamental reason
> XYZ", then that person should
>
> (a) create a base tree with _just_ that fundamental infrastructure change,
> and make sure that base branch is so obviously good that there is no
> question about merging it.
I don't disagree with this, but I think I should point out that making
something "obviously good" may be pretty hard. It's clearly a common
case that the infrastructure change goes through several rounds of
change -- perhaps prompted by exposure in -mm that shows a subtle
issue. So then if all other maintainers based their trees on this
tree, we're left with two not-so-great alternatives:
1) merge the original, broken infrastructure change into your
(Linus's) tree, leaving a known problem for bisecters to trip
over.
2) rebase the world.
I don't know if there's really a perfect answer here. I hope that
tree-wide infrastructure breakage is uncommon enough that we can just
handle these issues "by hand" as they come up.
- R.
--
How do you suggest that we fix the problems we had with the initial merge with the ARM tree. We were in the situation where, had I asked you to pull my tree, you'd have had a fair number of conflicts to resolve - I tested this by trying to merge your tree into my 'devel' head. The result wasn't pleasant - even I didn't know how to fix up a lot of those conflicts, so the only way I could move forward was to reconstruct the 'devel' head (which is just merges of other branches) omitting the problem branches. This then posed something of a problem - of which I saw three solutions: 1. ask the original authors of changes in those problem branches to come up with patches or fixes in their git trees (in the rare case that they have git trees) to bring the branch into line with the conflicting changes in mainline. The resulting branch with that change applied probably won't build, which means bisect pain for people who happen to bisect to that point. 2. botch the merge, publish the tree, and then hit heads together to try and get the problem resolved - again resulting in a commit point (the merge) which is nonsense and unbuildable. 3. rebase the branch on top of the conflicting change, throw out the patches which prove to be a problem and ask the original author of those patches to fix them up for the conflicting change. The result is a completely bisectable tree. (3) is the solution which I chose, and it worked _extremely_ well. (3) is effectively what akpm does with his tree - when a patch conflicts with other changes, he throws the changes out and bangs peoples heads together to get a new set of patches generated which work together. In that respect, it's no different, and it's been proven there to work well. So I see git as being no different. -- Russell King Linux kernel 2.6 ARM Linux - http://www.arm.linux.org.uk/ maintainer of: --
On Tue, 12 Feb 2008 20:03:54 +0000 Actually I'll usually (much >90%) just queue another patch which fixes the fallout and will drop that once the upstreams sort things out. I'll only drop a subsystem tree in extremis. One reason for this is that I'll have a pile of other patches which are dependent upon that subsystem tree and which would need reworking. And un-reworking once the subsytem tree is undropped. And the trees which are most likely to cause conflicts tend to also be the trees upon which other patches are dependent. Mainly because those trees are larger and more spread out. --
Don't get me wrong at all. Rebasing is fine for stuff you have committed yourself (which I assume was the case here). Rebasing is also a fine conflict resolution strategy when you try to basically turn a "big and complex one-time merge conflict" into "multiple much smaller ones by doing them one commit at a time". But what rebasing is _not_ is a fine "default strategy", especially if Right. And it's one of the fundamental differences between git and a patch queue. Patch queues are very flexible, but they simply don't scale. They don't scale in history (ie you cannot sanely keep track of multiple queues as they grow in the long run - you need a way to regularly "freeze" things into a release tar-ball or something like that). But they also don't scale with number of users - even just read-only ones. The latter example is something we see in -mm right now. When people report problems against -mm, it's usually an all-or-nothing thing (so-and-so mm release doesn't work), but even when people sometimes bisect to a specific point in mm, it's not "stable" in the sense that that point may not make any sense in a subsequent -mm queue version. And all of these issues are not about "-mm" per se, but are about patch queues in general. And "rebase" turns a git repository effectively to a patch queue too, with all the same downsides. And I don't think patch queues are evil: I use git rebase all the time myself (not for the kernel, but for git, where I'm not the top-level thing). But I'd not rebase commits from other peoples trees: I rebase commits that I applied as patches (ie the *author* may be somebody else, but the committer is me!) or that I committed myself, and that I haven't pushed out. Note that difference between "committer" and "author". There's nothing wrong with rebasing commits that are _authored_ by other people. But there absolutely is something wrong rebasing commits that are _commmitted_ by somebody else, unless you two are best ...
I'm sure I understand only less than half of what you said.
My current understanding is that all the aspects of your proposal that
are interesting for me can be summarized as follows:
- your tree stops compiling at the beginning of the merge window when
you merge the first subsystem tree
- your tree might compile again at the end of the merge window when you
merge the last subsystem tree
- git-bisect -> /dev/null
cu
Adrian
--
"Is there not promise of rain?" Ling Tan asked suddenly out
of the darkness. There had been need of rain for many days.
"Only a promise," Lao Er said.
Pearl S. Buck - Dragon Seed
--
From: Jeff Garzik <jeff@garzik.org> Understood. One of the key operations that I'm interested in is removing things from the history. If I could do that using GIT without any side effects and in a way that really would remove it from the tree, I would do that in a heartbeat. At 1500 changesets, a merge conflict shows up about once every day or two as 2.6.N nears it's release into final as bug fixes trickle in. I find using GIT to fixup merge errors on a tree of that scale to be really painful. And it only fixes up the final result in a merge changeset. Just like we expect submitters to refresh their patches as things change upstream, we as developers should be able to refresh trees of changes we are maintaining for similar effect. This is all made difficult because GIT's commit IDs are tied to where they are in the tree, and thus are dependant upon the wholeness of parents and subsequently their parents. That's why it's hard to "just remove" something completely without rebasing the tree. Let me give you a good example, just yesterday I had to rebase my net-2.6 tree a few times. It's because I put a patch in there that the person said was OK but it broke the build. There is zero reason for me to push that tree to Linus with the bad commit and the revert, it's just noise and makes the tree harder for people to review. Now, to preserver your tree's history I simply reapplied the stuff I wanted then repulled your tree. But right now I have to redo the include/linux/rcupdate.h that's in there because it has a bug. I now want to rebase again to just kick out the existing rcupdate.h changeset, because I know of no alternative that really lets me get rid of the original bogus changeset. --
Perhaps you need to switch to using quilt. This is the main reason why I use it. thanks, greg k-h --
Btw, on that note: if some quilt user can send an "annotated history file" of their quilt usage, it's something that git really can do, and I'll see if I can merge (or rather, coax Junio to merge) the relevant part of stgit to make it possible to just basically get "quilt behaviour" for the parts of a git tree that you haven't pushed out yet. A pure patch-stack will be faster at that thing than git would be (it's simply easier to just track patches), but on the other hand, using git would get some other advantages outside of the integration issue (eg the cherry-pick thing really is a proper three-way merge, not just an "apply patch", so it can do better). It wasn't the original goal of git, but not only are really doing all the series management anyway (that is largely what "rebase" is all about, after all), but the git goals have obviously expanded over time too. Linus --
From: Linus Torvalds <torvalds@linux-foundation.org> Sounds interesting and nice, but not relevant for my current workflow. I've always "already pushed out". --
So this is what I do for ext4 development. We maintain a quilt series in git, which is located here at: http://repo.or.cz/w/ext4-patch-queue.git A number of ext4 developers have write access to commit into that tree, and we coordinate amongst ourselves and on linux-ext4@vger.kernel.org. I tend to suck it into git using the "guilt" package, and do periodic merge testing with a number of git queues to detect potential merge conflicts. Not as many as James does, but I may start doing more of that once I steal his scripts. :-) The patch queue also gets automatic testing on a number different platforms; for that reason the series files comments which version of the kernel it was last based off of, so the ABAT system can know what version of the kernel to use as the base of the quilt series. I do a fair amount of QA, including copy editing and in some cases rewriting the patch descriptions (which are often pretty vile, due to a number of the ext4 developers not being native English speakers; not their fault, but more than once I've had no idea what the patch description is trying to say until I read through the patch very closely, which is also good for me to do from a code QA point of view :-). Periodically, the patch queue gets pushed into the ext4.git tree and as a patch series on ftp.kernel.org. I've never been very happy with stgit because of past experiences which has scarred me when it got get confused and lost my entire patch series (this was before git reflogs, so recovery was.... interesting). There's always been something deeply comforting about having the ASCII patch series since it's easy to back it up and know you're in no danger of losing everything in case of a bug. Also, having the patch series stored in ASCII as a quilt stack means that we can store the quilt stack itself in git, and with repo.or.cz it allows us to have multiple write access to the shared quilt stack, while still giving us the off-line access advantages of git. (Yes, I've spent plane ...
It got much better now :-). We are even working on transactions support There is "stg export" which could be made to export a patch As I said in a different e-mail, we got patch history tracking in StGIT. You can even annotate specific changes. For editing, there is "stg edit [--diff]" which allows both description and patch changes. However, the StGIT approach is not fully suited for multiple developers sharing the same set of patches, especially if they are allowed to modify the same patches. For this kind of workflow, you can export the series and add the patches to a separate shared repository. StGIT has a "sync" command to synchronise changes to patches in different series (either stored as quilt series or in a different branch). -- Catalin --
Ted's description matches mine (keep quilt tree in git, edit changelog entries, rebase on newer kernel versions, etc.) I can go into details I was amazed at how slow stgit was when I tried it out. I use git-quiltimport a lot and I don't think it's any slower than just using quilt on its own. So I think that the speed issue should be the same. I had a number of issues last time I tried stgit out, but maybe they are now resolved, I'll try it out tomorrow and report to the git list anything I find that doesn't work for me. thanks, greg k-h --
Ack. Same for PS3 and m68k (except I don't have the m68k patches in git (yet)).
Two issues with using quilt:
1. Sometimes a patch still applies after it was integrated upstream,
2. Sometimes it's a lot of work to fixup the patches after an upstream
change. I wrote a script to do a tree-way-merge in case of a
conflict, but I haven't really tested it yet (not suitable big
conflict has happened since ;-)
Gr{oetje,eeting}s,
Geert
--
Geert Uytterhoeven -- There's lots of Linux beyond ia32 -- geert@linux-m68k.org
In personal conversations with technical people, I call myself a hacker. But
when I'm talking to journalists I just say "programmer" or something like that.
-- Linus Torvalds
--
Add QUILT_PATCH_OPTS="--fuzz=0" to your ~/.quiltrc file. The default is fuzz=2 IIRC which tries to be too clever. I set fuzz to 0 after I got burned by an unnoticed "quilt patched it somewhere else" bug, which took me half a day to figure out. Thanks, tglx --
I like using "guilt" because I can easily reapply the patchset using "guilt push -a", which is just slightly fewer characters to type than "git-quiltimport". This also means that I don't need to switch back and forth between "git mode" and "quilt mode" when I'm editing the patches (either directly by editing the patch files, in which case afterwards I do a "guilt pop -a; guilt push -a", or by using "guilt pop", "guilt push", and "guilt refresh"). "guilt push -a" is a little bit slower than "quilt push -a", but not enough to be seriously annoying. And besides, "guilt pop -a" is slightly faster than "quilt pop -a". Using guilt is also nice because there is a bit of additional backup for previous work via the git reflogs, although to be honest I've rarely needed to use it. - Ted --
I had problems getting guilt to preserve metadata properly last time I tried it, and it forced me to work with the same location and format that the original developers used, which wasn't as flexable as quilt could handle from what I recall. But I'll try it again, it might have gotten better... thanks, greg k-h --
I added some time ago patch history tracking in stgit and you can run "stg log [--graphical] <patch>" to see all the changes for that patch (as a list or via gitk). This is done by keeping a separate DAG of It shouldn't be slower than git-quiltimport (at least the recent stgit versions) as they seem to have a similar approach (using "git apply"). There is probably an extra check stgit does for local changes before starting the import. Otherwise, just use git-quiltimport and "stg uncommit" to generate the patches. StGIT approach for pushing patches is to use git-apply and, only if this fails, switch to a three-way merge. These days it seems that the three-way merge is pretty fast anyway, we might drop the former (after Please try the last stable release, 0.14. The current HEAD has some restructuring done (but gets nice features like transactions, undo). Thanks. -- Catalin --
On Tue, 12 Feb 2008 15:58:53 -0800 (PST) Well there's a case in point. rcupdate.h is not a part of networking, and it is random tree-wandering like this which causes me problems and which will cause Stephen problems. Now, I don't know which tree "owns" rcupdate.h but it ain't networking. Probably git-sched. Nothing in networking depends upon that change (which has a typo in the comment, btw) hence it can and should have gone through whichever-tree-owns-that-file. For Stephen's sake: please. --
From: Andrew Morton <akpm@linux-foundation.org> At least thie time I did make sure that change got posted to linux-kernel and got properly reviewed by the de-facto maintainer (Paul McKenney). :-) I'll toss it. But how do I do that using GIT without rebasing and without having this ugly changeset and revert in there? That's the thing I want answered, and although Al claims it does, git cherry-pick does not seem to do what I want either. --
While I was there I spotted a howling bug in rcu_assign_pointer(): a double-touch of the second arg. Nobody has done rcu_assign_pointer(p, something_with_side_effects); before? That would be surpising... Who, me? umm, get git changed? It seems pretty clear that it isn't matching legitimate kernel development workflow. And it is a tool's job to --
Heh. I've had the reverse situation: "git rebase" often results in *more* conflicts than "git merge" (ie "pull"). But one issue is also that when conflicts happen, different people are used to different things. I'm particularly used to merge-type conflicts, and in fact there's some fairly advanced support in git for helping resolve them that people who *don't* do merge-level conflict resolution may not even be aware of. In particular, if you want to try it, do something that conflicts and then do gitk --merge to see what the conflict is all about. That is just fancy shorthand for gitk HEAD...MERGE_HEAD -- <list of conflicting files> so what it does is to show only the relevant history (the three dots means that it's a _symmetric_ set difference from HEAD to MERGE_HEAD) for the merge, and only for the particular files that had conflicts! This often means that even when you merge a thousand commits (or the thing you merge *into* has thousands of commits since the merge base, which is the common case for me), you actually only see a couple of commits - only the ones that actually modified the conflicting files! (If you have many files that conflict, you can further narrow it down to just one at a time by explicitly listing the file/directory you want to Actually, better than rebase in that situation is to just remove the bad commit. Yes, you'd use "rebase" for it, but you'd use it not to move the whole series to a newer place, you'd use it just to rebase the commits *after* the commit you remove. This is something where I actually think git could and should do better: git has the capability to act as more of a "quilt replacement", but because it wasn't part of the original design, we never actualy exposed the simple queue management commands to do this (stgit does things like that, though). So if you haven't pushed out, right now you'd have to do this stupid thing: [ start (and activate) a 'fixup' branch at X ] git checkout ...
From: Linus Torvalds <torvalds@linux-foundation.org> Thanks for all the useful info. But as soon as I've applied any patches to my tree I've "pushed out". So this scheme doesn't work for me. The first thing I do when I have changes to apply is clone a tree locally and on master.kernel.org, then I apply that first patch locally and push it out to master. What would be really cool is if you could do the rebase thing, push that to a remote tree you were already pushing into and others could pull from that and all the right things happen. A rebase is just a series of events, and those could propagate to people who are pulling from the tree. I'm pretty sure GIT could support this. --
I actually suggest you literally delay your push-out. I don't generally delay things by a lot, but I tend to try to at least do a compile in between pushing out - and even if I've pulled something in between the thing that broke, I'll just "git reset --hard" to a working state if something broke, and just re-pull instead of even trying to rebase or anything like that. (IOW, I often find it much easier to just start over and re-do than actually doing a rebase). I don't do it all the time, by any means, but there's really no huge reason to push out all the time. And that's doubly true for subsystem maintainers. Quite often, the right thing to do is to only push out when It would also be really cool if Claudia Schiffer had decided that hiding under my desk is a good idea. IOW, you really haven't though that through. That is how TLA and darcs worked, and it's a total disaster. Trust me, you don't know how good you have it. Linus --
From: Linus Torvalds <torvalds@linux-foundation.org> I know, preserving history is valuable. I'll take up the various suggestions and try working a little differently this time around. We'll see how well it works. --
Yes, this is exactly the feature I'm looking for. It would allow the downstream users of a rebased tree to rebase themselves correctly. All the information about the rebase is in the reflog ... it can't be too difficult to pass it through on a pull and allow the downstream tree to do the right thing. James --
Guys, you simply have no idea what you're talking about. Those downstream trees can have done things themselves. They *depended* on the state you gave them. You can't just say "oops, I lied, this is the state you should have used, now it's _your_ mess to sort out". OF COURSE it's what you'd like to use - it absolves you of any and all actual responsibility. But dammit, that's not what anybody else wants than the irresponsible person who cannot be bothered to stand up for his work! If you're not confident enough about your work, don't push it out! It's that simple. Pushing out to a public branch is a small "release". Have the f*cking back-bone to be able to stand behind what you did! Linus --
Erm, I would like this feature as a downstream user. I'm not asking for this to be the default or even easily available. However, when you know you've based a downstream tree on what you know to be a volatile base, it would be very useful information to have. Right at the moment, I maintain a <branch> and a <branch>-base and simply cherry pick the commits between the two to do the right thing when I know my volatile base has changed. It would be very helpful to have a version of rebase that new my base had been rebased. Basing on a tree I know to be volatile is sometimes a development decision I make as a downstream user ... I'm just wishing the tools could help me handle the problems better. James --
Hey, I know, you could use.. drumroll..
"git rebase"
I know that's a big leap of faith, to use git rebase for rebasing, but
there you have it. Us git people are kind of odd that way.
IOW, if you know the old broken base, and the new base, just do
git rebase --onto newbase oldbase
and it should do exactly that (basically lots of automated cherry-picks).
[ But the fact is, if you did anything fancy (like pulled in other peoples
work), you cannot sanely rebase _those_ peoples work. They didn't screw
up to begin with! You can play with "git rebase -i --preserve-merges",
of course, but I really think you're doing something wrong if you start
pulling other peoples work into an unstable thing, so while it may work,
I'd strongly suggest against even trying, because the problem is your
workflow ]
So let's say that you have a remote branch that you track that goes
rebasing (let's call it "origin/pu" to match the real-life git behaviour),
then you should literally be able to do
old=$(git rev-parse origin/pu) &&
git fetch &&
new=$(git rev-parse origin/pu) &&
git rebase --onto $new $old
and no, I didn't actually test it, but hey, it really should be that
simple.
[ And no, you don't really need to do those "old=" and "new=" things, they
are there to make it explicit - you could easily just have done
git fetch
.. oh, noticed that origin/pu changed ..
git rebase --onto origin/pu origin/pu@{1}
where we just let git take care of the old/new itself using the reflog,
so that "origin/pu@{1}" assumes that you just know that the only thing
that has changed origin/pu was that previous "git fetch", and that
really *did* change it. ]
In other words, git does give you exactly what you want, but nothing
really changes the fact that you should only rebase like this only if:
- you haven't already exported the result (or only exported it as those
unstables branches that people know to avoid)
- your changes on top .....and in case it wasn't clear - this is just a general way of saying "move the commits on this branch since $old to be based on top of $new" instead. You can pick out those old/new commit ID's using gitk or whatever if you wish. Neither the $new or the $old needs to even be an existing branch - just pick them with gitk. So if you literally want to just move the top 5 commits (assuming those top five cmmits are just a nice linear thing you did) from the current branch to be on top on another branch instead, you can literally do this: # save this state, maybe we want to keep it around. Call it "old" git branch old-branch # rebase the top five commits onto $target git rebase --onto $target HEAD~5 ta-daa - all done. The branch you are on will now have been rewritten to be the top five commits moved to be on top of the $target you chose, and if you want to get back the old state, it's nicely squirrelled away in "old-branch". (That obviously assumes no merge conflicts - you'll have to resolve those yourself ;) Of course, if you didn't even want to save the old branch, just skip the first step. If you have reflogs enabled (and git does that by default in any half-way recent version), you can always find it again, even without having to do "git fsck --lost-found", at least as long as you don't delete that branch, and it hasn't gotten pruned away (kept around for the next 90 days by default, iirc) Linus --
Even if you delete that branch, the "HEAD" reflog will still contain it, since it is separate from any particular branch reflog. Nicolas --
OK, smarty-pants ... that works nicely, thanks! I'm used to maintaining <branch> and <branch>-base, so this probably suits my workflow better than getting the information from the reflog. It wasn't clear from the git rebase man page that it would work like that. James --
Or, with Git 1.5.4, just: git pull --rebase And I didn't test it yet either. Same caveats do apply. Nicolas --
There's also a difference between a downstream user and a downstream developer. While rebasing does cause trouble for folks doing downstream development off of the tree in question, there's no reason why this should be the case for users that are simply trying to follow the tree. I push changes out to my tree so people have a chance to poke through it and to see what's going on, though I do not generally encourage people to fork off of it given that I end up rebasing periodically. At the moment the options seem to be down to the following: 1 - Push changes out without rebasing 2 - Push changes out periodically rebased 3 - Hide the tree away in my home directory on hera 4 - Force people to get at the current tree through -mm 4) generally isn't very realistic, given that -mm releases have far too many other changes and the releases are quite spread out. 3) is doable, but I publish the tree as a convenience to the folks wanting to see what's going on in the tree, and I would rather continue doing so. This leaves 2) my current workflow, and 1) which ends up creating a lot of extra metadata. The common cases thelre are patches + reversions and merge points. Holding on to a patch for some period of time before pushing it out to ensure that there won't be a reversion later rarely tends to work in practice. Most of the time I end up having to revert something it's because someone else found a problem with a given change _after_ the it was pushed out, and which was not caught with my local tree or testing. On the other hand, perhaps it's also useful to see the reversions in the history, particularly to see what the rationale for the change not working out was (which could be helpful for people working on the same stuff later on). This then leaves merge points. During merge window time people are pulling on a pretty frequent basis, which also breaks down to a few fairly common cases: 1 - Bringing in new stuff to be supported (ie, system calls). 2 - Infrastructure ...
git checkout -b new-tree old-tree # hack on new-tree: rebase, drop bad commits, whatever git merge -s ours old-tree git push your-public-repo new-tree:public-tree Now public-tree has a merge commit on the top with two parents, public-tree^1 and public-tree^2. public-tree^1 is the tip of the new cleaned-up history, and public-tree^2 points to the old stuff. I sometimes use that privately as a way to keep track of the history of a patch-series, but I assume Linus would shoot anyone that tried to submit such a monstrosity upstream.... --b. --
Even then, it will not work. Again, Roland isn't going to want to always pull in my driver tree just to build his tree. He wants to, and needs to do his own development effort. But when we merge them together, there would be problems. So, you can't just "drop" the IB tree. You can't just "drip" my tree. Where do you "fix this up" at? I can send a patch for the IB tree, but Roland can't put it in his tree, and I can't put it in my tree, it needs to go _after_ both of our trees. That's what -mm has been able to handle so far, and that needs to also work with -next. thanks, greg k-h --
Actually, we had exactly this issue with the SCSI bidirectional patches: They depended on the sg_table patches in block. The solution I adopted was two merge trees: One to go in immediately with no dependencies (scsi-misc-2.6) and the other based on the pieces of block (so it would compile and apply) to go in mid way through the merge round after block (scsi-bidi-2.6). What I did was keep rebasing the bidi tree until I could see there was nothing other than a plane base before merging it. Of course, this only worked because Jens has a git tree ... it would have been a lot harder (but not impossible) if I'd had entangled patches from a quilt tree. So I've already proven that the split tree solution is viable, if not pretty. The bidi tree had to be rebased an awful lot as the block trees changed and rebased. Unfortunately, git isn't very good at this, I eventually had to keep a base and a top reference and just try to cherry pick this series into the new constructed block tree. But it can be Actually, we never successfully got block and bidi via -mm. James --
My take on this, in retrospect, is that the code should probably have been developed in one branch off of one of the trees, or maybe even better in a third tree. The point is that the structure of git trees followed the organizational structure rather than the problem at hand and if contributions coming from different trees depend on each other, staging branches should be created in the integration tree for working out the conflicts and when ready, I developed git-rebase-tree (http://git.bhalevy.com/git/gitweb.cgi?p=git-tools.git;a=blob_plain;f=git-rebase-tree;h...) for exactly this reason - frequently rebasing sub-branches in the linux-pnfs tree and my experience so far is that it really speeds things up for me and Boaz. Please feel invited to try it out, I think it's very to close to be mature enough for general availability (I know, I know, it'd be perfect if this functionality --
For your development, possibly, but not for my merge ... there were a lot of other patches besides yours in the bidi tree (it was, in fact, misnamed, but I thought at the time I created it it would only contain Actually, I think you'll find the problem is that we can't build an integration tree that's both stable and long lived, and hence you can't base anything off it that requires a long lived base. So, the way I worked for your patches was to use the -mc tree to identify my conflicts, and only include the actual conflicting branches as a basis for the scsi-bidi tree beofre fixing up the patches. I then actually used the -mc tree as a canary to tell me when to rebase. James --
From: Greg KH <greg@kroah.com> Totally agreed. The fact is there are interdependencies, especially in driver land and you have to either: 1) Make the driver folks work on top of Greg's tree. 2) Constantly rebase the -next tree to deal with the conflicts. There are some other issues related to this which haven't be touched upon greatly yet. I rebase my tree all the time, at least once or twice per week. Why? Firstly, to remove crap. When you have "great idea A" then "oh shit A won't work, revert that" there is zero sense in keeping both changesets around. Secondly, I want to fix up the rejects caused by conflicts with upstream bug fixes and the like (and there are tons when the tree gets to 1500 or so patches like the networking did). I don't want git to merge the thing by hand, I want to see what the conflict is and make sure the "obvious" resolution is OK and the most efficient way I know how to do that is to suck my tree apart as patches, then suck them back into a fresh tree. It therefore might make sense to the linux-next tree to do something similar, constantly rebasing so that all the conflicts and reverted shit changes can be sorted out without having an incredibly ugly GIT history. --
FWIW, that is annoying and painful for us downstream jobbers, since it isn't really how git was meant to be used. You use it more like a patch queue, where commits are very fluid. Unfortunately, if there is any synchronization lag between me and you -- not uncommon -- then I cannot commit changes on top of the changes just sent, in my own local tree. Why? Because you rebase so often, I cannot even locally commit dependent patches due to the end result merge getting so nasty. I understand the desire to want a nice and clean history, but the frequency here really has a negative impact on your downstreams. It also totally screws the commit statistics, wiping me and John and the committers we have preserved out, replacing everybody's committer with David Miller. Jeff --
FWIW, I definitely have a (vocal minority) group of contributors who resent all the rebasing. There may be a few cases of 'vanity' represented here, but there are definitely reasonable complaints about not being able to do simple pulls to stay up-to-date and/or having to rebase before they can send patches to me. FWIW, I think it might save a bit of my time as well, although I have become pretty good and "riding the tide" of rebasing... :-( Unfortunately, I don't have any good suggestions to remedy the issue. One strategy might be a third layer of trees, e.g.: net-2.6 fixes for the current release net-2.6.26 updates certain to go to the next release net-2.6.26-maybe updates that might not make it to the next release Of course, managing what goes moves-up from -maybe is probably a big headache, and just sucks-up more of Dave's time. And, of course, virtually no one will run the -maybe tree... Just my $0.02... John -- John W. Linville linville@tuxdriver.com --
Definitely... that is the sort of stuff that upstream folks never see when they rebase -- it causes an explosion of downstream rebasing, where each rebase must wait for the one-step-up upstream to rebase, before patches can be sent. It's easy for the rebasing to get exponential quickly, the more downstream levels you have. Jeff --
From: "John W. Linville" <linville@tuxdriver.com> If I knew something was "maybe" ahead of time I simply would not apply it. Everything I apply to my tree I feel is likely to get integrated. If it isn't, I let the patch work itself out on the lists and amongst the interested developers. The real issue is deleting crap. Once something that seemed like a good idea turns sour I want to remove it entirely. This isn't doable without a rebase. Also, and Andrew does this a lot, I want to clean up changes which have problems I only notice later. In fact the rebase process turns up all kinds of things such as whitespace errors that I get when merging other people's trees. Having extra changesets with the small fixups adds zero value and just more churn to go over when reading changesets. A small annotation to the changelog will do, that answers the "where did this come from?". With patches and rebasing that operation is easy, and I'm willing to deal with all of the rebasing of subsequent changesets that is usually caused by such things. Yes, I'm even willing to do it to patch #1 in a 1500 patch tree. :-) --
But the "author" is still preserved, right? Why do you need the committer name to be preserved? (I'm not denying that there could be reasons, I'm just curious what they are.) --b. --
It's not that the committer should be preserved, but: - the chain from author -> committer should be visible in the Signed-off-by: lines. If you rebase somebody elses tree, you screw that up. You need to add your sign-off, since now *you* are the new committer, and *you* took somebody elses work! - you should respect the down-stream developer, and if that downstream developer continues to work with his branch or works with other people, you shouldn't screw that up! Both of those basically say that you should never rebase somebody elses work. You can use rebase to rebase your *own* work on top of somebody elses thing (since that doesn't change the sign-off chain, and you still respect the downstream developers development model)! But of course, if you rebase, you should respect the wishes of the up-stream developer too. I don't do rebases. So if you asked me to pull, the stuff I pulled can never be rebased, because it just *is* in my tree. Put another way: think of the absolute *chaos* that would happen if I were to rebase instead of just merging. Every time I pull from you I'd invalidate your whole tree, and you'd have to re-generate. It gets unmaintainable very quickly. And that's actually ignoring a real issue: stability of commits. The nice thing about stable commit naming is that all bug-reports from other people that told where the bug happened are basically 100% trust-worthy and the code is 100% reproducible not just for you, but for everybody else. In other words, you really shouldn't rebase stuff that has been exposed anywhere outside of your own private tree. But *within* your own private tree, and within the commits that have never seen the light of day, rebasing is fine. (And yes, there are exceptions. If it's a clear "throw-away tree" all the rules go out the window, of course, as long as everybody involved *knows* it's a throw-away tree, and know that if they pull it they have to synchronise 100% ...
From: Linus Torvalds <torvalds@linux-foundation.org> I agree with this and that is exactly what I screwed up by mistake this time around. Normally when I rebase I walk through the patches that came from other people's trees and add signoffs as needed. I understand that this I actually wouldn't mind that, the first thing I do when sending a pull request is I stop putting things into my tree and as soon as the recipient pulls I wipe out my tree and clone a fresh copy of their's. It's really not a big deal. The pusher can queue patches and other stuff up in their mailbox or in a directory somewhere. This quiet period also allows those patches to have some time to be reviewed on the lists before they actually end up in anyone's tree. I really like that mode of operation. --
*YOU* like it, because it never generates any issues for *you*. You're the top in your heap, and the people above you don't do that insane thing, so you get all of the advantages, with none of the downsides. Of *course* you like it. But as people have pointed out, it generates issues for the people under you! If I did it, the people who now complain about networking would not just be a couple, it would be everybody. Nobody could depend on anything out there, because everything would have to rebase. You just don't see the problems, because the only person above you isn't crazy enough to do what you propose. You also don't do ten merges a day of subsystems you don't know. The importance of merging (rather, not screwing up history in general) becomes really obvious when things go tits-up. Then they go tits-up *without* screwing up the history of the trees that were hopefully tested individually. If you re-base things that others developed, you lose that. Imagine if I merged first Greg's tree (by rebasing), and then there was some fundamental thing that didn't cause a conflict, but just made something not work, when I rebased yours on top. Think about what happens. Now I've merged (say) 1500 networking-related commits by rebasing, but because I rebased on top of Greg's tree that I had also rebased, absolutely *none* of that has been tested in any shape of form. I'd not use most of the things I pulled, so I'd never see it, I'd just push out something that was very different from *both* trees I pulled, with no way to really blame the merge - because it doesn't even exist. So as a result, some *random* commit that was actually fine on its own has now become a bug, just because it was re-written. You don't see the problem here? Yes, this is the *crap* you do all the time. You don't see the problems as much, because you merge probably only about a tenth of the volume I merge, and you can keep track of the subsystem more. But even though you ...
From: Linus Torvalds <torvalds@linux-foundation.org> Good point. Now how do I remove a bogus commit for a tree that I've already pushed out and published for other people, without any record of it appearing in the GIT tree any more? How do I insert build fixes into existing changesets so that the tree is more bisectable? If Jeff merged in a tree that introduced a ton of whitespace errors git is complaing about, is there a way I can fixup that changeset in-place? (or should I tell Jeff to start adhering to GIT's whitespace warning messages when he applies patches?) Those are basically the major operations that would allow me to seriously consider rebasing a ton less often. --
So, the answer is: if others have actually pulled, it's simply not possible. There simply is no way to undo history that isn't local any more. You just *have* to revert, or you need to find every single person that pulled (directly or indirectly) and ask them to undo all the work they did on top of your bad commit. This is not git-related, btw. That's just how the world works. It's a bit like the internet - if you said something stupid on #IRC and it made it to bash.org, there's not a whole lot you can do to "undo" your stupidity. You Just delay pushing out. There really is _zero_ downside to this. None at Umm. Git doesn't complain about whitespace _except_ when applying patches. So if you don't rebase his work, you'll never see the whitespace warnings either! Of course, we'd probably wish that Jeff cared about the whitespace warnings and pushed back on them, but the fact is, that warning isn't meant for you - because by the time you pull Jeff's tree, it's simply not your issue any more. Jeff already applied it. Either he's your trusted lietenant or he's not. Quite frankly, to me it sounds like you're not ready to "let go" and trust the people under you. Trust me, it's worth it. It's why your life is easy: I have let go and I trust you. Also, I'd *much* rather have a few problems in the tree than have people screw up history in order to hide them. Sure, we want to keep things bisectable, but quite frankly, if you do a reasonable job and compile the kernel before you push out, it will be "mostly bisectable". And yes, mistakes happen. Mistakes will *always* happen. It's ok. Relax. Let me put it another way: You're _both_ going to be *much* better off pushing back on Jeff, telling him that "I can't pull from you because your tree is ugly and doesn't compile", than taking his tree and rebasing it. Remember? I used to do that all the time. I berated the ACPI people for creating monster trees that were horrible and contained fifteen ...
One of the things which linux-next could/should do is to help weed out the silly build breaks, typos, missing documentation updates, missed checkpatch opportunities, etc, etc. As well as real bugs. So it would not be efficient for David to do all this queue-cleaning *prior* to putting the tree into linux-next, because more stuff will pop up anyway. So perhaps a better workflow would be keep the linux-next trees all messy, and then each developer can consolidate, rebase, join and drop things prior to sending their individual trees to Linus. If so, then git could perhaps help, by retaining sufficient metadata for the maintainer to track the fact that this-patch-here fixes that-patch-there. --
Well, what others have done is to have special "temporary branches". This is what git itself does, for example. The "pu" branch in git is used for experimental stuff, and it's _declared_ to be rebased, redone, and generally just unsafe at any moment. So it is easy to have a special "testing" branch that is just declared to be unsafe. Make Linux-next pull that testing branch - it will pollute the Linux-next tree (and anybody else who just wants to see what the current state is), but since those are re-generatd from scratch every day _anyway_, so who cares? But don't make it something people pull by mistake (ie never call it "master", and when mentioning it in some email message, always mention the fact that it's not a stable branch, and never ask anybody to pull it without making it very clear that it's just for testing, not for real merging). So git does have support for those things. They are very much "secondary" branches (any tree they are pulled into will itself become "poisoned" and unstable), but it's easy enough to have something like that for testing purposes. And if it all tests out fine, you can just move it as-is into the "real" branch if you want to. Linus --
From: Andrew Morton <akpm@linux-foundation.org> We could do that, but I believe Linus's main point is that if we go and change the trees then in the end we're merging in something different than what was actually tested, namely all the original trees as merged into linux-next. And I kind of agree with that. --
Make the distinction earlier. With ocfs2 and configfs (we got
this scheme from Jeff), we keep the topic branches as "unsafe" - that
is, officially rebaseable . We merge them all into a big "ALL" branch,
which is also "unsafe". Andrew pulls this for -mm, and it gets tested
here. If there is a brown-paper-bag problem, we can tell the original
author to fix it. Then we re-pull the topic - effectively a rebase.
The ALL is also rebased. But that's Ok, it will never go towards Linus.
When a topic is considered worthy of going upstream, we pull it
to a branch called "upstream-linus". This branch is *NEVER* rebased.
Now that the topic is in upstream-linus, the original topic branch can't
be rebased either. So any fixes to that topic going forward will stay
in the history. Since that topic was pulled into ALL for testing, we
are using the identical commits that got tested.
Joel
--
"I have never let my schooling interfere with my education."
- Mark Twain
Joel Becker
Principal Software Developer
Oracle
E-mail: joel.becker@oracle.com
Phone: (650) 506-8127
--
This is essentially the same process I've been using in wireless-2.6 with the (regularly rebased) 'everything' branch. Still I find that it causes lots of confusion and complaining. Perhaps I am not communicating clearly enough... :-) Do you find that people are happy with that process? Forgive me for not knowing, but how many developers are actively (or occasionaly) involved in ocfs2 and configfs? How many 'normal' users pull your tree looking for 'latest and greatest' code? John -- John W. Linville linville@tuxdriver.com --
Oh, I have no pretense that ocfs2 is as busy as wireless or net. Certainly Andrew has no problem with it. People I've interacted with while working on features, etc, also get it, but I'm usually pointing them to it up-front. I think you'd probably have lessons that larger trees could learn from. For example, do you have a 'things that will go to linus soon" branch that people can grab - something that's now stable/non-rebasing? Do you advertise the rebaseableness of "everything" in the web page where you say "grab 'everthing' for the latest stuff"? Joel -- "The suffering man ought really to consume his own smoke; there is no good in emitting smoke till you have made it into fire." - thomas carlyle Joel Becker Principal Software Developer Oracle E-mail: joel.becker@oracle.com Phone: (650) 506-8127 --
If there was a "fundamental thing that didn't cause a conflict", then the two trees in question probably didn't touch the same code, so would probably merge cleanly, for the same reason that one rebased onto the other cleanly. But depending on what the "fundamental thing" was, the merge might still introduce the same bug, right? --b. --
Absolutely. But if you do a true merge, the bug is clearly in the merge (automatedly clean or not), and the blame is there too. IOW, you can blame me for screwing up. Now, I will say "oh, me bad, I didn't realize how subtle the interaction was", so it's not like I'll be all that contrite, but at least it's obvious where the blame lies. In contrast, when you rebase, the same problem happens, but now a totally innocent commit is blamed just because it happened to no longer work in the location it was not tested in. The person who wrote that commit, the people who tested it and said it works, all that work is now basically worthless: the testing was done with another version, the original patch is bad, and the history and _reason_ for it being bad has been lost. And there's literally nothing left to indicate the fact that the patch and the testing _used_ to be perfectly valid. That may not sound like such a big deal, but what does that make of code review and tested-by, and the like? It just makes a mockery of trying to do a good job testing any sub-trees, when you know that eventually it will all quite possibly be pointless, and the fact that maybe the networking tree was tested exhaustively is all totally moot, because in the end the stuff that hit the main tree is something else altogether? I don't know about you, but I'd personally be really disappointed if it happened to me, and I felt that I did a really good job as a submaintainer. I'd also feel that the source control management sucked. Contrast that to the case where somebody simply does a merge error. The original work doesn't lose it's validity - so the original maintainer hasn't lost anything. And quite frankly, even the person who "screwed up" with the merge hasn't really done anything bad: these things _do_ happen. So bugs happen; not big deal. But the fact that the bugs are correctly attributed - or rather, not mis-attributed to somebody blameless - that _is_ a big deal. It's ...
The obvious advantage to rebasing in this case is that the blame (misplaced though it may be), at least lands on a commit that made a single small change, likely making the problem easier to diagnose. (As opposed to the case of a large merge, where all you may know is that somewhere in the hundreds of commits done on one side of the merge there was a conflict with the hundreds of commits on the other side.) I think a lot of people would see rebasing as an acceptable tradeof that gives up a small amount of accuracy in assigning blame to individuals in return for a large increase in ability to debug problems. I suppose one response to that would be that it's important that people learn how to work in parallel, that failures to do so are particularly important failures in the process, and that it's therefore worth it to make sure that such failures are always identified specifically as merge failures. It would be nice if merges, like patches, were broken up into somewhat smaller units. There's an understandable desire to wait to the last minute to actually commit to one's commits, but a willingness to do so a little earlier might avoid some of the problems that seem to come from having a lot of large merges happen all at once. --b. --
One idea that I thought about when debating rebase vs. merge (and it's far far from being fully baked) is versioned commits. The gist of it is that patches are assigned an hash identifier like today when they are first committed into the tree, but, and this is the main change: if they mutate, e.g. by a rebase, or even git commit --amend, their version is bumped up rather than SHA changed. This way all versions of the commit would be accessible and addressable using their respective SHA *and* version. I think that this can help keep the tree's history in a more intuitive way (since the patches' base identifier won't change, just its version number), and you get a bonus of seeing each commit's history, who changed it, and what was the change. Benny --
The SHA1 is uniquely determined by the contents of that commit--commit and author names and times, changelog message, snapshot of the tree at that point, and parents--hence, recursively, by the entire history leading up to that commit. Naming objects by their content in that way has a lot of advantages--for example, you can sign an entire history by signing just the SHA1 of the commit at its tip. So you can't break that link between the names of commits and their contents without ending up with a fundamentally different (and probably weaker, in some sense), system. I suspect there's an unavoidable tradeoff--if you want to be able to reliably and efficiently determine how two branches are related, then you can't just throw away their (possibly messy) history. The best you may be able to do, if you want the advantages of both rebasing and merging, is to maintain on your own the messier meta-history as a superset of the simpler history that you end up submitting. --
From: Jeff Garzik <jeff@garzik.org> Ok, fair enough. Any alternative suggestions on how to remove turds I am well aware of this downside, sorry. --
Ahem... Use of git-cherry-pick preserves commit information just fine. --
Not by default, at least (note they said "commiters", not "authors"): bfields@pig:~/local/linux-2.6$ git cherry-pick v2.6.25-rc1 Finished one cherry-pick. Created commit c445dc0: Linux 2.6.25-rc1 1 files changed, 2 insertions(+), 2 deletions(-) bfields@pig:~/local/linux-2.6$ git show --pretty=raw HEAD commit c445dc0a8b11877d6038dc528254733348c9f110 tree 4018d5d93f857d946dd89acbb4e45c9da04eadaf parent b6ce068a1285a24185b01be8a49021827516b3e1 author Linus Torvalds <torvalds@woody.linux-foundation.org> 1202681894 -0800 committer J. Bruce Fields <bfields@citi.umich.edu> 1202861481 -0500 --b. --
What version of git have you tried this with? At least on my version, -r is a no-op, and the results are the same; the author is still kept and the maintainer changed. I thought it had been that way for ages. --b. bfields@pig:~/local/linux-2.6$ git --version git version 1.5.4.rc2.60.gb2e62 bfields@pig:~/local/linux-2.6$ git cherry-pick -r v2.6.25-rc1 Finished one cherry-pick. Created commit 0277143: Linux 2.6.25-rc1 1 files changed, 2 insertions(+), 2 deletions(-) bfields@pig:~/local/linux-2.6$ git cat-file -p HEAD tree 4018d5d93f857d946dd89acbb4e45c9da04eadaf parent b6ce068a1285a24185b01be8a49021827516b3e1 author Linus Torvalds <torvalds@woody.linux-foundation.org> 1202681894 -0800 committer J. Bruce Fields <bfields@citi.umich.edu> 1202863894 -0500 Linux 2.6.25-rc1 .. and I really need to call it something else. Maybe it is time to bring back the weasel series, since weasels always make me feel good about a kernel. --
Hmm. "-r" is a no-op to git-cherry-pick. And even if you thought it should preserve committer information, it really _really_ shouldn't. You're creating a new commit, you're the new committer. The old committer is meaningless. It doesn't matter at all if you try to keep the old committer information (which you can do by faking GIT_COMMITER_NAME¦EMAIL): you're simply just _lying_ at that point. The original committer has a different commit in his tree, and if you try to claim that your cherry-picked commit is his, you're only doing everybody a disservice. If you meant using "-x", then yes, that retains the actual pointer to the original commit, but it's not the default, because it shouldn't be used unless you plan to carry both around on purpose (ie it's mainly useful for "maintain a stable branch that has commits cherry-picked from mainline" kinds of things). Linus --
*duh* OK, I plead the lack of caffeine when reading the original posting. -r used to be "reproduce the changeset", but that _excludes_ committer. Nevermind. FWIW, I prefer to keep many branches and use suffix (.b<number>) to distinguish between them. And cherry-pick/reorder/split/collapse as needed on transition to the next one. At least that avoids some problems for secondary trees - branches do not jump. Since branches tend to be relatively small, they don't get conflicts that open and I can postpone switch to new branch until it really has to be done. I don't know how to deal with tricky branch topology; every time when I get to structure like <branch X is on top of Y+Z> it becomes very painful to work on all these topics in parallel. For trees maintained by different people... <shudder> --
Thank you. Over time we might think about some sort of standard for This is one of the things that linux-next will hopefully let us discover Right. Except that "drop the tree" will probably only mean for a day or so i.e. it will be taken out of the current round but will reappear Excellent! However, I am hoping that these global api changes may be introduced in a more orderly fashion (some of which is happening already) by creating new api's and then switching to them (and them maybe changing the names back if necessary). And, yes, I realise that this is sometimes not possible Thanks, I will probably need it :-( --=20 Cheers, Stephen Rothwell sfr@canb.auug.org.au
See my response to Arjan for how this is going to be very difficult to do, unless you are willing to have patches in -next that are not in any It's possible to do in a series of patches, yes, but again, development happens in parallel, with no one stopping for anyone else, and that's fine, we work it out when we send stuff to Linus at merge time. So with that parallel development effort, there are problems like this, I just want you to be aware of it and plan properly for it, as it is going to happen... thanks, greg k-h --
Understood. --=20 Cheers, Stephen Rothwell sfr@canb.auug.org.au http://www.canb.auug.org.au/~sfr/
Color me a bit skeptical... we've always had this capability really: for multi-function devices, just register as many subsystems as you need. If your device is a network _and_ SCSI device, you can do that with today's APIs. I just went through a long thread (e1000/e1000e) explaining how much pain multiple drivers for the same PCI ID cause on the distro side of things... its a mess. In each case you must write special case code to resolve the ambiguity. You never know which is the best driver to choose, or proper module load order -- which may vary depending on machine, not just static information captured from a kernel Makefile. Jeff --
Oh, I agree, multiple drivers for the same functionality is not a nice thing from a distro standpoint. The work I'm doing here is for stupid PCI firmware engineers, who have created devices that are different things, all bound up under the same PCI device. I'm thinking of watchdog timers and random number generator and i2c controller on the same PCI device, or even the more basic, frame buffer and DRM access to the same PCI video device. The OLPC is a good example of hardware that needs this kind of functionality. In order to take advantage of the "multiple drivers per device" all drivers need to guarantee to play nice with others, as they need to be aware that other drivers will be sharing the hardware. I'll write up more details later this week, if I get this stuff to work properly :) thanks, greg k-h --
Yes, that has a known solution: have your driver register i2c, rng, watchdog, etc. functions. Works just fine inside today's infrastructure, no changes needed. Jeff --
Except that the individual drivers are a lot of the time written by different people, live in different portions of the tree, and are combined into different combinations depending on the chipset. For i2c devices, I see the scx200_acb, i2c-elektor, i2c-sis5595, and i2c-sis630 drivers needing this. The last one happens to share the pci device with a video driver, that doesn't always need to be / want to be loaded by users just so they can read the temperature of their processors. Oh, the EDAC code also needs this, and I know that no one wants to merge that stuff into their individual drivers :) Same goes for framebuffer vs. DRM. Yeah, I know DRM ended up "winning" here, but for some hardware setups, both are still valid, and it really would be good for both drivers to be notified when the system is about to go into suspend/resume and other stuff like that. If the driver core can provide this in a simple manner, I think it is worth it, especially as there are lots of hardware that seems to need it. thanks, greg k-h --
Yes -- the worst case is that people have to work together, and it So people have to work together... darn :) most drivers these days are organized nicely into nicely modular units anyway, making it easy to write a "shell" driver that simply registers each sub-driver, and helps arbitrate/provide resources to sub-units. Consider, for example, a PCI driver that loads, and then fills in platform_data to provide a specific set of resources to a platform driver (a common idiom). You would need to create parented struct devices (parent: pci_dev's device), but everything else should work within I could even forsee a future where most drivers are written in a generic platform-driver style, and PCI|sbus|embedded-bus|blah are simply bus-specific shells that fill in platform data. All of this should be nicely possible within the existing usage of struct device, platform drivers, generic DMA API, iomap, etc. Jeff --
Indeed. If you have a multi-function device that shows up as a single PCI function, just make it have its own "private bus", and make it show up as a "devices within a device". Create the fake PCI subdevices that have no "real" counterpart, except as parts of the stupid device that couldn't be bothered to be seen as multiple _real_ functions. That not only solves the infrastructure issues, it's actually The Truth with capital letters. It is, after all, how the device actually works internally. Linus --
That's how I first tried to do this (old pci-piggy patches in -mm were a result of this...). But, we have the issue of the foolish-in-retrospect sysdev devices. They are an example of "multiple drivers per device" type thing. To clean them up properly, they need to move to the general 'struct device' and if that is going to already support the one-to-many model, well, then any PCI implementation of such an internal bus can also use the same code. This is a lot of handwaving right now, let me go work on some code and see how it turns out. Don't worry, I'll expect a lot of review cycles :) thanks, greg k-h --
> The work I'm doing here is for stupid PCI firmware engineers, who have > created devices that are different things, all bound up under the same > PCI device. I'm thinking of watchdog timers and random number > generator and i2c controller on the same PCI device, or even the more > basic, frame buffer and DRM access to the same PCI video device. > > The OLPC is a good example of hardware that needs this kind of > functionality. Sounds interesting. I've been meaning to work on this too for quite a while, but I'm glad to see you beat me to it. An example of an in-tree use case for this would be the mlx4 drivers-- you can look at drivers/net/mlx4/intf.c to see the simple stupid solution I came up with to allow an IB and a NIC (not yet upstream) driver to share the same PCI device. A good test for your stuff would be if it simplifies the code from the ad hoc solution I came up with. - R. --
Great, I'll look into that as I need a test case for this kind of stuff. thanks, greg k-h --
Our APIs are written such that the PCI device probe function is the starting point for registering IB and NIC. No need for anything new. Jeff --
No, you can't have a tree like that. [森林 Not yours. 森林] Let's maximize the profit, by keeping the largest number of trees which do not cause a conflict amongst each other. --
On Tue, 12 Feb 2008 12:02:08 +1100 Stephen Rothwell <sfr@canb.auug.org.au> = I neglected to mention the other brave souls who volunteered: Frank Seidel, Ann Davis, and Harvey Harrison who I hope to be able to call on in times of need. --=20 Cheers, Stephen Rothwell sfr@canb.auug.org.au http://www.canb.auug.org.au/~sfr/
Hey, I'm sure you just temporarily forgot the discussion about that point we had the days before ;-) Never mind, just kidding! :-) Lets get serious. I cannot speak for Ann and Harvey, but I'm quite sure they also really hope - at least i very strongly do - you not only call on us when things become a burden, but let us help and assist you right from the start. Thanks, Frank --
I just started a little naive webpage where i collected some of the main infos about linux-next on http://linux.f-seidel.de/linux-next/ while the wiki there is probably the best filled part. Does this make any sense to you to continue this? Or do you already have something else in mind or even already prepared? Thanks, Frank --
Agreed. I'm happy to do daily builds (I have x86 and ppc systems available), write scripts, etc. In the meantime I'll just lurk and learn. :-) Thx, Ann --
Jan already does builds of all -git (and previously -bk) and -mm kernels
for all architectures for some years, and it might be easier if he
cu
Adrian
--
"Is there not promise of rain?" Ling Tan asked suddenly out
of the darkness. There had been need of rain for many days.
"Only a promise," Lao Er said.
Pearl S. Buck - Dragon Seed
--
As noted to Andrew in private email, I think this would be great. akpm's -mm tree tends to have "x+1" changes in it, and also stuff that's not fully baked yet. I would definitely like to see a tree composed of all the sub-trees where maintainers said "yes, I think this is baked and ready for x+1". Then -mm and not-yet-baked git changes can layer on top of that. As others are noting, API changes require some coordination amongst ourselves (should be our job, not yours, really) and historically we haven't been so good at that -- in part because IMO there hasn't been any system thing better than a heads-up email in place. Jeff --
Hi Stephen, Please merge the quilt series/patchset from http://oss.oracle.com/~rdunlap/kernel-doc-patches/current/ I expect this to be fairly small and localized (very few conflicts). Thanks, --- ~Randy --
After glancing at some of this thread its clear to me what Stephen's real goal is: 1. collect kernel trees (or underpants) 2. ? 3. profit http://en.wikipedia.org/wiki/Underpants_Gnomes - k --
> The first things I need from the subsystem maintainers (you know who you
> are) are a contact address (a list address is fine) and at least one git
> branch or quilt series that contains all the things you want to see go
> into 2.6.26.
For InfiniBand/RDMA, the tree is:
master.kernel.org:/pub/scm/linux/kernel/git/roland/infiniband.git for-next
or via git protocol:
git://git.kernel.org/pub/scm/linux/kernel/git/roland/infiniband.git for-next
contact addresses (me plus a mailing list):
rolandd@cisco.com
general@lists.openfabrics.org
thanks!
--
Hi Roland, Added, thanks. --=20 Cheers, Stephen Rothwell sfr@canb.auug.org.au http://www.canb.auug.org.au/~sfr/
Do you have the tree and build logs available anywhere? I'd like to turn off the merge tree builds when this is able to replace it. James --
Hi James, On Sat, 16 Feb 2008 09:14:32 -0600 James Bottomley <James.Bottomley@HansenP= The tree is at git://git.kernel.org/pub/scm/linux/kernel/git/sfr/linux-next.git - or did you mean the logs of creating the tree. I currently ceate the tree fairly manually (as I slowly script what can be) and so have no logs of that process. The build logs that I have some control over are at http://kisskb.ellerman.id.au/kisskb/branch/9/. I am hoping to expand on the arch/config combinations over time (I have to convince my tame cross compiler builder :-)). I also hope that others will build this tree for themselves and publish the results. --=20 Cheers, Stephen Rothwell sfr@canb.auug.org.au
Um, well, I've already pointed to tools that currently do this bit Oh, OK ... by build I mean combine the trees and quilts into the actual tree. Compiling is nice, but it's the tree construction logs I want to see to make sure there aren't any impending merge problems. Most people verify on a fairly constant basis that their own trees actually build. The only problems we have are edge configurations (like arm and sparc, which have SCSI drivers that don't build except on those architectures). James --
This restriction means that the value for the ARM architecture is soo limited it's probably not worth the hastle participating in this project. We already know that -mm picks up on very few ARM conflicts because Andrew doesn't run through the entire set of configurations; unfortunately ARM is one of those architectures which is very diverse [*], and because of that, ideas like "allyconfig" are just completely irrelevant to it. As mentioned elsewhere, what we need for ARM is to extend the kautobuild infrastructure (see armlinux.simtec.co.uk) so that we can have more trees at least compile tested regularly - but that requires the folk there to have additional compute power (which isn't going to happen unless folk stamp up some machines _or_ funding). More trees maintained by more people isn't going to help the situation - if anything, it's going to make it worse - more trees needing more testing by the already extremely limited resources we have available. For reference, even _I_ don't build test the entire set of ARM defconfigs - at about 7 minutes a build, 75 defconfigs, that's about 9 hours... I just build those which are important to myself, hope that the others are fine, and rely on kautobuild finding any breakage. * - plus, its very difficult to get maintainers to see why having a kernel able to support multiple platforms is a good thing. For example, I would absolutely love to be able to combine more platforms into one build (such as Lubbock, Mainstone and probably other PXA stuff), but there's issues with drivers preventing it. (For those two I just mentioned, it's the SMC91x net driver whose build needs to be configured to the precise machine due to the multitude of different ways to connect the hardware to the processor.) -- Russell King Linux kernel 2.6 ARM Linux - http://www.arm.linux.org.uk/ maintainer of: --
On Sat, 16 Feb 2008 00:09:43 +0000 you need a better box ;) cerfcube_defconfig: 35 seconds carmeva_defconfig: 23 seconds spitz_defconfig (one of the biggest): 45 seconds so would a stupid `for i in arch/arm/configs/*' script be sufficient coverage? --
It will certainly improve the situation significantly, and pick up on some non-ARM problems like (badge4_defconfig, since 2.6.24-git2): drivers/built-in.o: In function `v4l2_i2c_attach': drivers/media/video/v4l2-common.c:1035: undefined reference to `i2c_attach_client' Currently, the defconfigs known to fail (long term) in Linus' tree are clps7500_defconfig and trizeps4_defconfig - the former I'm tempted to remove, the latter seems to be because the PCMCIA changes were lost on the linux-pcmcia list and the trizeps folk have probably given up. -- Russell King Linux kernel 2.6 ARM Linux - http://www.arm.linux.org.uk/ maintainer of: --
On Sat, 16 Feb 2008 00:31:36 +0000 I appear to be pcmcia maintainer lately so if someone wants to dust them off and send them over we can see what we can do? --
I do this wildcard together with yes '' | make ARCH=arm ... oldconfig make Sans toolchain issues, it's pretty good -- Russell larts you when some defconfig becomes broken anyway. :^) --
Only when I check the kautobuild website - which I've not been doing regularly since about end of November, and it only covers Linus' kernels. -- Russell King Linux kernel 2.6 ARM Linux - http://www.arm.linux.org.uk/ maintainer of: --
Hi Russell, On Sat, 16 Feb 2008 00:09:43 +0000 Russell King <rmk+lkml@arm.linux.org.uk>= I now have an arm cross compiler (gcc-4.0.2-glibc-2.3.6 arm-unknown-linux-gnu). (See the results page at http://kisskb.ellerman.id.au/kisskb/branch/9/ - I must get a better name/place :-(.) Is this sufficient to help you out? What configs would be useful to build (as Andrew said, they don't take very long each). I really want as many subsystems as possible in the linux-next tree in an attempt to avoid some of the merge/conflict problems we have had in the past. What can we do to help? --=20 Cheers, Stephen Rothwell sfr@canb.auug.org.au http://www.canb.auug.org.au/~sfr/
Hi Russell, On Tue, 26 Feb 2008 14:54:18 +1100 Stephen Rothwell <sfr@canb.auug.org.au> = OK, I tried this out: I built all 75 arm defconfigs with the cross compiler above (the resulting log file is at http://ozlabs.org/~sfr/armall.log.bz2). 17 failed (I don't know why - I didn't even really look). If this is useful, I can get this added to our infrastructure so that every linux-next kernel (and others as well, maybe) will be built for all these. Would that help, Russell? --=20 Cheers, Stephen Rothwell sfr@canb.auug.org.au P.S. From the timestamps in the log file you can see that this would take around an hour each time.
2 errors are expected (long-standing breakages).
14 failures are due to your ancient gcc 4.0.2 not supporting -mabi=aapcs-linux:
/scratch/sfr/next/scripts/mod/empty.c:1: error: invalid ABI option: -mabi=aapcs-linux
Another one is an internal error in your ancient compiler:
/scratch/sfr/next/arch/arm/common/it8152.c:148: internal compiler error: in trunc_int_for_mode, at explow.c:53
Can you build a cross compiler using gcc 4.2.3 from ftp.gnu.org instead?
cu
Adrian
--
"Is there not promise of rain?" Ling Tan asked suddenly out
of the darkness. There had been need of rain for many days.
"Only a promise," Lao Er said.
Pearl S. Buck - Dragon Seed
--
Hi Adrian, OK, thanks for the feedback it is very helpful. I will look at building a newer compiler on Monday. --=20 Cheers, Stephen Rothwell sfr@canb.auug.org.au
| Greg KH | Og dreams of kernels |
| Jens Axboe | [PATCH 31/33] Fusion: sg chaining support |
| Arnd Bergmann | Re: finding your own dead "CONFIG_" variables |
| Mark Brown | [PATCH 2/2] Subject: natsemi: Allow users to disable workaround for DspCfg reset |
| Tony Breeds |