Hi all,

Andrew was looking for someone to run a linux-next tree that just
contained the subsystem git and quilt trees for 2.6.x+1 and I (in a
moment of madness) volunteered.  So, this is to announce the creating of
such a tree (it doesn't exist yet) which will require some (hopefully)
small amount of work on the part of subsystem maintainers.

Those interested in discussion about this are encouraged to join the
linux-next@vger.kernel.org mailing list.

The first things I need from the subsystem maintainers (you know who you
are) are a contact address (a list address is fine) and at least one git
branch or quilt series that contains all the things you want to see go
into 2.6.26.  I am happy for there to be multiple branches/series (in
fact there probably will need to be in some cases where there are
dependencies on others work).

The tree will be based on the current version of Linus' tree but you may
specify an earlier branch point if you need to (hopefully not - but more
likely for quilters, I guess).

I hope to recreate this tree every day automatically.  In order to do
this, any tree that has a conflict will be dropped from that days tree.
The maintainer will be notified.  I hope to provide some clue as to what
the conflict is with, but probably not initially.

I will attempt to build the tree between each merge (and a failed build
will again cause the offending tree to be dropped).  These builds will be
necessarily restricted to probably one architecture/config.  I will build
the entire tree on as many architectures/configs as seem sensible and
the results of that will be available on a web page (to be announced).

This is just a start ...

Comments?

[I suspect that Andrew's dream includes actual time to dream :-)]
--=20
Cheers,
Stephen Rothwell                    sfr@canb.auug.org.au
http://www.canb.auug.org.au/~sfr/

For reference - Andrew's original musings:

I perceive the following problems with kernel development:

- The 80-odd subsystem trees are develop...
[ message continues ]

This restriction means that the value for the ARM architecture is soo
limited it's probably not worth the hastle participating in this project.

We already know that -mm picks up on very few ARM conflicts because
Andrew doesn't run through the entire set of configurations; unfortunately
ARM is one of those architectures which is very diverse [*], and because
of that, ideas like "allyconfig" are just completely irrelevant to it.

As mentioned elsewhere, what we need for ARM is to extend the kautobuild
infrastructure (see armlinux.simtec.co.uk) so that we can have more trees
at least compile tested regularly - but that requires the folk there to
have additional compute power (which isn't going to happen unless folk
stamp up some machines _or_ funding).

More trees maintained by more people isn't going to help the situation -
if anything, it's going to make it worse - more trees needing more testing
by the already extremely limited resources we have available.

For reference, even _I_ don't build test the entire set of ARM defconfigs -
at about 7 minutes a build, 75 defconfigs, that's about 9 hours...  I
just build those which are important to myself, hope that the others are
fine, and rely on kautobuild finding any breakage.



* - plus, its very difficult to get maintainers to see why having a
kernel able to support multiple platforms is a good thing.  For example,
I would absolutely love to be able to combine more platforms into one
build (such as Lubbock, Mainstone and probably other PXA stuff), but
there's issues with drivers preventing it.  (For those two I just
mentioned, it's the SMC91x net driver whose build needs to be configured
to the precise machine due to the multitude of different ways to connect
the hardware to the processor.)

-- 
Russell King
 Linux kernel    2.6 ARM Linux   - http://www.arm.linux.org.uk/
 maintainer of:
--

Hi Russell,

On Sat, 16 Feb 2008 00:09:43 +0000 Russell King <rmk+lkml@arm.linux.org.uk>=

I now have an arm cross compiler (gcc-4.0.2-glibc-2.3.6
arm-unknown-linux-gnu).  (See the results page at
http://kisskb.ellerman.id.au/kisskb/branch/9/ - I must get a better
name/place :-(.)  Is this sufficient to help you out?  What configs would
be useful to build (as Andrew said, they don't take very long each).

I really want as many subsystems as possible in the linux-next tree in an
attempt to avoid some of the merge/conflict problems we have had in the
past.  What can we do to help?

--=20
Cheers,
Stephen Rothwell                    sfr@canb.auug.org.au
http://www.canb.auug.org.au/~sfr/

Hi Russell,

On Tue, 26 Feb 2008 14:54:18 +1100 Stephen Rothwell <sfr@canb.auug.org.au> =

OK, I tried this out:  I built all 75 arm defconfigs with the cross
compiler above (the resulting log file is at
http://ozlabs.org/~sfr/armall.log.bz2). 17 failed (I don't know why - I
didn't even really look).  If this is useful, I can get this added to our
infrastructure so that every linux-next kernel (and others as well, maybe)
will be built for all these.

Would that help, Russell?

--=20
Cheers,
Stephen Rothwell                    sfr@canb.auug.org.au

P.S. From the timestamps in the log file you can see that this would take
around an hour each time.

2 errors are expected (long-standing breakages).

14 failures are due to your ancient gcc 4.0.2 not supporting -mabi=aapcs-linux:
/scratch/sfr/next/scripts/mod/empty.c:1: error: invalid ABI option: -mabi=aapcs-linux

Another one is an internal error in your ancient compiler:
/scratch/sfr/next/arch/arm/common/it8152.c:148: internal compiler error: in trunc_int_for_mode, at explow.c:53

Can you build a cross compiler using gcc 4.2.3 from ftp.gnu.org instead?

cu
Adrian

-- 

       "Is there not promise of rain?" Ling Tan asked suddenly out
        of the darkness. There had been need of rain for many days.
       "Only a promise," Lao Er said.
                                       Pearl S. Buck - Dragon Seed

--

Hi Adrian,


OK, thanks for the feedback it is very helpful.  I will look at building
a newer compiler on Monday.

--=20
Cheers,
Stephen Rothwell                    sfr@canb.auug.org.au

On Sat, 16 Feb 2008 00:09:43 +0000

you need a better box ;)

cerfcube_defconfig:				35 seconds
carmeva_defconfig:				23 seconds
spitz_defconfig (one of the biggest):		45 seconds

so would a stupid `for i in arch/arm/configs/*' script be sufficient
coverage?
--

I do this wildcard together with 

	yes '' | make ARCH=arm ... oldconfig
	make

Sans toolchain issues, it's pretty good -- Russell larts you when some
defconfig becomes broken anyway. :^)
--

Only when I check the kautobuild website - which I've not been doing
regularly since about end of November, and it only covers Linus'
kernels.

-- 
Russell King
 Linux kernel    2.6 ARM Linux   - http://www.arm.linux.org.uk/
 maintainer of:
--

It will certainly improve the situation significantly, and pick up
on some non-ARM problems like (badge4_defconfig, since 2.6.24-git2):

drivers/built-in.o: In function `v4l2_i2c_attach':
drivers/media/video/v4l2-common.c:1035: undefined reference to `i2c_attach_client'

Currently, the defconfigs known to fail (long term) in Linus' tree
are clps7500_defconfig and trizeps4_defconfig - the former I'm tempted
to remove, the latter seems to be because the PCMCIA changes were lost
on the linux-pcmcia list and the trizeps folk have probably given up.

-- 
Russell King
 Linux kernel    2.6 ARM Linux   - http://www.arm.linux.org.uk/
 maintainer of:
--

On Sat, 16 Feb 2008 00:31:36 +0000


I appear to be pcmcia maintainer lately so if someone wants to dust them
off and send them over we can see what we can do?
--

> The first things I need from the subsystem maintainers (you know who you
 > are) are a contact address (a list address is fine) and at least one git
 > branch or quilt series that contains all the things you want to see go
 > into 2.6.26.

For InfiniBand/RDMA, the tree is:

    master.kernel.org:/pub/scm/linux/kernel/git/roland/infiniband.git for-next

or via git protocol:

    git://git.kernel.org/pub/scm/linux/kernel/git/roland/infiniband.git for-next

contact addresses (me plus a mailing list):

    rolandd@cisco.com
    general@lists.openfabrics.org

thanks!
--

Hi Roland,


Added, thanks.

--=20
Cheers,
Stephen Rothwell                    sfr@canb.auug.org.au
http://www.canb.auug.org.au/~sfr/

Do you have the tree and build logs available anywhere?  I'd like to
turn off the merge tree builds when this is able to replace it.

James


--

Hi James,

On Sat, 16 Feb 2008 09:14:32 -0600 James Bottomley <James.Bottomley@HansenP=

The tree is at
git://git.kernel.org/pub/scm/linux/kernel/git/sfr/linux-next.git - or did
you mean the logs of creating the tree.  I currently ceate the tree
fairly manually (as I slowly script what can be) and so have no logs of
that process.

The build logs that I have some control over are at
http://kisskb.ellerman.id.au/kisskb/branch/9/.  I am hoping to expand on
the arch/config combinations over time (I have to convince my tame cross
compiler builder :-)).  I also hope that others will build this tree for
themselves and publish the results.

--=20
Cheers,
Stephen Rothwell                    sfr@canb.auug.org.au

Um, well, I've already pointed to tools that currently do this bit

Oh, OK ... by build I mean combine the trees and quilts into the actual
tree.  Compiling is nice, but it's the tree construction logs I want to
see to make sure there aren't any impending merge problems.  Most people
verify on a fairly constant basis that their own trees actually build.
The only problems we have are edge configurations (like arm and sparc,
which have SCSI drivers that don't build except on those architectures).

James


--

After glancing at some of this thread its clear to me what Stephen's  
real goal is:

1. collect kernel trees (or underpants)
2. ?
3. profit

http://en.wikipedia.org/wiki/Underpants_Gnomes

- k
--

Hi Stephen,

Please merge the quilt series/patchset from
  http://oss.oracle.com/~rdunlap/kernel-doc-patches/current/

I expect this to be fairly small and localized (very few conflicts).

Thanks,
---
~Randy
--

As noted to Andrew in private email, I think this would be great.

akpm's -mm tree tends to have "x+1" changes in it, and also stuff that's 
not fully baked yet.  I would definitely like to see a tree composed of 
all the sub-trees where maintainers said "yes, I think this is baked and 
ready for x+1".  Then -mm and not-yet-baked git changes can layer on top 
of that.

As others are noting, API changes require some coordination amongst 
ourselves (should be our job, not yours, really) and historically we 
haven't been so good at that -- in part because IMO there hasn't been 
any system thing better than a heads-up email in place.

	Jeff


--

On Tue, 12 Feb 2008 12:02:08 +1100 Stephen Rothwell <sfr@canb.auug.org.au> =

I neglected to mention the other brave souls who volunteered: Frank
Seidel, Ann Davis, and Harvey Harrison who I hope to be able to call on
in times of need.

--=20
Cheers,
Stephen Rothwell                    sfr@canb.auug.org.au
http://www.canb.auug.org.au/~sfr/

Agreed.  I'm happy to do daily builds (I have x86 and ppc systems 
available), write scripts, etc.  In the meantime I'll just lurk and 
learn.  :-)

Thx,

Ann
--

Jan already does builds of all -git (and previously -bk) and -mm kernels 
for all architectures for some years, and it might be easier if he 

cu
Adrian

-- 

       "Is there not promise of rain?" Ling Tan asked suddenly out
        of the darkness. There had been need of rain for many days.
       "Only a promise," Lao Er said.
                                       Pearl S. Buck - Dragon Seed

--

Hey, I'm sure you just temporarily forgot the discussion about that point
we had the days before ;-) Never mind, just kidding! :-)

Lets get serious. I cannot speak for Ann and Harvey, but I'm quite sure they
also really hope - at least i very strongly do - you not only call on us when
things become a burden, but let us help and assist you right from the start.

Thanks,
Frank

--

I just started a little naive webpage where i collected some of the main
infos about linux-next on

http://linux.f-seidel.de/linux-next/

while the wiki there is probably the best filled part. Does this
make any sense to you to continue this? Or do you already have something
else in mind or even already prepared?

Thanks,
Frank
--

Note that a lot of these are already in the MAINTAINERS file.

But for the record, here's mine, in the order they need to be pulled
from.
  Driver core:
	kernel.org/pub/linux/kernel/people/gregkh/gregkh-2.6/gregkh-01-driver/
  PCI:
	kernel.org/pub/linux/kernel/people/gregkh/gregkh-2.6/gregkh-02-pci/
  USB:
	kernel.org/pub/linux/kernel/people/gregkh/gregkh-2.6/gregkh-03-usb/

These are all quilt trees, with the series file in the directory for the
order of the patches, and a README saying what kernel version they have


This is going to get really interesting, especially when (not if) we do
more global api changes.  Look at the last round of kobject changes.
That touched a lot of different places, and other trees ended up not
building because of it, because I changed apis and they had added new
code based on the old apis.

I think the only way to fix this is not going to just "drop the tree"
like you are suggesting, but to let both people know (the person who
caused the change, and the person who's tree broke after the merge), and
then either add a "fixup patch" for the build like Andrew has been
doing, or disabling something from the build section.

As I know I'm going to be changing more driver core apis[1] this week,
I'm sure we will get a very good set of examples of this for you to see
in action :)

Good luck,

greg k-h

[1] Hopefully the "multiple drivers for a single device" feature people
have been asking for for years will be landing soon, of course the
number of odd places in the kernel that made the assumption that we
could only have one driver per device is causing lots of fun...
--

No, you can't have a tree like that. [森林 Not yours. 森林]


Let's maximize the profit, by keeping the largest number of trees which 
do not cause a conflict amongst each other.
--

Color me a bit skeptical...  we've always had this capability really: 
for multi-function devices, just register as many subsystems as you 
need.  If your device is a network _and_ SCSI device, you can do that 
with today's APIs.

I just went through a long thread (e1000/e1000e) explaining how much 
pain multiple drivers for the same PCI ID cause on the distro side of 
things...  its a mess.  In each case you must write special case code to 
resolve the ambiguity.  You never know which is the best driver to 
choose, or proper module load order -- which may vary depending on 
machine, not just static information captured from a kernel Makefile.

	Jeff


--

Oh, I agree, multiple drivers for the same functionality is not a nice
thing from a distro standpoint.

The work I'm doing here is for stupid PCI firmware engineers, who have
created devices that are different things, all bound up under the same
PCI device.  I'm thinking of watchdog timers and random number
generator and i2c controller on the same PCI device, or even the more
basic, frame buffer and DRM access to the same PCI video device.

The OLPC is a good example of hardware that needs this kind of
functionality.

In order to take advantage of the "multiple drivers per device" all
drivers need to guarantee to play nice with others, as they need to be
aware that other drivers will be sharing the hardware.

I'll write up more details later this week, if I get this stuff to work
properly :)

thanks,

greg k-h
--

> The work I'm doing here is for stupid PCI firmware engineers, who have
 > created devices that are different things, all bound up under the same
 > PCI device.  I'm thinking of watchdog timers and random number
 > generator and i2c controller on the same PCI device, or even the more
 > basic, frame buffer and DRM access to the same PCI video device.
 > 
 > The OLPC is a good example of hardware that needs this kind of
 > functionality.

Sounds interesting.  I've been meaning to work on this too for quite a
while, but I'm glad to see you beat me to it.

An example of an in-tree use case for this would be the mlx4 drivers--
you can look at drivers/net/mlx4/intf.c to see the simple stupid
solution I came up with to allow an IB and a NIC (not yet upstream)
driver to share the same PCI device.  A good test for your stuff would
be if it simplifies the code from the ad hoc solution I came up with.

 - R.
--

Our APIs are written such that the PCI device probe function is the 
starting point for registering IB and NIC.  No need for anything new.

	Jeff



--

Great, I'll look into that as I need a test case for this kind of stuff.

thanks,

greg k-h
--

Yes, that has a known solution:  have your driver register i2c, rng, 
watchdog, etc. functions.

Works just fine inside today's infrastructure, no changes needed.

	Jeff


--

Indeed. If you have a multi-function device that shows up as a single PCI 
function, just make it have its own "private bus", and make it show up as 
a "devices within a device".

Create the fake PCI subdevices that have no "real" counterpart, except as 
parts of the stupid device that couldn't be bothered to be seen as 
multiple _real_ functions.

That not only solves the infrastructure issues, it's actually The Truth 
with capital letters. It is, after all, how the device actually works 
internally.

			Linus
--

That's how I first tried to do this (old pci-piggy patches in -mm were a
result of this...).

But, we have the issue of the foolish-in-retrospect sysdev devices.
They are an example of "multiple drivers per device" type thing.  To
clean them up properly, they need to move to the general 'struct device'
and if that is going to already support the one-to-many model, well,
then any PCI implementation of such an internal bus can also use the
same code.

This is a lot of handwaving right now, let me go work on some code and
see how it turns out.

Don't worry, I'll expect a lot of review cycles :)

thanks,

greg k-h
--

Except that the individual drivers are a lot of the time written by
different people, live in different portions of the tree, and are
combined into different combinations depending on the chipset.

For i2c devices, I see the scx200_acb, i2c-elektor, i2c-sis5595, and
i2c-sis630 drivers needing this.  The last one happens to share the pci
device with a video driver, that doesn't always need to be / want to be
loaded by users just so they can read the temperature of their
processors.

Oh, the EDAC code also needs this, and I know that no one wants to merge
that stuff into their individual drivers :)

Same goes for framebuffer vs. DRM.  Yeah, I know DRM ended up "winning"
here, but for some hardware setups, both are still valid, and it really
would be good for both drivers to be notified when the system is about
to go into suspend/resume and other stuff like that.

If the driver core can provide this in a simple manner, I think it is
worth it, especially as there are lots of hardware that seems to need
it.

thanks,

greg k-h
--

Yes -- the worst case is that people have to work together, and it 


So people have to work together...  darn :)  most drivers these days are 
organized nicely into nicely modular units anyway, making it easy to 
write a "shell" driver that simply registers each sub-driver, and helps 
arbitrate/provide resources to sub-units.

Consider, for example, a PCI driver that loads, and then fills in 
platform_data to provide a specific set of resources to a platform 
driver (a common idiom).  You would need to create parented struct 
devices (parent: pci_dev's device), but everything else should work within

I could even forsee a future where most drivers are written in a generic 
platform-driver style, and PCI|sbus|embedded-bus|blah are simply 
bus-specific shells that fill in platform data.

All of this should be nicely possible within the existing usage of 
struct device, platform drivers, generic DMA API, iomap, etc.

	Jeff



--

Thank you.  Over time we might think about some sort of standard for


This is one of the things that linux-next will hopefully let us discover

Right. Except that "drop the tree" will probably only mean for a day or
so i.e. it will be taken out of the current round but will reappear

Excellent!

However, I am hoping that these global api changes may be introduced in a
more orderly fashion (some of which is happening already) by creating new
api's and then switching to them (and them maybe changing the names back
if necessary).  And, yes, I realise that this is sometimes not possible

Thanks, I will probably need it :-(

--=20
Cheers,
Stephen Rothwell                    sfr@canb.auug.org.au

See my response to Arjan for how this is going to be very difficult to
do, unless you are willing to have patches in -next that are not in any

It's possible to do in a series of patches, yes, but again, development
happens in parallel, with no one stopping for anyone else, and that's
fine, we work it out when we send stuff to Linus at merge time.

So with that parallel development effort, there are problems like this,
I just want you to be aware of it and plan properly for it, as it is
going to happen...

thanks,

greg k-h
--

Understood.

--=20
Cheers,
Stephen Rothwell                    sfr@canb.auug.org.au
http://www.canb.auug.org.au/~sfr/

On Mon, 11 Feb 2008 20:21:33 -0800

in my experience, the only chance you have is doing API changes as first in the set of changes,
and then hoping (making) all other trees use the new APIs. Any other order just turns into
an impossible mismash.

I can see the point of doing an LKML annouce of the new API after the series is done, so that 
all other maintainers have a chance to fix their trees (this of course is only for new occurances
of the old api showing up; the api change series should convert all existing users)

-- 
If you want to reach me at my work email, use arjan@linux.intel.com
For development, discussion and tips for power savings, 
visit http://www.lesswatts.org
--

I agree, and that's what I do.

The problem is, the API change is still in my tree.  So, if for example,
the IB tree goes and adds some new functionality before my API changes
have landed, they need to use the "old" API in order for them to be able
to test and build things on their own.  Then, when the -next tree merges
everything together, the IB tree breaks the build, not my driver tree.

It's those "who goes first" type things that end up being the cause of a
lot of Andrew's headaches I think :)

thanks,

greg k-h
--

On Mon, 11 Feb 2008 20:43:14 -0800

this is why you need specific trees for just the API change, and these
need to EXPLICITLY go first before EVERYTHING ELSE. Yes this needs a bit of coordination,
but it's the only way.
--

Even then, it will not work.

Again, Roland isn't going to want to always pull in my driver tree just
to build his tree.  He wants to, and needs to do his own development
effort.

But when we merge them together, there would be problems.

So, you can't just "drop" the IB tree.
You can't just "drip" my tree.

Where do you "fix this up" at?  I can send a patch for the IB tree, but
Roland can't put it in his tree, and I can't put it in my tree, it needs
to go _after_ both of our trees.

That's what -mm has been able to handle so far, and that needs to also
work with -next.

thanks,

greg k-h
--

From: Greg KH <greg@kroah.com>

Totally agreed.

The fact is there are interdependencies, especially in driver
land and you have to either:

1) Make the driver folks work on top of Greg's tree.

2) Constantly rebase the -next tree to deal with the
   conflicts.

There are some other issues related to this which haven't
be touched upon greatly yet.

I rebase my tree all the time, at least once or twice per
week.  Why?

Firstly, to remove crap.  When you have "great idea A" then "oh shit A
won't work, revert that" there is zero sense in keeping both
changesets around.

Secondly, I want to fix up the rejects caused by conflicts with
upstream bug fixes and the like (and there are tons when the tree gets
to 1500 or so patches like the networking did).  I don't want git to
merge the thing by hand, I want to see what the conflict is and make
sure the "obvious" resolution is OK and the most efficient way I know
how to do that is to suck my tree apart as patches, then suck them
back into a fresh tree.

It therefore might make sense to the linux-next tree to do something
similar, constantly rebasing so that all the conflicts and reverted
shit changes can be sorted out without having an incredibly ugly
GIT history.
--

FWIW, that is annoying and painful for us downstream jobbers, since it 
isn't really how git was meant to be used.  You use it more like a patch 
queue, where commits are very fluid.

Unfortunately, if there is any synchronization lag between me and you -- 
not uncommon -- then I cannot commit changes on top of the changes just 
sent, in my own local tree.  Why?  Because you rebase so often, I cannot 
even locally commit dependent patches due to the end result merge 
getting so nasty.

I understand the desire to want a nice and clean history, but the 
frequency here really has a negative impact on your downstreams.

It also totally screws the commit statistics, wiping me and John and the 
committers we have preserved out, replacing everybody's committer with 
David Miller.

	Jeff


--

From: Jeff Garzik <jeff@garzik.org>

Ok, fair enough.  Any alternative suggestions on how to remove turds

I am well aware of this downside, sorry.
--

Ahem...  Use of git-cherry-pick preserves commit information just fine.
--

Not by default, at least (note they said "commiters", not "authors"):

bfields@pig:~/local/linux-2.6$ git cherry-pick v2.6.25-rc1
Finished one cherry-pick.
Created commit c445dc0: Linux 2.6.25-rc1
 1 files changed, 2 insertions(+), 2 deletions(-)
bfields@pig:~/local/linux-2.6$ git show --pretty=raw HEAD
commit c445dc0a8b11877d6038dc528254733348c9f110
tree 4018d5d93f857d946dd89acbb4e45c9da04eadaf
parent b6ce068a1285a24185b01be8a49021827516b3e1
author Linus Torvalds <torvalds@woody.linux-foundation.org> 1202681894 -0800
committer J. Bruce Fields <bfields@citi.umich.edu> 1202861481 -0500


--b.
--

That's why you give it -r.
--

Hmm. "-r" is a no-op to git-cherry-pick.

And even if you thought it should preserve committer information, it 
really _really_ shouldn't. You're creating a new commit, you're the new 
committer. The old committer is meaningless. It doesn't matter at all if 
you try to keep the old committer information (which you can do by faking 
GIT_COMMITER_NAME¦EMAIL): you're simply just _lying_ at that point. The 
original committer has a different commit in his tree, and if you try to 
claim that your cherry-picked commit is his, you're only doing everybody a 
disservice.

If you meant using "-x", then yes, that retains the actual pointer to the 
original commit, but it's not the default, because it shouldn't be used 
unless you plan to carry both around on purpose (ie it's mainly useful for 
"maintain a stable branch that has commits cherry-picked from mainline" 
kinds of things).

			Linus
--

*duh*

OK, I plead the lack of caffeine when reading the original posting.
-r used to be "reproduce the changeset", but that _excludes_ committer.
Nevermind.

FWIW, I prefer to keep many branches and use suffix (.b<number>) to
distinguish between them.  And cherry-pick/reorder/split/collapse
as needed on transition to the next one.  At least that avoids
some problems for secondary trees - branches do not jump.

Since branches tend to be relatively small, they don't get conflicts
that open and I can postpone switch to new branch until it really
has to be done.

I don't know how to deal with tricky branch topology; every time when I
get to structure like <branch X is on top of Y+Z> it becomes very painful
to work on all these topics in parallel.  For trees maintained by different
people... <shudder>
--

What version of git have you tried this with?  At least on my version,
-r is a no-op, and the results are the same; the author is still kept
and the maintainer changed.  I thought it had been that way for ages.

--b.

bfields@pig:~/local/linux-2.6$ git --version
git version 1.5.4.rc2.60.gb2e62
bfields@pig:~/local/linux-2.6$ git cherry-pick -r v2.6.25-rc1
Finished one cherry-pick.
Created commit 0277143: Linux 2.6.25-rc1
 1 files changed, 2 insertions(+), 2 deletions(-)
bfields@pig:~/local/linux-2.6$ git cat-file -p HEAD
tree 4018d5d93f857d946dd89acbb4e45c9da04eadaf
parent b6ce068a1285a24185b01be8a49021827516b3e1
author Linus Torvalds <torvalds@woody.linux-foundation.org> 1202681894 -0800
committer J. Bruce Fields <bfields@citi.umich.edu> 1202863894 -0500

Linux 2.6.25-rc1

.. and I really need to call it something else.  Maybe it is time to
bring back the weasel series, since weasels always make me feel good
about a kernel.

--

But the "author" is still preserved, right?  Why do you need the
committer name to be preserved?  (I'm not denying that there could be
reasons, I'm just curious what they are.)

--b.
--

It's not that the committer should be preserved, but:

 - the chain from author -> committer should be visible in the 
   Signed-off-by: lines.

   If you rebase somebody elses tree, you screw that up. You need to add 
   your sign-off, since now *you* are the new committer, and *you* took
   somebody elses work!

 - you should respect the down-stream developer, and if that downstream 
   developer continues to work with his branch or works with other people, 
   you shouldn't screw that up!

Both of those basically say that you should never rebase somebody elses 
work. You can use rebase to rebase your *own* work on top of somebody 
elses thing (since that doesn't change the sign-off chain, and you still 
respect the downstream developers development model)!

But of course, if you rebase, you should respect the wishes of the 
up-stream developer too. I don't do rebases. So if you asked me to pull, 
the stuff I pulled can never be rebased, because it just *is* in my tree. 

Put another way: think of the absolute *chaos* that would happen if I were 
to rebase instead of just merging. Every time I pull from you I'd 
invalidate your whole tree, and you'd have to re-generate. It gets 
unmaintainable very quickly.

And that's actually ignoring a real issue: stability of commits. The nice 
thing about stable commit naming is that all bug-reports from other people 
that told where the bug happened are basically 100% trust-worthy and the 
code is 100% reproducible not just for you, but for everybody else. 

In other words, you really shouldn't rebase stuff that has been exposed 
anywhere outside of your own private tree. But *within* your own private 
tree, and within the commits that have never seen the light of day, 
rebasing is fine.

(And yes, there are exceptions. If it's a clear "throw-away tree" all the 
rules go out the window, of course, as long as everybody involved *knows* 
it's a throw-away tree, and know that if they pull it they have to 
synchronise 100% with...
[ message continues ]

From: Linus Torvalds <torvalds@linux-foundation.org>

I agree with this and that is exactly what I screwed up by mistake
this time around.

Normally when I rebase I walk through the patches that came from other
people's trees and add signoffs as needed.  I understand that this

I actually wouldn't mind that, the first thing I do when sending a
pull request is I stop putting things into my tree and as soon as the
recipient pulls I wipe out my tree and clone a fresh copy of their's.

It's really not a big deal.  The pusher can queue patches and other
stuff up in their mailbox or in a directory somewhere.  This quiet
period also allows those patches to have some time to be reviewed on
the lists before they actually end up in anyone's tree.

I really like that mode of operation.

--

*YOU* like it, because it never generates any issues for *you*. You're the 
top in your heap, and the people above you don't do that insane thing, so 
you get all of the advantages, with none of the downsides. Of *course* you 
like it.

But as people have pointed out, it generates issues for the people under 
you! If I did it, the people who now complain about networking would 
not just be a couple, it would be everybody. Nobody could depend on 
anything out there, because everything would have to rebase.

You just don't see the problems, because the only person above you isn't 
crazy enough to do what you propose. You also don't do ten merges a day of 
subsystems you don't know.

The importance of merging (rather, not screwing up history in general) 
becomes really obvious when things go tits-up. Then they go tits-up 
*without* screwing up the history of the trees that were hopefully tested 
individually.

If you re-base things that others developed, you lose that. Imagine if I 
merged first Greg's tree (by rebasing), and then there was some 
fundamental thing that didn't cause a conflict, but just made something 
not work, when I rebased yours on top. Think about what happens.

Now I've merged (say) 1500 networking-related commits by rebasing, but 
because I rebased on top of Greg's tree that I had also rebased, 
absolutely *none* of that has been tested in any shape of form. I'd not 
use most of the things I pulled, so I'd never see it, I'd just push out 
something that was very different from *both* trees I pulled, with no way 
to really blame the merge - because it doesn't even exist.

So as a result, some *random* commit that was actually fine on its own has 
now become a bug, just because it was re-written. 

You don't see the problem here?

Yes, this is the *crap* you do all the time. You don't see the problems as 
much, because you merge probably only about a tenth of the volume I merge, 
and you can keep track of the subsystem more. But even though you don't...
[ message continues ]

If there was a "fundamental thing that didn't cause a conflict", then
the two trees in question probably didn't touch the same code, so would
probably merge cleanly, for the same reason that one rebased onto the
other cleanly.  But depending on what the "fundamental thing" was, the
merge might still introduce the same bug, right?

--b.
--

Absolutely. But if you do a true merge, the bug is clearly in the merge 
(automatedly clean or not), and the blame is there too. IOW, you can blame 
me for screwing up. Now, I will say "oh, me bad, I didn't realize how 
subtle the interaction was", so it's not like I'll be all that contrite, 
but at least it's obvious where the blame lies.

In contrast, when you rebase, the same problem happens, but now a totally 
innocent commit is blamed just because it happened to no longer work in 
the location it was not tested in. The person who wrote that commit, the 
people who tested it and said it works, all that work is now basically 
worthless: the testing was done with another version, the original patch 
is bad, and the history and _reason_ for it being bad has been lost.

And there's literally nothing left to indicate the fact that the patch and 
the testing _used_ to be perfectly valid.

That may not sound like such a big deal, but what does that make of code 
review and tested-by, and the like? It just makes a mockery of trying to 
do a good job testing any sub-trees, when you know that eventually it will 
all quite possibly be pointless, and the fact that maybe the networking 
tree was tested exhaustively is all totally moot, because in the end the 
stuff that hit the main tree is something else altogether?

I don't know about you, but I'd personally be really disappointed if it 
happened to me, and I felt that I did a really good job as a 
submaintainer. I'd also feel that the source control management sucked.

Contrast that to the case where somebody simply does a merge error. The 
original work doesn't lose it's validity - so the original maintainer 
hasn't lost anything. And quite frankly, even the person who "screwed up" 
with the merge hasn't really done anything bad: these things _do_ happen. 

So bugs happen; not big deal. But the fact that the bugs are correctly 
attributed - or rather, not mis-attributed to somebody blameless - that 
_is_ a big deal.

It's not ...
[ message continues ]

The obvious advantage to rebasing in this case is that the blame
(misplaced though it may be), at least lands on a commit that made a
single small change, likely making the problem easier to diagnose.

(As opposed to the case of a large merge, where all you may know is that
somewhere in the hundreds of commits done on one side of the merge there
was a conflict with the hundreds of commits on the other side.)

I think a lot of people would see rebasing as an acceptable tradeof that
gives up a small amount of accuracy in assigning blame to individuals in
return for a large increase in ability to debug problems.

I suppose one response to that would be that it's important that people
learn how to work in parallel, that failures to do so are particularly
important failures in the process, and that it's therefore worth it to
make sure that such failures are always identified specifically as merge
failures.

It would be nice if merges, like patches, were broken up into somewhat
smaller units.  There's an understandable desire to wait to the last
minute to actually commit to one's commits, but a willingness to do so a
little earlier might avoid some of the problems that seem to come from
having a lot of large merges happen all at once.

--b.
--

One idea that I thought about when debating rebase vs. merge (and it's
far far from being fully baked) is versioned commits.  The gist of it
is that patches are assigned an hash identifier like today when they are
first committed into the tree, but, and this is the main change: if they mutate,
e.g. by a rebase, or even git commit --amend, their version is bumped up rather
than SHA changed.

This way all versions of the commit would be accessible and addressable using
their respective SHA *and* version. I think that this can help keep
the tree's history in a more intuitive way (since the patches' base identifier
won't change, just its version number), and you get a bonus of seeing each
commit's history, who changed it, and what was the change.

Benny

--

The SHA1 is uniquely determined by the contents of that commit--commit
and author names and times, changelog message, snapshot of the tree at
that point, and parents--hence, recursively, by the entire history
leading up to that commit.

Naming objects by their content in that way has a lot of advantages--for
example, you can sign an entire history by signing just the SHA1 of the
commit at its tip.  So you can't break that link between the names of
commits and their contents without ending up with a fundamentally
different (and probably weaker, in some sense), system.

I suspect there's an unavoidable tradeoff--if you want to be able to
reliably and efficiently determine how two branches are related, then
you can't just throw away their (possibly messy) history.  The best you
may be able to do, if you want the advantages of both rebasing and
merging, is to maintain on your own the messier meta-history as a
superset of the simpler history that you end up submitting.

--

From: Linus Torvalds <torvalds@linux-foundation.org>

Good point.

Now how do I remove a bogus commit for a tree that I've already pushed
out and published for other people, without any record of it appearing
in the GIT tree any more?

How do I insert build fixes into existing changesets so that the tree
is more bisectable?

If Jeff merged in a tree that introduced a ton of whitespace errors
git is complaing about, is there a way I can fixup that changeset
in-place? (or should I tell Jeff to start adhering to GIT's whitespace
warning messages when he applies patches?)

Those are basically the major operations that would allow me to
seriously consider rebasing a ton less often.
--

So, the answer is: if others have actually pulled, it's simply not 
possible.

There simply is no way to undo history that isn't local any more. You just 
*have* to revert, or you need to find every single person that pulled 
(directly or indirectly) and ask them to undo all the work they did on top 
of your bad commit.

This is not git-related, btw. That's just how the world works. It's a bit 
like the internet - if you said something stupid on #IRC and it made it to 
bash.org, there's not a whole lot you can do to "undo" your stupidity. You 

Just delay pushing out. There really is _zero_ downside to this. None at 

Umm. Git doesn't complain about whitespace _except_ when applying patches. 
So if you don't rebase his work, you'll never see the whitespace warnings 
either!

Of course, we'd probably wish that Jeff cared about the whitespace 
warnings and pushed back on them, but the fact is, that warning isn't 
meant for you - because by the time you pull Jeff's tree, it's simply not 
your issue any more. Jeff already applied it. Either he's your trusted 
lietenant or he's not.

Quite frankly, to me it sounds like you're not ready to "let go" and trust 
the people under you. Trust me, it's worth it. It's why your life is easy: 
I have let go and I trust you. 

Also, I'd *much* rather have a few problems in the tree than have people 
screw up history in order to hide them. Sure, we want to keep things 
bisectable, but quite frankly, if you do a reasonable job and compile the 
kernel before you push out, it will be "mostly bisectable".

And yes, mistakes happen. Mistakes will *always* happen. It's ok. Relax. 

Let me put it another way: You're _both_ going to be *much* better off 
pushing back on Jeff, telling him that "I can't pull from you because your 
tree is ugly and doesn't compile", than taking his tree and rebasing it.

Remember? I used to do that all the time. I berated the ACPI people for 
creating monster trees that were horrible and contained fifteen merges ...
[ message continues ]

One of the things which linux-next could/should do is to help weed out the
silly build breaks, typos, missing documentation updates, missed checkpatch
opportunities, etc, etc. As well as real bugs.

So it would not be effici