On Thu, 12 Apr 2007, Sam Vilain wrote:
quoted text
> >
> > And yes, that's absolutely true, but it's technically no different from 
> > just using GIT_OBJECT_DIRECTORY to share objects between totally unrelated 
> > projects, or using git/alternates to share objects between (probably 
> > *less* unrelated repositories, but still clearly individual repos).
> 
> Would that be the only distinction?
> 
> Would submodules be descended into for object reachability questions?

I think we'll eventually want that *regardless* of how the object handling 
is done (a kind of "cross-submodule boundary check"), but I think that's 
actually outside of the scope of the current fsck.

The current fsck goes to great lengths to make sure that the internal 
consistency of a repository is good. That's also why it takes so long, and 
why it is such an expensive operation to do (notably when you do a 
"--full" check).

In contrast, the "cross-submodule boundary check" is a much cheaper 
operation, *if* you have already verified that the projects are internally 
consistent. It literally boils down to doing a very simplified commit 
chain walker that only parses tree objects and simply spits out the 
SHA1's of the sub-tree commits (and their location in the tree), and then 
a separate phase that just verifies those against the submodules.

And that separate phase - once you've done the fsck for all the 
*individual* repositories - is truly trivial. It's literally just a matter 
of "is that SHA1 a valid commit object". That's *cheap*.

See?

quoted text
> I'm particularly interested in repositories with, say, thousands of
> submodules but only a few hundred meg. I really want to avoid the
> situation where each of those submodules gets checked or descended into
> separately for updates etc.

So I think that the way to verify a superproject is:

 - fsck each and every project totally independently. This is something 
   you have to do *anyway*.

 - either as you fsck, or as a separate phase after the fsck, just 
   traverse the trees and spit out "these are the SHA1's of subprojects"

 - finally, just go through the list of SHA1's (after every project has 
   been fsck'd) and verify that they exist (since if they exist, they will 
   have everything that is reachable from them, as that's one of the 
   things that the *local* fsck verifies)

Notice? At no point do you actually need to do a "global fsck". You can do 
totally independent local fsck's, and then a really cheap test of 
connectedness once those fsck's have completed.

The reason a *full* global fsck is so expensive is that it would have an 
absolutely humungous working set, and effectively keep everything in 
memory through it all. Doing it in stages ("fsck smaller individiual trees 
separately") is actually the same amount of absolute work, but the working 
set never grows, so it scales much better.

(fsck'ing projects individually also happens to allow you to do the 
sub-project fsck's in parallel across multiple CPU's or multiple machines, 
so it actually scales much better that way too - but the big problem 
tends to be excessive memory use, so the "SMP parallel version" only 
makes sense if you have tons of memory and can afford to do these things 
at the same time!)

			Linus
-
To unsubscribe from this list: send the line "unsubscribe git" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html