Re: [RFC] Submodules in GIT

Previous message: [thread] [date] [author]
Next message: [thread] [date] [author]
From: Linus Torvalds
Date: Friday, December 1, 2006 - 5:12 pm

On Sat, 2 Dec 2006, Josef Weidendorfer wrote:

The only _true_ namespace would be the SHA1 of the commit (and maybe allow 
a pointer to a tag too, but the namespace ends up being the same).

How to _find_ a repository that contains that SHA1 must be left to higher 
levels. After all, repositories move around, and the place you found them 
originally is not a stable name.

So within the supermodule, on a "git object" level, a submodule should 
just be named by the SHA1 that was it's HEAD when it was committed within 
the supermodule. So in the "tree object", you'd see something like the 
following when you go "git ls-tree HEAD" on the superproject:

	...
	100644 blob 08602f522183dc43787616f37cba9b8af4e3dade	xdiff-interface.c
	100644 blob 1346908bea31319aabeabdfd955e2ea9aab37456	xdiff-interface.h
	040000 tree 959dd5d97e665998eb26c764d3a889ae7903d9c2	xdiff
	050000 link 0215ffb08ce99e2bb59eca114a99499a4d06e704	xyzzy

where that 050000 is the new magic type (I picked one out of my *ss: it's 
not a valid type for a file mode, so it's a godo choice, but it could be 
anythign that cannot conflict with a real file), which just specifies the 
"link" part. The SHA1 is the SHA1 of the commit, and the "xyzzy" is 
obviously just the name within the directory of the submodule.

That's all that is actually required for a lot of git commands that 
already expect all objects to be available (ie "git checkout", "git diff" 
etc).

It only gets interesting for commands that fetch new objects, ie do a 
"pull/fetch" op, and you'd need to know where/how to fetch new objects for 
the xyzzy subproject, so that's a "naming" issue. You have a few choices:

 - get all the objects directly from the subproject as if it was one big 
   project.

   I actually think this sucks. Why? Because it puts an insane load on the 
   server side, which basically needs to traverse the object list of the 
   _sum_ of all projects. An initial clone (or a really big pull, which 
   comes to the same thing) would be absolutely horrendous

So I'd strongly argue against that approach, for scalability reasons. So 
instead, you should really try to do pulls etc one git repo at a time:

 - take the "list of subprojects" from the supermodule, and pull them all 
   one by one.

   This again makes subprojects "less seamless", and makes each subproject 
   more of a separate thing, with the project list gotten from the 
   superproject and parsed separately. But it means you have none of the 
   scalability problems, since you never see things as one huge project 
   with millions of files and even more objects.

The second approach also means that you can see the "supermodule" support 
in git as less of a "plumbing" thing, and it's largely just a thin veneer 
around the core plumbing that really doesn't understand about multiple 
repositories at all (apart from the single "link" extension in the tree 
object), and it's really just scripting to get the subprojects to "look" 
like one thing, when they really are pretty much independent.

		Linus
-
To unsubscribe from this list: send the line "unsubscribe git" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html
Previous message: [thread] [date] [author]
Next message: [thread] [date] [author]

Messages in current thread:
Re: [RFC] Submodules in GIT, Jakub Narebski, (Tue Nov 28, 3:50 am)
Re: [RFC] Submodules in GIT, Andy Parkins, (Tue Nov 28, 6:35 am)
Re: [RFC] Submodules in GIT, Shawn Pearce, (Tue Nov 28, 8:44 am)
Re: [RFC] Submodules in GIT, Andy Parkins, (Tue Nov 28, 9:29 am)
Re: [RFC] Submodules in GIT, Shawn Pearce, (Tue Nov 28, 9:36 am)
Re: [RFC] Submodules in GIT, Jon Loeliger, (Tue Nov 28, 10:38 am)
Re: [RFC] Submodules in GIT, Steven Grimm, (Tue Nov 28, 12:58 pm)
Re: [RFC] Submodules in GIT, Shawn Pearce, (Tue Nov 28, 2:02 pm)
Re: [RFC] Submodules in GIT, Martin Waitz, (Wed Nov 29, 9:03 am)
Re: [RFC] Submodules in GIT, Martin Waitz, (Wed Nov 29, 9:15 am)
Re: [RFC] Submodules in GIT, Andy Parkins, (Wed Nov 29, 1:00 pm)
Re: [RFC] Submodules in GIT, sf, (Thu Nov 30, 4:57 am)
Re: [RFC] Submodules in GIT, Andreas Ericsson, (Thu Nov 30, 5:16 am)
Re: [RFC] Submodules in GIT, Andy Parkins, (Thu Nov 30, 5:40 am)
Re: [RFC] Submodules in GIT, Martin Waitz, (Thu Nov 30, 10:06 am)
Re: [RFC] Submodules in GIT, Andreas Ericsson, (Thu Nov 30, 11:57 am)
Re: [RFC] Submodules in GIT, Andy Parkins, (Fri Dec 1, 1:49 am)
Re: [RFC] Submodules in GIT, Andy Parkins, (Fri Dec 1, 2:02 am)
Re: [RFC] Submodules in GIT, Andreas Ericsson, (Fri Dec 1, 2:33 am)
Re: [RFC] Submodules in GIT, Andy Parkins, (Fri Dec 1, 3:38 am)
Re: [RFC] Submodules in GIT, Martin Waitz, (Fri Dec 1, 4:00 am)
Re: [RFC] Submodules in GIT, sf, (Fri Dec 1, 5:03 am)
Re: [RFC] Submodules in GIT, sf, (Fri Dec 1, 5:09 am)
Re: [RFC] Submodules in GIT, Martin Waitz, (Fri Dec 1, 5:11 am)
Re: [RFC] Submodules in GIT, Martin Waitz, (Fri Dec 1, 5:12 am)
Re: [RFC] Submodules in GIT, sf, (Fri Dec 1, 6:05 am)
Re: [RFC] Submodules in GIT, sf, (Fri Dec 1, 6:21 am)
Re: [RFC] Submodules in GIT, Martin Waitz, (Fri Dec 1, 6:35 am)
Re: [RFC] Submodules in GIT, Martin Waitz, (Fri Dec 1, 6:43 am)
Re: [RFC] Submodules in GIT, Andreas Ericsson, (Fri Dec 1, 6:43 am)
Re: [RFC] Submodules in GIT, Martin Waitz, (Fri Dec 1, 6:46 am)
Re: [RFC] Submodules in GIT, Stephan Feder, (Fri Dec 1, 6:51 am)
Re: [RFC] Submodules in GIT, Stephan Feder, (Fri Dec 1, 7:23 am)
Re: [RFC] Submodules in GIT, Andreas Ericsson, (Fri Dec 1, 7:52 am)
Re: [RFC] Submodules in GIT, Martin Waitz, (Fri Dec 1, 7:58 am)
Re: [RFC] Submodules in GIT, Martin Waitz, (Fri Dec 1, 8:00 am)
Re: [RFC] Submodules in GIT, Martin Waitz, (Fri Dec 1, 8:07 am)
Re: [RFC] Submodules in GIT, Stephan Feder, (Fri Dec 1, 8:47 am)
Re: [RFC] Submodules in GIT, Stephan Feder, (Fri Dec 1, 9:04 am)
Re: [RFC] Submodules in GIT, Martin Waitz, (Fri Dec 1, 9:15 am)
Re: [RFC] Submodules in GIT, Andreas Ericsson, (Fri Dec 1, 9:38 am)
Re: [RFC] Submodules in GIT, Linus Torvalds, (Fri Dec 1, 9:49 am)
Re: [RFC] Submodules in GIT, Martin Waitz, (Fri Dec 1, 9:54 am)
Re: [RFC] Submodules in GIT, Martin Waitz, (Fri Dec 1, 9:57 am)
Re: [RFC] Submodules in GIT, sf, (Fri Dec 1, 10:08 am)
Re: [RFC] Submodules in GIT, Martin Waitz, (Fri Dec 1, 10:14 am)
Re: [RFC] Submodules in GIT, Stephan Feder, (Fri Dec 1, 10:33 am)
Re: [RFC] Submodules in GIT, Andreas Ericsson, (Fri Dec 1, 11:06 am)
Re: [RFC] Submodules in GIT, Andreas Ericsson, (Fri Dec 1, 11:08 am)
Re: [RFC] Submodules in GIT, Martin Waitz, (Fri Dec 1, 11:48 am)
Re: [RFC] Submodules in GIT, Martin Waitz, (Fri Dec 1, 11:51 am)
Re: [RFC] Submodules in GIT, Andy Parkins, (Fri Dec 1, 12:17 pm)
Re: [RFC] Submodules in GIT, Martin Waitz, (Fri Dec 1, 12:38 pm)
Re: [RFC] Submodules in GIT, Linus Torvalds, (Fri Dec 1, 1:13 pm)
Re: [RFC] Submodules in GIT, Martin Waitz, (Fri Dec 1, 1:30 pm)
Re: [RFC] Submodules in GIT, Andy Parkins, (Fri Dec 1, 2:04 pm)
Re: [RFC] Submodules in GIT, Martin Waitz, (Fri Dec 1, 2:37 pm)
Re: [RFC] Submodules in GIT, Andy Parkins, (Fri Dec 1, 2:54 pm)
Re: [RFC] Submodules in GIT, Josef Weidendorfer, (Fri Dec 1, 3:06 pm)
Re: [RFC] Submodules in GIT, Martin Waitz, (Fri Dec 1, 3:08 pm)
Re: [RFC] Submodules in GIT, Martin Waitz, (Fri Dec 1, 3:12 pm)
Re: [RFC] Submodules in GIT, Josef Weidendorfer, (Fri Dec 1, 3:26 pm)
Re: [RFC] Submodules in GIT, Linus Torvalds, (Fri Dec 1, 3:26 pm)
Re: [RFC] Submodules in GIT, sf, (Fri Dec 1, 3:35 pm)
Re: [RFC] Submodules in GIT, Martin Waitz, (Fri Dec 1, 3:40 pm)
Re: [RFC] Submodules in GIT, sf, (Fri Dec 1, 3:41 pm)
Re: [RFC] Submodules in GIT, Josef Weidendorfer, (Fri Dec 1, 3:55 pm)
Re: [RFC] Submodules in GIT, Josef Weidendorfer, (Fri Dec 1, 4:03 pm)
Re: [RFC] Submodules in GIT, Martin Waitz, (Fri Dec 1, 4:07 pm)
Re: [RFC] Submodules in GIT, Linus Torvalds, (Fri Dec 1, 4:09 pm)
Re: [RFC] Submodules in GIT, Josef Weidendorfer, (Fri Dec 1, 4:17 pm)
Re: [RFC] Submodules in GIT, Alan Chandler, (Fri Dec 1, 4:23 pm)
Re: [RFC] Submodules in GIT, Linus Torvalds, (Fri Dec 1, 4:30 pm)
Re: [RFC] Submodules in GIT, sf, (Fri Dec 1, 4:34 pm)
Re: [RFC] Submodules in GIT, Josef Weidendorfer, (Fri Dec 1, 4:36 pm)
Re: [RFC] Submodules in GIT, sf, (Fri Dec 1, 4:49 pm)
Re: [RFC] Submodules in GIT, Linus Torvalds, (Fri Dec 1, 5:12 pm)
Re: [RFC] Submodules in GIT, Josef Weidendorfer, (Fri Dec 1, 5:14 pm)
Re: [RFC] Submodules in GIT, Linus Torvalds, (Fri Dec 1, 5:33 pm)
Re: [RFC] Submodules in GIT, Andy Parkins, (Sat Dec 2, 2:22 am)
Re: [RFC] Submodules in GIT, Andy Parkins, (Sat Dec 2, 2:27 am)
Re: [RFC] Submodules in GIT, Andy Parkins, (Sat Dec 2, 3:04 am)
Re: [RFC] Submodules in GIT, Josef Weidendorfer, (Sat Dec 2, 4:32 am)
Re: [RFC] Submodules in GIT, Josef Weidendorfer, (Sat Dec 2, 6:50 am)
Re: [RFC] Submodules in GIT, Torgil Svensson, (Sat Dec 2, 11:57 am)
Re: [RFC] Submodules in GIT, Linus Torvalds, (Sat Dec 2, 12:41 pm)
Re: [RFC] Submodules in GIT, Martin Waitz, (Sat Dec 2, 12:46 pm)
Re: [RFC] Submodules in GIT, Linus Torvalds, (Sat Dec 2, 12:52 pm)
Re: [RFC] Submodules in GIT, Martin Waitz, (Sat Dec 2, 1:12 pm)
Re: [RFC] Submodules in GIT, Martin Waitz, (Sat Dec 2, 1:18 pm)
Re: [RFC] Submodules in GIT, Martin Waitz, (Sat Dec 2, 1:21 pm)
Re: [RFC] Submodules in GIT, Martin Waitz, (Sat Dec 2, 1:24 pm)
Re: [RFC] Submodules in GIT, Martin Waitz, (Sat Dec 2, 1:40 pm)
Re: [RFC] Submodules in GIT, Martin Waitz, (Sat Dec 2, 1:43 pm)
Re: [RFC] Submodules in GIT, Linus Torvalds, (Sat Dec 2, 1:44 pm)
Re: [RFC] Submodules in GIT, Linus Torvalds, (Sat Dec 2, 1:46 pm)
Re: [RFC] Submodules in GIT, Martin Waitz, (Sat Dec 2, 1:58 pm)
Re: [RFC] Submodules in GIT, Martin Waitz, (Sat Dec 2, 2:06 pm)
Re: [RFC] Submodules in GIT, Linus Torvalds, (Sat Dec 2, 2:22 pm)
Re: [RFC] Submodules in GIT, Linus Torvalds, (Sat Dec 2, 2:29 pm)
Re: [RFC] Submodules in GIT, Josef Weidendorfer, (Sat Dec 2, 5:55 pm)
Re: [RFC] Submodules in GIT, Josef Weidendorfer, (Sat Dec 2, 6:02 pm)
Re: [RFC] Submodules in GIT, Josef Weidendorfer, (Sat Dec 2, 6:11 pm)
Thoughts about memory requirements in traversals [Was: Re: ..., Josef Weidendorfer, (Sat Dec 2, 7:07 pm)
Re: Thoughts about memory requirements in traversals [Was: ..., Josef Weidendorfer, (Sat Dec 2, 8:21 pm)
Re: [RFC] Submodules in GIT, Martin Waitz, (Sat Dec 2, 11:29 pm)
Re: [RFC] Submodules in GIT, Torgil Svensson, (Sun Dec 3, 2:19 am)
Re: [RFC] Submodules in GIT, Linus Torvalds, (Sun Dec 3, 10:54 am)
Re: [RFC] Submodules in GIT, Andy Parkins, (Sun Dec 3, 12:33 pm)
Re: [RFC] Submodules in GIT, Martin Waitz, (Sun Dec 3, 1:46 pm)
Re: [RFC] Submodules in GIT, Sven Verdoolaege, (Sun Dec 3, 3:16 pm)
Re: [RFC] Submodules in GIT, Linus Torvalds, (Sun Dec 3, 3:32 pm)
Re: [RFC] Submodules in GIT, Josef Weidendorfer, (Mon Dec 4, 4:12 am)
Re: [RFC] Submodules in GIT, Michael K. Edwards, (Mon Dec 4, 11:56 am)
Re: [RFC] Submodules in GIT, Torgil Svensson, (Mon Dec 4, 1:26 pm)
Re: [RFC] Submodules in GIT, Linus Torvalds, (Mon Dec 4, 1:41 pm)
Re: [RFC] Submodules in GIT, Torgil Svensson, (Mon Dec 4, 2:36 pm)
Re: [RFC] Submodules in GIT, Sam Vilain, (Mon Dec 4, 6:31 pm)
Re: [RFC] Submodules in GIT, Daniel Barkalow, (Mon Dec 4, 7:33 pm)
Re: [RFC] Submodules in GIT, Uwe Kleine-Koenig, (Tue Dec 5, 2:01 am)
Re: [RFC] Submodules in GIT, Andreas Ericsson, (Tue Dec 5, 3:33 am)
Re: [RFC] Submodules in GIT, Andreas Ericsson, (Tue Dec 5, 3:38 am)
Re: [RFC] Submodules in GIT, Andreas Ericsson, (Tue Dec 5, 3:42 am)
Re: [RFC] Submodules in GIT, Uwe Kleine-Koenig, (Tue Dec 5, 8:02 am)
Re: [RFC] Submodules in GIT, Andreas Ericsson, (Tue Dec 5, 8:30 am)
Re: [RFC] Submodules in GIT, Sven Verdoolaege, (Tue Dec 5, 9:00 am)
Re: [RFC] Submodules in GIT, sf, (Tue Dec 5, 3:07 pm)
Re: [RFC] Submodules in GIT, Jon Loeliger, (Fri Dec 8, 11:29 am)
Re: [RFC] Submodules in GIT, Sven Verdoolaege, (Fri Dec 8, 11:45 am)
Re: [RFC] Submodules in GIT, R. Steve McKown, (Sat Dec 9, 2:34 pm)
Re: [RFC] Submodules in GIT, Torgil Svensson, (Sun Dec 10, 4:47 am)
Re: [RFC] Submodules in GIT, Andreas Ericsson, (Tue Dec 12, 1:32 am)
Re: [RFC] Submodules in GIT, Torgil Svensson, (Thu Dec 14, 2:27 pm)
Re: [RFC] Submodules in GIT, Josef Weidendorfer, (Thu Dec 14, 4:07 pm)
Re: [RFC] Submodules in GIT, Torgil Svensson, (Fri Dec 15, 10:43 am)
Re: [RFC] Submodules in GIT, Josef Weidendorfer, (Fri Dec 15, 2:42 pm)
Re: [RFC] Submodules in GIT, Torgil Svensson, (Fri Dec 15, 4:43 pm)
Re: [RFC] Submodules in GIT, Torgil Svensson, (Fri Dec 15, 6:13 pm)
Re: [RFC] Submodules in GIT, Torgil Svensson, (Fri Dec 15, 6:20 pm)
Re: [RFC] Submodules in GIT, Linus Torvalds, (Fri Dec 15, 6:49 pm)
Re: [RFC] Submodules in GIT, Linus Torvalds, (Fri Dec 15, 7:12 pm)
Re: [RFC] Submodules in GIT, Torgil Svensson, (Sat Dec 16, 1:50 am)