Re: Smart fetch via HTTP?

Previous message: [thread] [date] [author]
Next message: [thread] [date] [author]
From: Nicolas Pitre
Date: Wednesday, May 16, 2007 - 8:45 pm

On Wed, 16 May 2007, Shawn O. Pearce wrote:


Well, in the commit objects case you're likely to have a bunch of them 
all contigous.

For tree and blob objects it is less likely.

And of course there is the question of deltas for which you might or 
might not have the base object locally already.

Still... I wonder if this could be actually workable.  A typical daily 
update on the Linux kernel repository might consist of a couple hundreds 
or a few tousands objects.  This could still be faster to fetch parts of 
a pack than the whole pack if the size difference is above a certain 
treshold.  It is certainly not worse than fetching loose objects.

Things would be pretty horrid if you think of fetching a commit object, 
parsing it to find out what tree object to fetch, then parse that tree 
object to find out what other objects to fetch, and so on.

But if you only take the approach of fetching the pack index files, 
finding out about the objects that the remote has that are not available 
locally, and then fetching all those objects from within pack files 
without even looking at them (except for deltas), then it should be 
possible to issue a couple requests in parallel and possibly have decent 
performances.  And if it turns out that more than, say, 70% of a 
particular pack is to be fetched (you can determine that up front), then 
it might be decided to fetch the whole pack.

There is no way to sensibly keep those objects packed on the receiving 
end of course, but storing them as loose objects and repacking them 
afterwards should be just fine.

Of course you'll get objects from branches in the remote repository you 
might not be interested in, but that's a price to pay for such a hack.  
On average the overhead shouldn't be that big anyway if branches within 
a repository are somewhat related.

I think this is something worth experimenting.


Nicolas
-
To unsubscribe from this list: send the line "unsubscribe git" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html
Previous message: [thread] [date] [author]
Next message: [thread] [date] [author]

Messages in current thread:
Smart fetch via HTTP?, Jan Hudec, (Tue May 15, 1:10 pm)
Re: Smart fetch via HTTP?, A Large Angry SCM, (Tue May 15, 3:30 pm)
Re: Smart fetch via HTTP?, Shawn O. Pearce, (Tue May 15, 4:29 pm)
Re: Smart fetch via HTTP?, Junio C Hamano, (Tue May 15, 5:38 pm)
Re: Smart fetch via HTTP?, Martin Langhoff, (Tue May 15, 10:25 pm)
Re: Smart fetch via HTTP?, Johannes Schindelin, (Wed May 16, 4:33 am)
Re: Smart fetch via HTTP?, Martin Langhoff, (Wed May 16, 2:26 pm)
Re: Smart fetch via HTTP?, Johannes Schindelin, (Wed May 16, 5:52 pm)
Re: Smart fetch via HTTP?, Shawn O. Pearce, (Wed May 16, 6:03 pm)
Re: Smart fetch via HTTP?, david, (Wed May 16, 6:04 pm)
Re: Smart fetch via HTTP?, Shawn O. Pearce, (Wed May 16, 6:26 pm)
Re: Smart fetch via HTTP?, Shawn O. Pearce, (Wed May 16, 6:45 pm)
Re: Smart fetch via HTTP?, Nicolas Pitre, (Wed May 16, 8:45 pm)
Re: Smart fetch via HTTP?, Johannes Schindelin, (Thu May 17, 3:48 am)
Re: Smart fetch via HTTP?, Matthieu Moy, (Thu May 17, 4:28 am)
Re: Smart fetch via HTTP?, Theodore Tso, (Thu May 17, 5:36 am)
Re: Smart fetch via HTTP?, Petr Baudis, (Thu May 17, 5:40 am)
Re: Smart fetch via HTTP?, Matthieu Moy, (Thu May 17, 5:48 am)
Re: Smart fetch via HTTP?, Martin Langhoff, (Thu May 17, 6:10 am)
Re: Smart fetch via HTTP?, Johannes Schindelin, (Thu May 17, 6:47 am)
Re: Smart fetch via HTTP?, Matthieu Moy, (Thu May 17, 7:05 am)
Re: Smart fetch via HTTP?, Martin Langhoff, (Thu May 17, 7:09 am)
Re: Smart fetch via HTTP?, Nicolas Pitre, (Thu May 17, 7:41 am)
Re: Smart fetch via HTTP?, Nicolas Pitre, (Thu May 17, 7:50 am)
Re: Smart fetch via HTTP?, Nicolas Pitre, (Thu May 17, 8:01 am)
Re: Smart fetch via HTTP?, Martin Langhoff, (Thu May 17, 8:24 am)
Re: Smart fetch via HTTP?, Nicolas Pitre, (Thu May 17, 8:34 am)
Re: Smart fetch via HTTP?, Jan Hudec, (Thu May 17, 1:04 pm)
Re: Smart fetch via HTTP?, Jan Hudec, (Thu May 17, 1:26 pm)
Re: Smart fetch via HTTP?, Nicolas Pitre, (Thu May 17, 1:31 pm)
Re: Smart fetch via HTTP?, Nicolas Pitre, (Thu May 17, 1:38 pm)
Re: Smart fetch via HTTP?, david, (Thu May 17, 2:00 pm)
Re: Smart fetch via HTTP?, Johannes Schindelin, (Fri May 18, 2:01 am)
Re: Smart fetch via HTTP?, Jan Hudec, (Fri May 18, 10:35 am)
Re: Smart fetch via HTTP?, Jan Hudec, (Fri May 18, 10:51 am)
Re: Smart fetch via HTTP?, Linus Torvalds, (Fri May 18, 11:27 am)
Re: Smart fetch via HTTP?, alan, (Fri May 18, 11:33 am)
Re: Smart fetch via HTTP?, Joel Becker, (Fri May 18, 12:01 pm)
Re: Smart fetch via HTTP?, Matthieu Moy, (Fri May 18, 1:06 pm)
Re: Smart fetch via HTTP?, Linus Torvalds, (Fri May 18, 1:13 pm)
Re: Smart fetch via HTTP?, Joel Becker, (Fri May 18, 2:56 pm)
Re: Smart fetch via HTTP?, david, (Fri May 18, 5:50 pm)
Re: Smart fetch via HTTP?, Shawn O. Pearce, (Fri May 18, 8:58 pm)
Re: Smart fetch via HTTP?, david, (Fri May 18, 9:58 pm)
Re: Smart fetch via HTTP?, Jan Hudec, (Sun May 20, 3:30 am)