Re: git pack/unpack over bittorrent - works!

Previous message: [thread] [date] [author]
Next message: [thread] [date] [author]
From: Theodore Tso
Date: Friday, September 3, 2010 - 3:37 am

On Sep 2, 2010, at 7:09 PM, Nicolas Pitre wrote:


I wonder if this would work.   Assume for the moment that once every N days (where N might be one month), the "Gittorrent master" generates a "canonical pack construction file" that contains information about the objects, their delta pairing, the version of diff and zlib required, etc., and the SHA1 of the expected resulting encoding of this pack file.   This would contain the canonical pack for the entire repository, and for the Linux kernel, it might be released on www.kernel.org, and contain all of the objects from the beginning of time to the latest commit in Linus Torvalds' repository.

Although everybody's git repository will have different packs that they use and store natively, by following the "canonical pack construction file" they will be able to build up a canonical encoding for the pack that can then be used to more quickly allow newbie developers who are pulling the full clone of Linus's tree for the first time to download using a peer2peer Bittorrent style download.   So people who are willing to participate as part of the peer2peer network can download the instructions for how to make the canonical pack once a month, and use it to create the canonical pack.  If the "Gittorrent master" has spent a lot of time to carefully compute the most efficient set of delta pairings, they will get the slight benefit of a more efficient pack which they could use instead of their local one without having to use large values of --window and --depth to "git repack". 

This allows the peer2peer download to be used where it most matters --- for the bulk download for people who are cloning from the "canonical repository" for the first time.  After that, they will no doubt find it far more efficient to download incremental uploads using the git protocol. 

-- Ted

--
To unsubscribe from this list: send the line "unsubscribe git" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html
Previous message: [thread] [date] [author]
Next message: [thread] [date] [author]

Messages in current thread:
git pack/unpack over bittorrent - works!, Luke Kenneth Casson ..., (Wed Sep 1, 7:36 am)
Re: git pack/unpack over bittorrent - works!, Nguyen Thai Ngoc Duy, (Wed Sep 1, 3:04 pm)
Re: git pack/unpack over bittorrent - works!, Luke Kenneth Casson ..., (Thu Sep 2, 6:37 am)
Re: git pack/unpack over bittorrent - works!, Luke Kenneth Casson ..., (Thu Sep 2, 6:53 am)
Re: git pack/unpack over bittorrent - works!, Ævar Arnfjörð Bjarmason, (Thu Sep 2, 7:08 am)
Re: git pack/unpack over bittorrent - works!, A Large Angry SCM, (Thu Sep 2, 8:33 am)
Re: git pack/unpack over bittorrent - works!, Luke Kenneth Casson ..., (Thu Sep 2, 8:42 am)
Re: git pack/unpack over bittorrent - works!, Luke Kenneth Casson ..., (Thu Sep 2, 8:51 am)
Re: git pack/unpack over bittorrent - works!, Jeff King, (Thu Sep 2, 8:58 am)
Re: git pack/unpack over bittorrent - works!, Nicolas Pitre, (Thu Sep 2, 9:41 am)
Re: git pack/unpack over bittorrent - works!, A Large Angry SCM, (Thu Sep 2, 9:58 am)
Re: git pack/unpack over bittorrent - works!, A Large Angry SCM, (Thu Sep 2, 10:06 am)
Re: git pack/unpack over bittorrent - works!, A Large Angry SCM, (Thu Sep 2, 10:09 am)
Re: git pack/unpack over bittorrent - works!, Nicolas Pitre, (Thu Sep 2, 10:21 am)
Re: git pack/unpack over bittorrent - works!, Nicolas Pitre, (Thu Sep 2, 10:31 am)
Re: git pack/unpack over bittorrent - works!, Luke Kenneth Casson ..., (Thu Sep 2, 11:07 am)
Re: git pack/unpack over bittorrent - works!, Casey Dahlin, (Thu Sep 2, 11:23 am)
Re: git pack/unpack over bittorrent - works!, Luke Kenneth Casson ..., (Thu Sep 2, 12:17 pm)
Re: git pack/unpack over bittorrent - works!, Shawn O. Pearce, (Thu Sep 2, 12:29 pm)
Re: git pack/unpack over bittorrent - works!, Luke Kenneth Casson ..., (Thu Sep 2, 12:41 pm)
Re: git pack/unpack over bittorrent - works!, Luke Kenneth Casson ..., (Thu Sep 2, 12:51 pm)
Re: git pack/unpack over bittorrent - works!, A Large Angry SCM, (Thu Sep 2, 12:52 pm)
Re: git pack/unpack over bittorrent - works!, Luke Kenneth Casson ..., (Thu Sep 2, 1:06 pm)
Re: git pack/unpack over bittorrent - works!, Brandon Casey, (Thu Sep 2, 1:28 pm)
Re: git pack/unpack over bittorrent - works!, Jakub Narebski, (Thu Sep 2, 1:45 pm)
Re: git pack/unpack over bittorrent - works!, Luke Kenneth Casson ..., (Thu Sep 2, 1:48 pm)
Re: git pack/unpack over bittorrent - works!, Luke Kenneth Casson ..., (Thu Sep 2, 2:10 pm)
Re: git pack/unpack over bittorrent - works!, Luke Kenneth Casson ..., (Thu Sep 2, 2:19 pm)
Re: git pack/unpack over bittorrent - works!, Nicolas Pitre, (Thu Sep 2, 4:09 pm)
Re: git pack/unpack over bittorrent - works!, Nicolas Pitre, (Thu Sep 2, 5:29 pm)
Re: git pack/unpack over bittorrent - works!, Nicolas Pitre, (Thu Sep 2, 5:36 pm)
Re: git pack/unpack over bittorrent - works!, Nguyen Thai Ngoc Duy, (Thu Sep 2, 7:48 pm)
Re: git pack/unpack over bittorrent - works!, Luke Kenneth Casson ..., (Fri Sep 3, 3:23 am)
Re: git pack/unpack over bittorrent - works!, Luke Kenneth Casson ..., (Fri Sep 3, 3:34 am)
Re: git pack/unpack over bittorrent - works!, Theodore Tso, (Fri Sep 3, 3:37 am)
Re: git pack/unpack over bittorrent - works!, Luke Kenneth Casson ..., (Fri Sep 3, 3:54 am)
Re: git pack/unpack over bittorrent - works!, Luke Kenneth Casson ..., (Fri Sep 3, 3:55 am)
Re: git pack/unpack over bittorrent - works!, Luke Kenneth Casson ..., (Fri Sep 3, 4:04 am)
Re: git pack/unpack over bittorrent - works!, Junio C Hamano, (Fri Sep 3, 10:03 am)
Re: git pack/unpack over bittorrent - works!, Junio C Hamano, (Fri Sep 3, 10:12 am)
Re: git pack/unpack over bittorrent - works!, Ted Ts'o, (Fri Sep 3, 11:31 am)
Re: git pack/unpack over bittorrent - works!, Nicolas Pitre, (Fri Sep 3, 12:41 pm)
Re: git pack/unpack over bittorrent - works!, Luke Kenneth Casson ..., (Fri Sep 3, 2:11 pm)
Re: git pack/unpack over bittorrent - works!, Nguyen Thai Ngoc Duy, (Fri Sep 3, 5:24 pm)
Re: git pack/unpack over bittorrent - works!, Nguyen Thai Ngoc Duy, (Fri Sep 3, 5:57 pm)
Re: git pack/unpack over bittorrent - works!, Artur Skawina, (Fri Sep 3, 6:52 pm)
Re: git pack/unpack over bittorrent - works!, Theodore Tso, (Fri Sep 3, 6:57 pm)
Re: git pack/unpack over bittorrent - works!, Nicolas Pitre, (Fri Sep 3, 9:39 pm)
Re: git pack/unpack over bittorrent - works!, Kyle Moffett, (Fri Sep 3, 10:23 pm)
Re: git pack/unpack over bittorrent - works!, Nicolas Pitre, (Fri Sep 3, 10:40 pm)
Re: git pack/unpack over bittorrent - works!, Artur Skawina, (Fri Sep 3, 10:42 pm)
Re: git pack/unpack over bittorrent - works!, Nicolas Pitre, (Fri Sep 3, 11:13 pm)
Re: git pack/unpack over bittorrent - works!, Theodore Tso, (Sat Sep 4, 4:46 am)
Re: git pack/unpack over bittorrent - works!, Luke Kenneth Casson ..., (Sat Sep 4, 4:58 am)
Re: git pack/unpack over bittorrent - works!, Theodore Tso, (Sat Sep 4, 5:00 am)
Re: git pack/unpack over bittorrent - works!, Luke Kenneth Casson ..., (Sat Sep 4, 5:33 am)
Re: git pack/unpack over bittorrent - works!, Luke Kenneth Casson ..., (Sat Sep 4, 5:44 am)
Re: git pack/unpack over bittorrent - works!, Luke Kenneth Casson ..., (Sat Sep 4, 6:14 am)
Re: git pack/unpack over bittorrent - works!, Artur Skawina, (Sat Sep 4, 6:42 am)
Re: git pack/unpack over bittorrent - works!, Luke Kenneth Casson ..., (Sat Sep 4, 7:06 am)
Re: git pack/unpack over bittorrent - works!, Luke Kenneth Casson ..., (Sat Sep 4, 7:50 am)
Re: git pack/unpack over bittorrent - works!, Artur Skawina, (Sat Sep 4, 10:23 am)
Re: git pack/unpack over bittorrent - works!, Ted Ts'o, (Sat Sep 4, 11:14 am)
Re: git pack/unpack over bittorrent - works!, Artur Skawina, (Sat Sep 4, 11:46 am)
Re: git pack/unpack over bittorrent - works!, Luke Kenneth Casson ..., (Sat Sep 4, 1:00 pm)
Re: git pack/unpack over bittorrent - works!, Jakub Narebski, (Sat Sep 4, 1:20 pm)
Re: git pack/unpack over bittorrent - works!, Luke Kenneth Casson ..., (Sat Sep 4, 1:47 pm)
Re: git pack/unpack over bittorrent - works!, Jakub Narebski, (Sat Sep 4, 2:16 pm)
Re: git pack/unpack over bittorrent - works!, Luke Kenneth Casson ..., (Sat Sep 4, 2:24 pm)
Re: git pack/unpack over bittorrent - works!, Ted Ts'o, (Sat Sep 4, 3:41 pm)
Re: git pack/unpack over bittorrent - works!, Ted Ts'o, (Sat Sep 4, 3:47 pm)
Re: git pack/unpack over bittorrent - works!, Nicolas Pitre, (Sat Sep 4, 6:18 pm)
Re: git pack/unpack over bittorrent - works!, Nicolas Pitre, (Sat Sep 4, 6:32 pm)
Re: git pack/unpack over bittorrent - works!, Tomas Carnecky, (Sat Sep 4, 6:43 pm)
Re: git pack/unpack over bittorrent - works!, Nicolas Pitre, (Sat Sep 4, 7:16 pm)
Re: git pack/unpack over bittorrent - works!, Luke Kenneth Casson ..., (Sun Sep 5, 10:16 am)
Re: git pack/unpack over bittorrent - works!, Luke Kenneth Casson ..., (Sun Sep 5, 10:22 am)
Re: git pack/unpack over bittorrent - works!, Luke Kenneth Casson ..., (Sun Sep 5, 10:25 am)
Re: git pack/unpack over bittorrent - works!, Luke Kenneth Casson ..., (Sun Sep 5, 11:05 am)
Re: git pack/unpack over bittorrent - works!, Nicolas Pitre, (Sun Sep 5, 4:52 pm)
Re: git pack/unpack over bittorrent - works!, Nicolas Pitre, (Sun Sep 5, 5:05 pm)
Re: git pack/unpack over bittorrent - works!, Luke Kenneth Casson ..., (Mon Sep 6, 6:23 am)
Re: git pack/unpack over bittorrent - works!, Nicolas Pitre, (Mon Sep 6, 9:51 am)
Re: git pack/unpack over bittorrent - works!, Luke Kenneth Casson ..., (Mon Sep 6, 3:33 pm)
Re: git pack/unpack over bittorrent - works!, Junio C Hamano, (Mon Sep 6, 4:34 pm)
Re: git pack/unpack over bittorrent - works!, Nicolas Pitre, (Mon Sep 6, 4:57 pm)
Re: git pack/unpack over bittorrent - works!, Luke Kenneth Casson ..., (Mon Sep 6, 5:17 pm)
Re: git pack/unpack over bittorrent - works!, Luke Kenneth Casson ..., (Mon Sep 6, 5:29 pm)