Re: pack operation is thrashing my server

Previous message: [thread] [date] [author]
Next message: [thread] [date] [author]
To: Nicolas Pitre <nico@...>
Cc: Geert Bosch <bosch@...>, Andi Kleen <andi@...>, Ken Pratt <ken@...>, <git@...>
Date: Wednesday, August 13, 2008 - 11:50 am

Nicolas Pitre wrote:

Don't forget that we need not just struct object_entry, but
also the struct commit/tree/blob, their hash tables, and the
struct object_entry* in the sorted object list table, and
the pack reverse index table. It does add up.

> > Have 500k objects and its suddenly something quite real in terms

I think we're closer to 100MB here due to the extra overheads
I just alluded to above, and which weren't in your 104 byte
per object figure.

> However there is a point where we should be realistic and just admit

Yea. But it would also be nice to be able to just concat packs
together. Especially if the repository in question is an open source
one and everything published is already known to be in the wild,
as say it is also available over dumb HTTP. Yea, I know people
like the 'security feature' of the packer not including objects
which aren't reachable.

But how many times has Linus published something to his linux-2.6
tree that he didn't mean to publish and had to rewind? I think
that may be "never". Yet how many times per day does his tree get
cloned from scratch?

This is also true for many internal corporate repositories.
Users probably have full read access to the object database anyway,
and maybe even have direct write access to it. Doing the object
enumeration there is pointless as a security measure.

I'm too busy to write a pack concat implementation proposal, so
I'll just shutup now. But it wouldn't be hard if someone wanted
to improve at least the initial clone serving case.

--
Shawn.
--
To unsubscribe from this list: send the line "unsubscribe git" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at http://vger.kernel.org/majordomo-info.html

Previous message: [thread] [date] [author]
Next message: [thread] [date] [author]

Messages in current thread:
pack operation is thrashing my server, Ken Pratt, (Sun Aug 10, 3:47 pm)
Re: pack operation is thrashing my server, Jakub Narebski, (Wed Aug 13, 8:43 am)
Re: pack operation is thrashing my server, Shawn O. Pearce, (Sun Aug 10, 11:04 pm)
Re: pack operation is thrashing my server, Ken Pratt, (Mon Aug 11, 3:43 am)
Re: pack operation is thrashing my server, Andi Kleen, (Mon Aug 11, 3:10 pm)
Re: pack operation is thrashing my server, Geert Bosch, (Tue Aug 12, 11:12 pm)
Re: pack operation is thrashing my server, Nicolas Pitre, (Wed Aug 13, 10:35 am)
Re: pack operation is thrashing my server, Geert Bosch, (Wed Aug 13, 12:01 pm)
Re: pack operation is thrashing my server, Nicolas Pitre, (Wed Aug 13, 1:26 pm)
Re: pack operation is thrashing my server, Dana How, (Wed Aug 13, 1:13 pm)
Re: pack operation is thrashing my server, Shawn O. Pearce, (Wed Aug 13, 10:59 am)
Re: pack operation is thrashing my server, Nicolas Pitre, (Wed Aug 13, 11:43 am)
Re: pack operation is thrashing my server, Shawn O. Pearce, (Wed Aug 13, 11:50 am)
Re: pack operation is thrashing my server, Nicolas Pitre, (Wed Aug 13, 1:04 pm)
Re: pack operation is thrashing my server, Linus Torvalds, (Thu Aug 14, 1:21 pm)
Re: pack operation is thrashing my server, Nicolas Pitre, (Thu Aug 14, 2:38 pm)
Re: pack operation is thrashing my server, Linus Torvalds, (Thu Aug 14, 2:55 pm)
Re: pack operation is thrashing my server, Linus Torvalds, (Thu Aug 14, 1:58 pm)
Re: pack operation is thrashing my server, Nicolas Pitre, (Thu Aug 14, 3:04 pm)
Re: pack operation is thrashing my server, Linus Torvalds, (Thu Aug 14, 3:44 pm)
Re: pack operation is thrashing my server, Nicolas Pitre, (Thu Aug 14, 5:50 pm)
Re: pack operation is thrashing my server, Linus Torvalds, (Thu Aug 14, 7:14 pm)
Re: pack operation is thrashing my server, Linus Torvalds, (Fri Aug 15, 8:34 pm)
Re: pack operation is thrashing my server, Junio C Hamano, (Sat Sep 6, 9:03 pm)
Re: pack operation is thrashing my server, Linus Torvalds, (Sat Sep 6, 9:46 pm)
Re: pack operation is thrashing my server, Mike Hommey, (Sun Sep 7, 3:45 am)
Re: pack operation is thrashing my server, Jon Smirl, (Sat Sep 6, 10:50 pm)
Re: pack operation is thrashing my server, Linus Torvalds, (Sat Sep 6, 11:07 pm)
Re: pack operation is thrashing my server, Andreas Ericsson, (Sun Sep 7, 4:18 am)
Re: pack operation is thrashing my server, Jon Smirl, (Sat Sep 6, 11:43 pm)
Re: pack operation is thrashing my server, Linus Torvalds, (Sun Sep 7, 12:50 am)
Re: pack operation is thrashing my server, Jon Smirl, (Sun Sep 7, 9:58 am)
Re: pack operation is thrashing my server, Nicolas Pitre, (Sun Sep 7, 1:08 pm)
Re: pack operation is thrashing my server, Jon Smirl, (Sun Sep 7, 4:33 pm)
Re: pack operation is thrashing my server, Nicolas Pitre, (Mon Sep 8, 10:17 am)
Re: pack operation is thrashing my server, Jon Smirl, (Mon Sep 8, 11:12 am)
Re: pack operation is thrashing my server, Jon Smirl, (Mon Sep 8, 12:01 pm)
Re: pack operation is thrashing my server, Junio C Hamano, (Sat Sep 6, 10:33 pm)
Re: pack operation is thrashing my server, Nicolas Pitre, (Sun Sep 7, 1:11 pm)
Re: pack operation is thrashing my server, Junio C Hamano, (Sun Sep 7, 1:41 pm)
Re: pack operation is thrashing my server, Björn, (Thu Aug 14, 7:39 pm)
Re: pack operation is thrashing my server, Linus Torvalds, (Thu Aug 14, 8:06 pm)
Re: pack operation is thrashing my server, Björn, (Sat Aug 16, 8:47 am)
Re: pack operation is thrashing my server, Linus Torvalds, (Thu Aug 14, 8:25 pm)
Re: pack operation is thrashing my server, Andi Kleen, (Thu Aug 14, 5:30 pm)
Re: pack operation is thrashing my server, Linus Torvalds, (Fri Aug 15, 12:15 pm)
Re: pack operation is thrashing my server, Andreas Ericsson, (Thu Aug 14, 2:33 am)
Re: pack operation is thrashing my server, Nicolas Pitre, (Thu Aug 14, 10:01 am)
Re: pack operation is thrashing my server, Thomas Rast, (Thu Aug 14, 6:04 am)
Re: pack operation is thrashing my server, Andreas Ericsson, (Thu Aug 14, 6:15 am)
Re: pack operation is thrashing my server, Shawn O. Pearce, (Thu Aug 14, 6:33 pm)
Re: pack operation is thrashing my server, Nicolas Pitre, (Thu Aug 14, 9:46 pm)
Re: pack operation is thrashing my server, Shawn O. Pearce, (Wed Aug 13, 1:19 pm)
Re: pack operation is thrashing my server, Shawn O. Pearce, (Tue Aug 12, 11:15 pm)
Re: pack operation is thrashing my server, Geert Bosch, (Tue Aug 12, 11:58 pm)
Re: pack operation is thrashing my server, Nicolas Pitre, (Wed Aug 13, 10:37 am)
Re: pack operation is thrashing my server, Jakub Narebski, (Wed Aug 13, 10:56 am)
Re: pack operation is thrashing my server, Shawn O. Pearce, (Wed Aug 13, 11:04 am)
Re: pack operation is thrashing my server, Johan Herland, (Wed Aug 13, 12:10 pm)
Re: pack operation is thrashing my server, Ken Pratt, (Wed Aug 13, 1:38 pm)
Re: pack operation is thrashing my server, Nicolas Pitre, (Wed Aug 13, 1:57 pm)
Re: pack operation is thrashing my server, David Tweed, (Wed Aug 13, 11:26 am)
Re: pack operation is thrashing my server, Martin Langhoff, (Wed Aug 13, 7:54 pm)
Re: pack operation is thrashing my server, David Tweed, (Thu Aug 14, 5:04 am)
Re: pack operation is thrashing my server, Shawn O. Pearce, (Mon Aug 11, 3:22 pm)
Re: pack operation is thrashing my server, Ken Pratt, (Mon Aug 11, 3:29 pm)
Re: pack operation is thrashing my server, Shawn O. Pearce, (Mon Aug 11, 3:34 pm)
Re: pack operation is thrashing my server, Andi Kleen, (Mon Aug 11, 4:10 pm)
Re: pack operation is thrashing my server, Ken Pratt, (Mon Aug 11, 3:15 pm)
Re: pack operation is thrashing my server, Nicolas Pitre, (Tue Aug 12, 10:38 pm)
Re: pack operation is thrashing my server, Andi Kleen, (Tue Aug 12, 10:50 pm)
Re: pack operation is thrashing my server, Shawn O. Pearce, (Tue Aug 12, 10:57 pm)
Re: pack operation is thrashing my server, Shawn O. Pearce, (Mon Aug 11, 11:01 am)
Re: pack operation is thrashing my server, Ken Pratt, (Mon Aug 11, 3:13 pm)
Re: pack operation is thrashing my server, Avery Pennarun, (Mon Aug 11, 11:40 am)
Re: pack operation is thrashing my server, Shawn O. Pearce, (Mon Aug 11, 11:59 am)
Re: pack operation is thrashing my server, Martin Langhoff, (Sun Aug 10, 7:06 pm)
Re: pack operation is thrashing my server, Ken Pratt, (Sun Aug 10, 7:12 pm)
Re: pack operation is thrashing my server, Martin Langhoff, (Sun Aug 10, 7:30 pm)
Re: pack operation is thrashing my server, Ken Pratt, (Sun Aug 10, 7:34 pm)