Re: pack operation is thrashing my server

Previous message: [thread] [date] [author]
Next message: [thread] [date] [author]
From: Björn
Date: Saturday, August 16, 2008 - 5:47 am

On 2008.08.14 17:06:13 -0700, Linus Torvalds wrote:

So I added some stupid tracing to cache_or_unpack entry to see how often
we reread the same stuff. The whole thing just logs the base_offset in
case of a cache miss. I've gc'ed my linux-2.6.git before the run, so
that there's only a single packed_git around (at least I hope so), and I
can ignore that for the tracing.

The whole log for a "git rev-list --objects HEAD" has about 1.2M
entries, while the output of the rev-list command has about 870k lines.
Some postprocessing of the trace shows that the majority of objects is
read only once or twice. A few percent are read three to ten times, and
some are read more than two hundred times.

I'll attach the post-processed thing. The format is:
 x y

Meaning that there were x base_offset values for which we had y cache
misses.

Bj
Previous message: [thread] [date] [author]
Next message: [thread] [date] [author]

Messages in current thread:
pack operation is thrashing my server, Ken Pratt, (Sun Aug 10, 12:47 pm)
Re: pack operation is thrashing my server, Martin Langhoff, (Sun Aug 10, 4:06 pm)
Re: pack operation is thrashing my server, Ken Pratt, (Sun Aug 10, 4:12 pm)
Re: pack operation is thrashing my server, Martin Langhoff, (Sun Aug 10, 4:30 pm)
Re: pack operation is thrashing my server, Ken Pratt, (Sun Aug 10, 4:34 pm)
Re: pack operation is thrashing my server, Shawn O. Pearce, (Sun Aug 10, 8:04 pm)
Re: pack operation is thrashing my server, Ken Pratt, (Mon Aug 11, 12:43 am)
Re: pack operation is thrashing my server, Shawn O. Pearce, (Mon Aug 11, 8:01 am)
Re: pack operation is thrashing my server, Avery Pennarun, (Mon Aug 11, 8:40 am)
Re: pack operation is thrashing my server, Shawn O. Pearce, (Mon Aug 11, 8:59 am)
Re: pack operation is thrashing my server, Andi Kleen, (Mon Aug 11, 12:10 pm)
Re: pack operation is thrashing my server, Ken Pratt, (Mon Aug 11, 12:13 pm)
Re: pack operation is thrashing my server, Ken Pratt, (Mon Aug 11, 12:15 pm)
Re: pack operation is thrashing my server, Shawn O. Pearce, (Mon Aug 11, 12:22 pm)
Re: pack operation is thrashing my server, Ken Pratt, (Mon Aug 11, 12:29 pm)
Re: pack operation is thrashing my server, Shawn O. Pearce, (Mon Aug 11, 12:34 pm)
Re: pack operation is thrashing my server, Andi Kleen, (Mon Aug 11, 1:10 pm)
Re: pack operation is thrashing my server, Nicolas Pitre, (Tue Aug 12, 7:38 pm)
Re: pack operation is thrashing my server, Andi Kleen, (Tue Aug 12, 7:50 pm)
Re: pack operation is thrashing my server, Shawn O. Pearce, (Tue Aug 12, 7:57 pm)
Re: pack operation is thrashing my server, Geert Bosch, (Tue Aug 12, 8:12 pm)
Re: pack operation is thrashing my server, Shawn O. Pearce, (Tue Aug 12, 8:15 pm)
Re: pack operation is thrashing my server, Geert Bosch, (Tue Aug 12, 8:58 pm)
Re: pack operation is thrashing my server, Jakub Narebski, (Wed Aug 13, 5:43 am)
Re: pack operation is thrashing my server, Nicolas Pitre, (Wed Aug 13, 7:35 am)
Re: pack operation is thrashing my server, Nicolas Pitre, (Wed Aug 13, 7:37 am)
Re: pack operation is thrashing my server, Jakub Narebski, (Wed Aug 13, 7:56 am)
Re: pack operation is thrashing my server, Shawn O. Pearce, (Wed Aug 13, 7:59 am)
Re: pack operation is thrashing my server, Shawn O. Pearce, (Wed Aug 13, 8:04 am)
Re: pack operation is thrashing my server, David Tweed, (Wed Aug 13, 8:26 am)
Re: pack operation is thrashing my server, Nicolas Pitre, (Wed Aug 13, 8:43 am)
Re: pack operation is thrashing my server, Shawn O. Pearce, (Wed Aug 13, 8:50 am)
Re: pack operation is thrashing my server, Geert Bosch, (Wed Aug 13, 9:01 am)
Re: pack operation is thrashing my server, Johan Herland, (Wed Aug 13, 9:10 am)
Re: pack operation is thrashing my server, Nicolas Pitre, (Wed Aug 13, 10:04 am)
Re: pack operation is thrashing my server, Dana How, (Wed Aug 13, 10:13 am)
Re: pack operation is thrashing my server, Shawn O. Pearce, (Wed Aug 13, 10:19 am)
Re: pack operation is thrashing my server, Nicolas Pitre, (Wed Aug 13, 10:26 am)
Re: pack operation is thrashing my server, Ken Pratt, (Wed Aug 13, 10:38 am)
Re: pack operation is thrashing my server, Nicolas Pitre, (Wed Aug 13, 10:57 am)
Re: pack operation is thrashing my server, Martin Langhoff, (Wed Aug 13, 4:54 pm)
Re: pack operation is thrashing my server, Andreas Ericsson, (Wed Aug 13, 11:33 pm)
Re: pack operation is thrashing my server, David Tweed, (Thu Aug 14, 2:04 am)
Re: pack operation is thrashing my server, Thomas Rast, (Thu Aug 14, 3:04 am)
Re: pack operation is thrashing my server, Andreas Ericsson, (Thu Aug 14, 3:15 am)
Re: pack operation is thrashing my server, Nicolas Pitre, (Thu Aug 14, 7:01 am)
Re: pack operation is thrashing my server, Linus Torvalds, (Thu Aug 14, 10:21 am)
Re: pack operation is thrashing my server, Linus Torvalds, (Thu Aug 14, 10:58 am)
Re: pack operation is thrashing my server, Nicolas Pitre, (Thu Aug 14, 11:38 am)
Re: pack operation is thrashing my server, Linus Torvalds, (Thu Aug 14, 11:55 am)
Re: pack operation is thrashing my server, Nicolas Pitre, (Thu Aug 14, 12:04 pm)
Re: pack operation is thrashing my server, Linus Torvalds, (Thu Aug 14, 12:44 pm)
Re: pack operation is thrashing my server, Andi Kleen, (Thu Aug 14, 2:30 pm)
Re: pack operation is thrashing my server, Nicolas Pitre, (Thu Aug 14, 2:50 pm)
Re: pack operation is thrashing my server, Shawn O. Pearce, (Thu Aug 14, 3:33 pm)
Re: pack operation is thrashing my server, Linus Torvalds, (Thu Aug 14, 4:14 pm)
Re: pack operation is thrashing my server, Björn, (Thu Aug 14, 4:39 pm)
Re: pack operation is thrashing my server, Linus Torvalds, (Thu Aug 14, 5:06 pm)
Re: pack operation is thrashing my server, Linus Torvalds, (Thu Aug 14, 5:25 pm)
Re: pack operation is thrashing my server, Nicolas Pitre, (Thu Aug 14, 6:46 pm)
Re: pack operation is thrashing my server, Linus Torvalds, (Fri Aug 15, 9:15 am)
Re: pack operation is thrashing my server, Linus Torvalds, (Fri Aug 15, 5:34 pm)
Re: pack operation is thrashing my server, Björn, (Sat Aug 16, 5:47 am)
Re: pack operation is thrashing my server, Junio C Hamano, (Sat Sep 6, 6:03 pm)
Re: pack operation is thrashing my server, Linus Torvalds, (Sat Sep 6, 6:46 pm)
Re: pack operation is thrashing my server, Junio C Hamano, (Sat Sep 6, 7:33 pm)
Re: pack operation is thrashing my server, Jon Smirl, (Sat Sep 6, 7:50 pm)
Re: pack operation is thrashing my server, Linus Torvalds, (Sat Sep 6, 8:07 pm)
Re: pack operation is thrashing my server, Jon Smirl, (Sat Sep 6, 8:43 pm)
Re: pack operation is thrashing my server, Linus Torvalds, (Sat Sep 6, 9:50 pm)
Re: pack operation is thrashing my server, Mike Hommey, (Sun Sep 7, 12:45 am)
Re: pack operation is thrashing my server, Andreas Ericsson, (Sun Sep 7, 1:18 am)
Re: pack operation is thrashing my server, Jon Smirl, (Sun Sep 7, 6:58 am)
Re: pack operation is thrashing my server, Nicolas Pitre, (Sun Sep 7, 10:08 am)
Re: pack operation is thrashing my server, Nicolas Pitre, (Sun Sep 7, 10:11 am)
Re: pack operation is thrashing my server, Junio C Hamano, (Sun Sep 7, 10:41 am)
Re: pack operation is thrashing my server, Jon Smirl, (Sun Sep 7, 1:33 pm)
Re: pack operation is thrashing my server, Nicolas Pitre, (Mon Sep 8, 7:17 am)
Re: pack operation is thrashing my server, Jon Smirl, (Mon Sep 8, 8:12 am)
Re: pack operation is thrashing my server, Jon Smirl, (Mon Sep 8, 9:01 am)