cleaner/better zlib sources?

!MAILaRCHIVE_VOTE_RePLACE
Previous message: [thread] [date] [author]
Next message: [thread] [date] [author]
To: Git Mailing List <git@...>
Date: Thursday, March 15, 2007 - 9:04 pm

I looked at git profiles yesterday, and some of them are pretty scary. We 
spend about 50% of the time under some loads in just zlib uncompression, 
and when I actually looked closer at the zlib sources I can kind of 
understand why. That thing is horrid.

The sad part is that it looks like it should be quite possible to make 
zlib simply just perform better. The profiles seem to say that a lot of 
the cost is literally in the "inflate()" state machine code (and by that I 
mean *not* the code itself, but literally in the indirect jump generated 
by the case-statement).

Now, on any high-performance CPU, doing state-machines by having

	for (;;)
		switch (data->state) {
			...
			data->state = NEW_STATE;
			continue;
		}

(which is what zlib seems to be doing) is just about the worst possible 
way to code things.

Now, it's possible that I'm just wrong, but the instruction-level profile 
really did pinpoint the "look up state branch pointer and jump to it" as 
some of the hottest part of that function. Which is just *evil*. You can 
most likely use direct jumps within the loop (zero cost at all on most OoO 
CPU's) most of the time, and the entry condition is likely quite 
predictable too, so a lot of that overhead seems to be just sad and 
unnecessary.

Now, I'm just wondering if anybody knows if there are better zlib 
implementations out there? This really looks like it could be a noticeable 
performance issue, but I'm lazy and would be much happier to hear that 
somebody has already played with optimizing zlib. Especially since I'm not 
100% sure it's really going to be noticeable..

		Linus
-
To unsubscribe from this list: send the line "unsubscribe git" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html
Previous message: [thread] [date] [author]
Next message: [thread] [date] [author]

Messages in current thread:
cleaner/better zlib sources?, Linus Torvalds, (Thu Mar 15, 9:04 pm)
Re: cleaner/better zlib sources?, Davide Libenzi, (Thu Mar 15, 9:33 pm)
Re: cleaner/better zlib sources?, Davide Libenzi, (Thu Mar 15, 10:06 pm)
Re: cleaner/better zlib sources?, Jeff Garzik, (Thu Mar 15, 9:11 pm)
Re: cleaner/better zlib sources?, Linus Torvalds, (Thu Mar 15, 9:46 pm)
Re: cleaner/better zlib sources?, Linus Torvalds, (Thu Mar 15, 9:54 pm)
Re: cleaner/better zlib sources?, Davide Libenzi, (Thu Mar 15, 10:43 pm)
Re: cleaner/better zlib sources?, Linus Torvalds, (Thu Mar 15, 10:56 pm)
Re: cleaner/better zlib sources?, Davide Libenzi, (Thu Mar 15, 11:16 pm)
Re: cleaner/better zlib sources?, Linus Torvalds, (Fri Mar 16, 12:21 pm)
Re: cleaner/better zlib sources?, Nicolas Pitre, (Fri Mar 16, 1:06 pm)
Re: cleaner/better zlib sources?, Linus Torvalds, (Fri Mar 16, 1:51 pm)
Re: cleaner/better zlib sources?, Nicolas Pitre, (Fri Mar 16, 2:09 pm)
Re: cleaner/better zlib sources?, Jeff Garzik, (Fri Mar 16, 12:35 pm)
Re: cleaner/better zlib sources?, Shawn O. Pearce, (Fri Mar 16, 7:22 pm)
Re: cleaner/better zlib sources?, Nicolas Pitre, (Fri Mar 16, 1:12 pm)
Re: cleaner/better zlib sources?, Matt Mackall, (Fri Mar 16, 12:42 pm)
Re: cleaner/better zlib sources?, Linus Torvalds, (Fri Mar 16, 12:51 pm)
Re: cleaner/better zlib sources?, Davide Libenzi, (Fri Mar 16, 12:24 pm)
Re: cleaner/better zlib sources?, Linus Torvalds, (Fri Mar 16, 12:35 pm)
Re: cleaner/better zlib sources?, Davide Libenzi, (Fri Mar 16, 3:21 pm)
Re: cleaner/better zlib sources?, Linus Torvalds, (Fri Mar 16, 8:01 pm)
Re: cleaner/better zlib sources?, Linus Torvalds, (Fri Mar 16, 9:11 pm)
Re: cleaner/better zlib sources?, Nicolas Pitre, (Fri Mar 16, 11:28 pm)
Re: cleaner/better zlib sources?, Linus Torvalds, (Sat Mar 17, 1:55 pm)
Re: cleaner/better zlib sources?, Linus Torvalds, (Sat Mar 17, 3:40 pm)
[PATCH 2/2] Implement a simple delta_base cache, Linus Torvalds, (Sat Mar 17, 3:44 pm)
Re: [PATCH 2/2] Implement a simple delta_base cache, Linus Torvalds, (Sat Mar 17, 6:44 pm)
Re: [PATCH 2/2] Implement a simple delta_base cache, Linus Torvalds, (Sat Mar 17, 5:45 pm)
Re: [PATCH 2/2] Implement a simple delta_base cache, Morten Welinder, (Sat Mar 17, 9:14 pm)
Re: [PATCH 2/2] Implement a simple delta_base cache, Avi Kivity, (Sun Mar 18, 2:28 am)
Re: [PATCH 2/2] Implement a simple delta_base cache, Linus Torvalds, (Sat Mar 17, 9:29 pm)
Re: [PATCH 2/2] Implement a simple delta_base cache, Linus Torvalds, (Sat Mar 17, 9:44 pm)
Re: [PATCH 2/2] Implement a simple delta_base cache, Nicolas Pitre, (Sat Mar 17, 9:38 pm)
Re: [PATCH 2/2] Implement a simple delta_base cache, Linus Torvalds, (Sat Mar 17, 9:55 pm)
Re: [PATCH 2/2] Implement a simple delta_base cache, Nicolas Pitre, (Sat Mar 17, 10:03 pm)
Re: [PATCH 2/2] Implement a simple delta_base cache, Linus Torvalds, (Sat Mar 17, 10:20 pm)
[PATCH 3/2] Avoid unnecessary strlen() calls, Linus Torvalds, (Sat Mar 17, 11:06 pm)
Re: [PATCH 3/2] Avoid unnecessary strlen() calls, Junio C Hamano, (Sun Mar 18, 5:45 am)
Re: [PATCH 3/2] Avoid unnecessary strlen() calls, Linus Torvalds, (Sun Mar 18, 11:54 am)
Re: [PATCH 3/2] Avoid unnecessary strlen() calls, Junio C Hamano, (Mon Mar 19, 11:16 pm)
Re: [PATCH 3/2] Avoid unnecessary strlen() calls, Linus Torvalds, (Tue Mar 20, 12:31 am)
Re: [PATCH 3/2] Avoid unnecessary strlen() calls, Shawn O. Pearce, (Tue Mar 20, 12:39 am)
Re: [PATCH 3/2] Avoid unnecessary strlen() calls, Linus Torvalds, (Tue Mar 20, 12:57 am)
Re: [PATCH 3/2] Avoid unnecessary strlen() calls, Johannes Schindelin, (Mon Mar 19, 11:05 pm)
Re: [PATCH 3/2] Avoid unnecessary strlen() calls, Shawn O. Pearce, (Mon Mar 19, 11:29 pm)
Re: [PATCH 3/2] Avoid unnecessary strlen() calls, Shawn O. Pearce, (Mon Mar 19, 11:40 pm)
Re: [PATCH 3/2] Avoid unnecessary strlen() calls, Linus Torvalds, (Tue Mar 20, 12:11 am)
Re: [PATCH 3/2] Avoid unnecessary strlen() calls, Junio C Hamano, (Tue Mar 20, 1:44 am)
Re: [PATCH 3/2] Avoid unnecessary strlen() calls, Shawn O. Pearce, (Tue Mar 20, 12:18 am)
Re: [PATCH 3/2] Avoid unnecessary strlen() calls, Linus Torvalds, (Tue Mar 20, 12:45 am)
Re: [PATCH 3/2] Avoid unnecessary strlen() calls, Linus Torvalds, (Sun Mar 18, 11:57 am)
Re: [PATCH 3/2] Avoid unnecessary strlen() calls, Shawn O. Pearce, (Sun Mar 18, 5:38 pm)
Re: [PATCH 3/2] Avoid unnecessary strlen() calls, Linus Torvalds, (Sun Mar 18, 5:48 pm)
Re: [PATCH 2/2] Implement a simple delta_base cache, Nicolas Pitre, (Sat Mar 17, 11:00 pm)
Re: [PATCH 2/2] Implement a simple delta_base cache, Linus Torvalds, (Sat Mar 17, 11:31 pm)
Re: [PATCH 2/2] Implement a simple delta_base cache, Robin Rosenberg, (Sun Mar 18, 6:53 am)
Re: [PATCH 2/2] Implement a simple delta_base cache, Linus Torvalds, (Sun Mar 18, 1:34 pm)
Re: [PATCH 2/2] Implement a simple delta_base cache, Robin Rosenberg, (Sun Mar 18, 2:29 pm)
Re: [PATCH 2/2] Implement a simple delta_base cache, David Brodsky, (Mon Mar 19, 9:16 am)
Re: [PATCH 2/2] Implement a simple delta_base cache, Robin Rosenberg, (Tue Mar 20, 2:35 am)
Re: [PATCH 2/2] Implement a simple delta_base cache, David Brodsky, (Tue Mar 20, 5:13 am)
Re: [PATCH 2/2] Implement a simple delta_base cache, Linus Torvalds, (Tue Mar 20, 10:37 pm)
Re: [PATCH 2/2] Implement a simple delta_base cache, Nicolas Pitre, (Tue Mar 20, 10:54 pm)
Re: [PATCH 2/2] Implement a simple delta_base cache, Shawn O. Pearce, (Sun Mar 18, 5:25 pm)
Re: [PATCH 2/2] Implement a simple delta_base cache, Julian Phillips, (Sun Mar 18, 1:30 am)
Re: [PATCH 2/2] Implement a simple delta_base cache, Linus Torvalds, (Sun Mar 18, 1:23 pm)
Re: [PATCH 2/2] Implement a simple delta_base cache, Junio C Hamano, (Sat Mar 17, 7:12 pm)
Re: [PATCH 2/2] Implement a simple delta_base cache, Linus Torvalds, (Sat Mar 17, 7:24 pm)
Re: [PATCH 2/2] Implement a simple delta_base cache, Jon Smirl, (Sat Mar 17, 7:52 pm)
Re: [PATCH 2/2] Implement a simple delta_base cache, Junio C Hamano, (Sat Mar 17, 6:37 pm)
Re: [PATCH 2/2] Implement a simple delta_base cache, Nicolas Pitre, (Sat Mar 17, 9:13 pm)
Re: [PATCH 2/2] Implement a simple delta_base cache, Junio C Hamano, (Sun Mar 18, 3:47 am)
Re: [PATCH 2/2] Implement a simple delta_base cache, Linus Torvalds, (Sat Mar 17, 7:09 pm)
Re: [PATCH 2/2] Implement a simple delta_base cache, Linus Torvalds, (Sat Mar 17, 7:54 pm)
Re: cleaner/better zlib sources?, Shawn O. Pearce, (Sat Mar 17, 1:19 am)
Re: cleaner/better zlib sources?, Matt Mackall, (Thu Mar 15, 9:14 pm)
Re: cleaner/better zlib sources?, Shawn O. Pearce, (Thu Mar 15, 9:10 pm)