The question was recently raised on the lkml why such a wide range of GCC [forum] versions were being actively supported by the Linux kernel. One reason offered was, "because the new compilers are a lot slower", an argument that left some perplexed. "Why is this an issue when compiling a kernel? How often do you compile your kernel?"
Linux creator Linus Torvalds agreed that the speed of the compiler was indeed one of the reasons that older versions are still supported, "for some people that is literally where _most_ of the CPU cycles go". Beyond that, he pointed out that earlier versions of the GCC 3.x compiler would generate worse code than 2.95.x, and that they were simply buggier. Linus explained, "for a _long_ time, the only reason to upgrade gcc was literally C++ support: basic C support was getting _worse_ with new compilers in pretty much every regard." He went on to add, "things seem to have improved a bit lately. The gcc-3.x series was basically not worth it for plain C until 3.3 or so."
From: Timothy Miller [email blocked] To: Linux Kernel Mailing List [email blocked] Subject: support of older compilers Date: Wed, 03 Nov 2004 16:02:49 -0500 I'm just curious about why there seems to be so much work going into supporting a wide range of GCC versions. If people are willing to download and compile a new kernel (and migrating from 2.4 to 2.6 is non-trivial for some systems, like RH9), why aren't they willing to also download and build a new compiler?
From: Christoph Hellwig [email blocked] Subject: Re: support of older compilers Date: Wed, 3 Nov 2004 21:13:53 +0000 On Wed, Nov 03, 2004 at 04:02:49PM -0500, Timothy Miller wrote: > I'm just curious about why there seems to be so much work going into > supporting a wide range of GCC versions. If people are willing to > download and compile a new kernel (and migrating from 2.4 to 2.6 is > non-trivial for some systems, like RH9), why aren't they willing to also > download and build a new compiler? Because the new compilers are a lot slower.
From: Adam Heath [email blocked] Subject: Re: support of older compilers Date: Wed, 3 Nov 2004 17:06:56 -0600 (CST) On Wed, 3 Nov 2004, Christoph Hellwig wrote: > On Wed, Nov 03, 2004 at 04:02:49PM -0500, Timothy Miller wrote: > > I'm just curious about why there seems to be so much work going into > > supporting a wide range of GCC versions. If people are willing to > > download and compile a new kernel (and migrating from 2.4 to 2.6 is > > non-trivial for some systems, like RH9), why aren't they willing to also > > download and build a new compiler? > > Because the new compilers are a lot slower. You can't be serious that this is a problem.
From: Chris Wedgwood [email blocked] Subject: Re: support of older compilers Date: Wed, 3 Nov 2004 15:30:29 -0800 On Wed, Nov 03, 2004 at 05:06:56PM -0600, Adam Heath wrote: > You can't be serious that this is a problem. try it, say gcc 2.95 vs gcc 4.0 ... i think last i checked the older gcc was over twice as fast
From: Adam Heath [email blocked] Subject: Re: support of older compilers Date: Thu, 4 Nov 2004 10:50:38 -0600 (CST) On Wed, 3 Nov 2004, Chris Wedgwood wrote: > On Wed, Nov 03, 2004 at 05:06:56PM -0600, Adam Heath wrote: > > > You can't be serious that this is a problem. > > try it, say gcc 2.95 vs gcc 4.0 ... i think last i checked the older > gcc was over twice as fast I didn't deny the speed difference of older and newer compilers. But why is this an issue when compiling a kernel? How often do you compile your kernel?
From: Linus Torvalds [email blocked] Subject: Re: support of older compilers Date: Thu, 4 Nov 2004 11:38:47 -0800 (PST) On Thu, 4 Nov 2004, Adam Heath wrote: > > I didn't deny the speed difference of older and newer compilers. > > But why is this an issue when compiling a kernel? How often do you compile > your kernel? First off, for some people that is literally where _most_ of the CPU cycles go. Second, it's not just that the compilers are slower. Historically, new gcc versions are: - slower - generate worse code - buggier For a _long_ time, the only reason to upgrade gcc was literally C++ support: basic C support was getting _worse_ with new compilers in pretty much every regard. Things seem to have improved a bit lately. The gcc-3.x series was basically not worth it for plain C until 3.3 or so. Linus
From: Adam Heath [email blocked] Subject: Re: support of older compilers Date: Thu, 4 Nov 2004 15:47:51 -0600 (CST) On Thu, 4 Nov 2004, Linus Torvalds wrote: > > I didn't deny the speed difference of older and newer compilers. > > > > But why is this an issue when compiling a kernel? How often do you compile > > your kernel? > > First off, for some people that is literally where _most_ of the CPU > cycles go. So find a fast machine. As I have already said, you don't need to compile a kernel for a slow machine/arch *on* a slow machine/arch. > Second, it's not just that the compilers are slower. Historically, new gcc > versions are: > - slower Again, that's a straw man. > - generate worse code > - buggier I don't doubt these are issues. That's not what I am discussing.
From: Linus Torvalds [email blocked] Subject: Re: support of older compilers Date: Thu, 4 Nov 2004 13:55:43 -0800 (PST) On Thu, 4 Nov 2004, Adam Heath wrote: > > > > First off, for some people that is literally where _most_ of the CPU > > cycles go. > > So find a fast machine. As I have already said, you don't need to compile a > kernel for a slow machine/arch *on* a slow machine/arch. I _have_ a fast machine. Others don't. And quite frankly, even I tend to prioritize things like "nice and quiet" over absolute speed. > I don't doubt these are issues. That's not what I am discussing. Sure it is. You're complaining that developers use old versions of gcc. They do so for a reason. Old versions of gcc are sometimes better. They are better in many ways. Your "use new versions of gcc even if it is slower" argument doesn't make any _sense_. If the new versions aren't any better, why would you want to use them? Linus
From: Adam Heath [email blocked] To: Linus Torvalds [email blocked] Subject: Re: support of older compilers Date: Thu, 4 Nov 2004 17:39:08 -0600 (CST) On Thu, 4 Nov 2004, Linus Torvalds wrote: > > > On Thu, 4 Nov 2004, Adam Heath wrote: > > > > > > First off, for some people that is literally where _most_ of the CPU > > > cycles go. > > > > So find a fast machine. As I have already said, you don't need to compile a > > kernel for a slow machine/arch *on* a slow machine/arch. > > I _have_ a fast machine. Others don't. And quite frankly, even I tend to > prioritize things like "nice and quiet" over absolute speed. > > > I don't doubt these are issues. That's not what I am discussing. > > Sure it is. You're complaining that developers use old versions of gcc. > They do so for a reason. Old versions of gcc are sometimes better. They > are better in many ways. Using an old version of gcc because it is faster at compiling is a non-argument. If people don't bother using newer compilers, complaining about their inefficiencies, then the issues will never be resolved. I have no problem with older gccs if they produce more correct code. > Your "use new versions of gcc even if it is slower" argument doesn't make > any _sense_. If the new versions aren't any better, why would you want to > use them? That's not my argument. Never has been. I am against people who say not to use newer gccs only on the grounds that they are slower. If they produce bad code, then that's a valid reason. If they produce larger code, that is a valid reason. But slowness doesn't mean wrong, just by being slow. ps: it seldom makes sense to use a single metric as a measure of the quality of some specific item in some specific situation.
From: Linus Torvalds [email blocked] Subject: Re: support of older compilers Date: Thu, 4 Nov 2004 15:52:21 -0800 (PST) On Thu, 4 Nov 2004, Adam Heath wrote: > > But slowness doesn't mean wrong, just by being slow. No. "Right" and "wrong" have _nothing_ to do with anything. he only thing that matters is "what is the best tool". And yes, performance _is_ an issue in selecting the best tool. It's not the only one, but it's a major one. You said so yourself when you claimed people should just buy faster hardware. Again, the machine you use is just another tool. Why should you buy a faster machine if performance doesn't matter? I don't understand why you first dismiss performance, and then go on to ignore all the _other_ issues I told you about too. And your argument about "things will get fixed if you use the newer version" is also not actually true. First off, if it ain't broke in the older version, then things _literally_ will get fixed by not upgrading in the first place. And telling a developer "I'm not using your new version because it sucks compared to the old one" is actually a perfectly valid thing to do, and is likely to be _more_ motivational for the developer to get it fixed than having users that just follow the newest version like stupid sheep. There are people out there using Linux-2.0. There are probably people even using linux-1.2. And hey, that's OK. For older machines it may really be the right choice, especially if they are still doing the same thing they did several years ago. The notion that you somehow "have to" upgrade to the newest version of software is BOGUS. Linus
GCC, whatev
I think the argument that I should upgrade GCC so that the developers can solve problems is hilarious. I love software, but I don't have time to be QA for every software project under the sun. No matter what kind of geek you are, at some point you have to be just a user. And that's me and the compiler. I couldn't give a rats rear as long as it get's my job done. I recompile the kernel on a right regular basis, compilation speed matters. "get new hardware" ain't the answer. I'm running the fastest machine I can afford. Why shouldn't I run the free solution that works, instead of the expensive solution? Isn't that one of Linux's (and GCC's) major draws? Quality on the cheap?
GCC's optimization for modern
GCC's optimization for modern x86 chips has gotten remarkably better since 2.95.x (I think for a certain benchmark on an athlon-xp chip the GCC team reported a 30% speedup between 2.95.3 and 3.0.x, which was otherwise a pretty buggy release). For a general purpose compiler that's pretty important, especially with all the comparisons GCC is getting to Intel's compiler these days. Free software needs to stay competitive.
However, as far as the kernel is concerned I'm not sure that the optimizer improvements matter that much. The kernel folks hand optimize the hell out of their code where it matters and generally try to make it as simple as possible for the compiler to generate fast code. So the latest and greatest GCC doesn't buy you much with the kernel (except slower compile times).
If you compile other packages regularly you probably should keep a modern GCC around. If the kernel is the only thing you ever build yourself, having GCC 2.95.3 won't hurt and will make your builds go a little quicker.
Compiled code quality
You said:
What perplexes me is that I've heard multiple times about how much better GCC 3.x's code generation is, but Linus seems to think otherwise. I wonder what the true story is.
I wouldn't be surprised if some of the paths in the kernel got slower, only because they were optimized at the C level to how old GCC generated code. Now, the newer GCC doesn't optimize the same way, so it ends up generating worse code. Fair benchmark? Not really.
true story?!?
you said you were wondering what the "true story is"
define "true"
define "story"
define "is"
what are you talking about?!? you replied a very well writen message, which was also self explanatory...
do you have problems understanding that the kernel is lower level (with more explicit optimization) and that it's writen in c?
do you understand the different between c and c++?
do you understand what a front-end is, what a back-end is, how come somebody goes like "oh, this is way slower" while somebody else goes like "oh, this is so much faster" and (read catch22) they're both right?
Daniel
I shouldn't even answer this troll.
Calm down. Take a Prozac!
FWIW, I'm quite familiar with all the phases of compilation, and I'm very comfortable with the difference between C and C++, thankyouverymuch.
My point is that we have a number of statements from multiple sources that superficially contradict each other. When I say I want to know what the "true story is," I want to know those missing details that make the whole picture make sense. Basically, what I'm curious about are the answers to these questions:
Did GCC 3.0 through 3.2.x produce worse quality code than 2.95? If so, on what platforms?
Is Linus' chief complaint with the quality of GCC's output, or the quality of the output relative to the cost of getting it? Linus might only be complaining that GCC compilation times and memory footprint both grew with no discernable difference in performance. Or, he may be saying that GCC's output suffered too. His exact complaint is not clear from the quoted thread.
If GCC is producing worse code for the kernel, but better code for everyone else, is it a function of how the kernel's been coded? That is, did people spend time making compiler-specific "voodoo tweaks" in the kernel source that no longer apply to the newer optimizer technology in GCC 3.x? Tweaks that may help some optimizers could throw off others.
Get the idea?
gcc-3.3.6-ss is solid as a rock!!!
* gcc-3.3.6-ss-20041103 compiles faster as gcc-2.95.4 does.
( ftp://gcc.gnu.org/pub/gcc/snapshots/3.3-20041103/ )
* gcc-3.3.6-ss-20041103 is solid as gcc-2.95.4 while that gcc-3.4.3 is stable.
* gcc-3.3.6-ss-20041103 generates better code ( -O2 | -Os ) (and still much better for 64 bit) than gcc-2.95.4 does.
* gcc-3.3.6-ss-20041103 supports long long that gcc-2.95.4 doesn't support it.
* gcc-3.3.6-ss-20041103 reports better the warnings & errors than gcc-2.95.4 does.
* gcc-3.3.6-ss-20041103 recognizes better the C++ language than gcc-2.95.4 does.
open4free ©
FUD
gcc 2.95.3 had perfect long long support on ILP32 and LP64
arches under OpenBSD and MirOS.
Tracking gcc performance
There are several places which track the performance of gcc compiled code as a function of time. Here are SPEC results.
AMD64 optimization
So if you have some AMD64 CPU it is better to use the newer GCC which does more optimization for the new hardware?
since 2.9.x doesn't support AMD64
you don't have much choice there, do you?
Compile time, not optimization
If I understand the discussion correctly, the matter at hand is how long GCC takes to compile the code, not how fast the resulting code is.
Having better optimization capability, but a longer compile time is intolerable for some (many?) circumstances, such as the slower machines referenced in the LKML thread. If a person only has one machine, and it is older and slower, and needs to compile a newer kernel, the resulting load the compiling can place upon the machine can make it unusable for its intended purpose.
What some (not necessarily you) people forget is their circumstances do not equal everyone elses circumstances.
I am curious.
Then why still gcc being developped??
But,IMHO, i am sure there is a reason for the new gccs.
i guess that gcc developers are doing various experiments.
from this article, also i guess that kernel developer and gcc developer seldom talk each other. :)
--
Both, actually.
Actually, Linus was complaining about both compile time and compiled code performance, from what I could tell. Basically, he seemed to be saying GCC 3.x was a step backwards for C support. Not only was GCC slower, but it produced slower output.
For people who build their kernels infrequently, it shouldn't matter too much that GCC's slow. But for people who edit-build-debug-edit-build-debug, it can really matter.
Maybe they should look into TCCBOOT to handle those cases, and fall back to GCC only when you've converged on something you think's stable.
CCACHE
In that situation, something like ccache is extremely useful.
Hm
If your build system works correctly--ie, doesn't rebuild modules that don't need to be rebuilt--I don't see how there's any need for ccache.
header files
Imagine fixing a typo in one headerfile which happens to be included almost everywhere.
Ccache does not fix that
If you mean a typo in the code, everything including that header *must* be recompiled. That's the end.
If you instead mean a typo in the comments, since ccache works on preprocessed source, then you are right, ccache will avoid rebuilding everything.
About the "build system works", 2.4 kbuild is known not to always work (experienced myself multiple times). Also, arch/um (i.e. UserModeLinux) still suffers from the build system problems (i.e. requires a make clean after changing the configuration) even in 2.6 (I'm working at fixing that, but it's not easy at all).
The only thing I should add is that you can configure ccache to recognize when a change is only in the amount of whitespace, and it then understands that it does not need recompilation.
-- Blaisorblade (UML developer)
yes,
I meant a typo in a comment.
Since when do you need to rec
Since when do you need to recompile after fixing a comment??
You don't, at least right away.
You don't need to immediately. Here's the scenario, though, where it matters: Files A, B, C, D, E, F, G all include header H. You fix comment in header H, and tweak actual code in file B. Now type "make." Which files get rebuilt?
You'd like to recompile only B and then relink. But, because all 7 files include header H, Make will rebuilt all 7 by default.
Scale this up to the kernel. Now suppose you fix a typo in some central header, like "linux/types.h". Have fun watching half your kernel recompile on the next Make.
TCCBOOT is just a toy
I've read the homepage the other day - and the author himself says "there are probably quite a few other subtle bugs" and "it supports only against 2.4.27 currently" (i.e. the kernel patch to avoid it shaking out too many TinyCC bugs is only against 2.4.27).
About GCC 3.x, I think nobody can say that the resulting output is always slower. Actually, Linus says "generate worse code", but in most cases, gcc 3.0 is faster than other versions.
But *very* buggier.
In fact, many distro shipped 2.95.x, even 2.96 (which was indeed a buggy beta used by RH, in fact older RH used a gcc 2.95.x to compile the kernel), and many one used 3.2 and onwards. Nobody actually used versions in the middle as the default compiler.
well, then
If this isn't the case already, GCC should be made flexible enough to toggle between "high cpu-load + expensive optimizations" and "low cpu-load + inexpensive optimizations". This would allow people for whom compile time is the chief issue to disable the cpu-hog functionality, while "everybody else" (showing my bias) can have kernels built using the latest & greatest gcc features.
To what extent can the latest GCC be tweaked to improve compile-time at the expense of code optimization? Can't most of the expensive optimizations be toggled off piecemeal?
RE: well, then
The optimization levels change the binary, and can introduce different bugs. A developer building at low-optimization level and testing does nothing to help people building at high-optimization level and using the software.
Kinda...
Assume the optimizer itself is not broken. For most cases, this is a reasonable assumption, but obviously not always true. If you change optimization levels and that changes bugs or exposes bugs in your program, then you likely have code that isn't valid C, or doesn't mean what you think it means (e.g. forgetting 'volatile' on something). These are subtle bugs that don't jump out at you right away. They're also the bugs you tend to tackle last.
Most bugs are logic bugs. You simply have a wrong program in the first place. You forgot to take a lock when you needed to. You used the wrong interface to some functionality. You screwed up a linked-list manipulation. You called a function that sleeps while you held BKL. Optimization doesn't change these bugs or their behavior most of the time. Therefore, it makes sense to get rid of these while optimization is still off.
While you clean out the logic bugs, you'll often notice and fix the semantic bugs--the stuff that would get broken by the optimizer. Once your code appears logically correct, it makes sense then to turn on optimization and flush out the remaining bugs, if/where they exist. I've been programming C for a dozen years now, and I still slip occasionally and forget a 'volatile' here or play fast-and-loose with side effects there.
Nothing requires you to build with optimization on all the time. In fact, for some projects, it may make the most sense to build with optimization off while you're sorting out your logic bugs. The subtle issues that the optimizer might expose may get in the way of you debugging your logic!
If anything, using a variety of compilers and optimization levels should bring out all the semantic bugs. Saying "building at low-optimization level and testing does nothing to help people building at high-optimization level and using the software" overstates things. It also overlooks the benefit that testing with a variety of optimization levels can bring to the stability of the software.
--Joe
Huh?
GCC has four preset levels of optimization (-O, -O2, -O3, -Os) and many dozens of flags to enable/disable the various individual optimizations that the engine can perform. Read the GCC documentation for details. If compile time is a concern use either -O or no optimization (-O0).
The kernel won't work that way
If optimization level is not an option they offer, there is a reason.
Actually, it is likely that newer GCC version solve this particular issue, but older one never do any inlining if optimizations are not enabled.
Now, in the kernel there are quite a few places where disabling inlining unvariably causes compilation to fail. For instance, from include/asm-i386/fixmap.h (IIRC):
if (param1 satisifes a condition that should never happen, and that is known compile-time) {
__this_fixmap_does_not_exist();
}
That function, indeed, does not exist, so things would break.
When optimizing, instead, if the calling code is correct, param1 is known compile-time and assumes a valid value, so GCC will know at compile-time that the above condition is never satisifed, and omits the call to __this_fixmap_does_not_exist().
Now, actually some gcc can inline even with -O0 (do "grep always_inline include/linux/compiler-gcc*.h" to find out which ones). But other issues can still come across.
GCC v3.x
The statement that GCC 3.x generates worse code than the 2.95 series may be specific to certain architectures. GCC has recently had a number of changes in the register allocator and the new SSA stuff (coming in 4.0).
My experience has been that on some processor GCC 3.x generates considerably faster code (m68k embedded, low-level kernel type stuff). Although the way the Linux kernel is written it tends to have many places where memory barriers are inserted that cause the optimizer to somewhat stop in its tracks and re-think things.
Amusingly enough, GCC v2.95 does not support __builtin_expect. All of the if (unlikely(xxx)) stuff in the kernel is not enabled on older GCC and the branch prediction hints are pretty important in PPC (Linus' principle architecture).
So either older GCC makes very good gueses on branch hints or something else is throwing the performance off on the kernel (for PPC).
- myg
GCC
Just because GCC was historially buggy does not mean new version will be buggy.
You can not compare GCC 2.95 against GCC 4.0. GCC 4.0 is still in development, It's like comparing a mature software against software that is being developed, obviously mature software is better.
For the record, I have downloaded and compiled GCC 3.4.3 from the source code.
Still in dev, good bet it is still buggy.
Looks like most of the people making comments missed the RH fiasco when it started shipping the gcc 3.x complier instead of the 2.95. At the time if you tried to compile kernel it _broke_ the kernel. This was teh situation until late in the 3.2.X series. Gcc 3x had good C++ support but broke a lot of C stuff and did wierd things when trying to optimize C code. There are still cases in the kernel where runtime issues can be traced directly back to it being compiled with a gcc 3.x instead of 2.95, Yes, some of this is issues in the the kernel code, Linus has said as much in the past, but most of it has been in the way Gcc handle branch prediction and/or dealt with the C/asm structure of the kernel.
Actually, it wasn't 3.x that
Actually, it wasn't 3.x that Redhat (and a few others) shipped - it was 2.96, which was some hybrid of 2.95 with 3.x features backported.
2.96
No, RH's 2.96 was a snapshot of GCC CVS mainline at some point during the 3.0 development cycle that was stabilized and shipped by RH. It wasn't some "hybrid", it was just a development snapshot of 3.0
This is not entirely true. Re
This is not entirely true. Red Hat's "2.96" GCC (from prerelease 3.x CVS) that they shipped with RHL 7.x did indeed break the kernel, but official GCC releases were not such a problem. I compiled many kernels with 3.0.x up through every stable release without any major problems that I can recall. If I remember correctly, the GCC release team uses successful kernel compilation as one of the release criteria.
maybe just use the best gcc for the job?
i use various gccs. (patched) gcc 2.95.2 for compiling kernels for the arm and gcc 2.95.3 and gcc 3.3.4 for x86. isn't free software about choice?
as long as i can use the best gcc there is for the job it's ok, if i can use worse ones it's even better.
the only argument i see here is "i have this new shining linux distro, and it's gcc will not compile the kernel correctly" - please complain to (in descending order):
- the distro makers, a separare compiler for building the kernel is a perfectly valid concept (as long as it's not a reason to include a broken compiler for userspace apps - see the redhat kgcc problems)
- the gcc authors (hey - some properly written kernel code worked well withouth hackery berore, why should it not work now ?)
- the kernel authors (hey - your code has so much hacks, that the newest gcc optimisation renders it nonworkable, use the -O0 for newer gccs)
if you are forced to exhibit some behaviour that prohibits you from using the most tested gcc on whatever you're trying to build, then maybe just chainging the distro would solve the problem?
New versus old
What I think Linus wants to say is simply - as long as the new is not better than the old, why use it. That is, the group producing the "new" has lost the ball, and should do some "deep" thinking about it.
In the commercial world it is different.
MS could ask (and force) consumers to accept a new version of Windows, say "Shorthorn" because it is good and important for MS, even if the new version of Windows had no good or inportant features for the user.
But that is an other matter.
So skip the "old" and the "new" words and the question is why should
I start to use an unsuperior product of something or anything for that matter.