login
Search
Hello, Im new to this forum. This post is related to debugging.
I have encoutered many times that problem with GCC:
Using optimization flags (-O1 .. -O3) introduces bugs in the program.
The bug itself does not come from GCC optimisations, but from a mathematical inconstitency in the source code. The problem then is for debugging. For small source code it is often easy to isolate the function(s) with bugs and rewrite them. With larger source code, and especially with complex data structures, it is not obvious where the bug comes from. It might be possible to isolate buggy functions but the problem might not come from them, rather from other initialisation, pre-process or post-process functions. Sometimes the wrong computation is due to a data structure inconsitency, which is really hard to debug with common debuggers: the data structure may hold exactely what data you expect it to hold.
To sum up, I am in the worst case: large source code, complex data structures, no obvious bugs neither in the functions nor in the datas held by the structure. The mathematical model the program relies on does not seem to have inconstitencies either.
Since the ouput of the program was correct and I would never find out that there was a bug without changing compilation options, one would say I am facing a shroedinger bug :)
My question is: anyone have tips to deal with that sort bugs? Any debugging method to rely on?
Dreaming of an -OEXPLAIN option for GCC, ala SQL ;)
Thanx
rand(...)
I fount out what was wrong.
I was using the rand(...) function to test with random data sets. When compiling with optimisations it changes the order the rand() function is called.
Eeew, side effects!
That should be considered a bug in GCC then, unless you were calling rand() more than once between sequence points. Then it's *your* bug.
For instance, I ran into that problem trying to port DOOM. In the code, they had used the following idiom multiple times: (I may have the exact names wrong, it's been a few years...)
They were relying on left-to-right evaluation to get consistent results, and the compiler I was using (not GCC) emitted the calls right-to-left. So, things like the prerecorded demo sequence just didn't work as expected. The ANSI C standard says you shouldn't rely on evaluation order between sequence points, and so this bug was DOOM's bug, not the compiler's.
The fix was to make a function (I called it something like P_randDiff) that called P_rand() twice, putting each into a temp, and then returned the difference in temps:
The magic was in introducing explicit sequence points.
printf is your friend
The gcc version of -OEXPLAIN is -S (assembler output) or "objdump -j .text -S" (show assembler code together with source code)... There is nothing else, which shows you exactly what happens.
Some machines' (most important: i386) floating point registers have "too much" precision, i.e. there it makes a difference, if the values are stored temporarily to a memory location (losing the excess precision) or not; optimization of course eliminates such temporary stores. Use gcc's -ffloat-store to switch this off, and don't use any option activated by -ffast-math to get predictable (i.e. same at every optimization level) results (but slower code).
Sometimes tiny differences result in something like the butterfly effect, you get one solution or another one, both valid. If you want to avoid this, you have to select algorithms, which are more robust.
One source of precision differences is the order in which you add a series of numbers; maybe you can force the compiler to not reorder your sums; and select a good order yourself. You could search for algorithmic instabilities, e.g. watch the condition numbers.
To check a program (this is valid for every program on earth...) you should have a set of understood or solved problems, where you can test, if the program's output matches your prior knowledge. If you can't compare the results with anything, you will not even notice your bugs (as you described). Test for invariants (e.g. sum of values must be zero). Note: the program itself is always correct, it exactly does, what you coded :)
To compare different program runs (e.g. only differing in -O-number) you can put printfs everywhere, print some number every couple of lines. Then diff the output (xxdiff is good for that). Round the numbers to not get overwhelmed by differences in insignificant digits (OR print every bit of data if these small differences are the problem later on), and if possible, don't print raw numbers, but aggregated ones, whichever make sense (e.g. if you deal with triangulated surfaces, you can compare the surface areas), to reduce the output. The different behaviour must start /somewhere/, so you can analyze what happens just before the first difference. The good thing about printf is, that you can still fully optimize your code between these calls; though the function calls of course stop the optimizer from reordering instructions around them...
Assertions are your friend
Not sure if you're code is C/C++ but if you can use assertions or something like them they can really help. For instance, #ifdef all your assertions based on some global flag, then assume certain standard input. Then you enable your assertions, give them the input you expect and only see printf's for the code that is incorrect. This makes it a lot easier then just putting printfs all over the place. If there's not built in assertion support, you can fairly easily code up your own.
Don't #ifdef your assertation
Don't #ifdef your assertations, it's already done for you! #define NDEBUG if you don't want the assertations compiled in. Also see man assert.
Or, just leave them in!
I'd benchmark with and without assertions, actually. Depending on what you're asserting, the compiler may actually generate better code *with* the assertions. You see, the assertions indicate true statements to the optimizer, which for things like loop trip counts, etc. may encourage it to unroll more (or less), identify and eliminate more dead code, etc.
Some compilers have a "happy medium" version of assert() that provides all the semantic information to the compiler, but doesn't generate a run-time check. On the TI DSP compiler, this goes by the name _nassert(). I don't know if GCC has a similar concept, but it's a useful one. You can convert assert() to _nassert() on that compiler with #define NASSERT instead of #define NDEBUG.
My question is: anyone have t
General regression and search. You have a program that behaves differently with optimization. You want to know where.
Try turning optimization on and off on selected parts of the program (using something like
#pragma optimization_level { 0 | 1 | 2 | 3 | reset } or #pragma optimize("", off)or whatever syntax your compiler uses. If your compiler doesn't let you do this for individual functions, you've got trouble.
See which part of the program is affected by the optimization. Then look at the assembly code or single-step the function in question.