"It looks to be about 2.1% increase in time to do the make/mount/unmount operations with the marker patches in place and no blktrace operations," Alan Brunelle summarized some benchmarks testing the overhead of the kernel markers patches. He continued, "with the blktrace operations in place we see about a 3.8% decrease in time to do the same ops." Block layer maintainer Jens Axboe responded favorably, "thanks for running these numbers. I don't think you have to bother with it more. My main concern was a performance regression, increasing the overhead of running blktrace." He added, "I'd say the above is Good Enough for me," acking the kernel marker patches.
Jens went on to muse, "I do wonder about that performance _increase_ with blktrace enabled. I remember that we have seen and discussed something like this before, it's still a puzzle to me..." Mathieu Desnoyers agreed, "interesting question indeed," going on to suggest possible future tests to understand the unexpected performance increase. blktrace is a block layer IO tracing tool for providing detailed information about request queue operations, originally developed by Jens Axboe and merged into the mainline kernel in 2.6.17-rc1.
From: Alan D. Brunelle <Alan.Brunelle@...> Subject: Re: Linux Kernel Markers - performance characterization with large IO load on large-ish system Date: Sep 25, 10:58 am 2007 Taking Linux 2.6.23-rc6 + 2.6.23-rc6-mm1 as a basis, I took some sample runs of the following on both it and after applying Mathieu Desnoyers 11-patch sequence (19 September 2007). * 32-way IA64 + 132GiB + 10 FC adapters + 10 HP MSA 1000s (one 72GiB volume per MSA used) * 10 runs with each configuration, averages shown below o 2.6.23-rc6 + 2.6.23-rc6-mm1 without blktrace running o 2.6.23-rc6 + 2.6.23-rc6-mm1 with blktrace running o 2.6.23-rc6 + 2.6.23-rc6-mm1 + markers without blktrace running o 2.6.23-rc6 + 2.6.23-rc6-mm1 + markers with blktrace running * A run consists of doing the following in parallel: o Make an ext3 FS on each of the 10 volumes o Mount & unmount each volume + The unmounting generates a tremendous amount of writes to the disks - thus stressing the intended storage devices (10 volumes) plus the separate volume for all the blktrace data (when blk tracing is enabled). + Note the times reported below only cover the make/mount/unmount time - the actual blktrace runs extended beyond the times measured (took quite a while for the blk trace data to be output). We're only concerned with the impact on the "application" performance in this instance. Results are: Kernel w/out BT STDDEV w/ BT STDDEV ------------------------------------- --------- ------ --------- ------ 2.6.23-rc6 + 2.6.23-rc6-mm1 14.679982 0.34 27.754796 2.09 2.6.23-rc6 + 2.6.23-rc6-mm1 + markers 14.993041 0.59 26.694993 3.23 It looks to be about 2.1% increase in time to do the make/mount/unmount operations with the marker patches in place and no blktrace operations. With the blktrace operations in place we see about a 3.8% decrease in time to do the same ops. When our Oracle benchmarking machine frees up, and when the marker/blktrace patches are more stable, we'll try to get some "real" Oracle benchmark runs done to gage the impact of the markers changes to performance... Alan D. Brunelle Hewlett-Packard / Open Source and Linux Organization / Scalability and Performance Group -
From: Mathieu Desnoyers <mathieu.desnoyers@...> Subject: Re: Linux Kernel Markers - performance characterization with large IO load on large-ish system Date: Sep 25, 1:13 pm 2007 * Alan D. Brunelle (Alan.Brunelle@hp.com) wrote: > Taking Linux 2.6.23-rc6 + 2.6.23-rc6-mm1 as a basis, I took some sample > runs of the following on both it and after applying Mathieu Desnoyers > 11-patch sequence (19 September 2007). > > * 32-way IA64 + 132GiB + 10 FC adapters + 10 HP MSA 1000s (one 72GiB > volume per MSA used) > > * 10 runs with each configuration, averages shown below > o 2.6.23-rc6 + 2.6.23-rc6-mm1 without blktrace running > o 2.6.23-rc6 + 2.6.23-rc6-mm1 with blktrace running > o 2.6.23-rc6 + 2.6.23-rc6-mm1 + markers without blktrace running > o 2.6.23-rc6 + 2.6.23-rc6-mm1 + markers with blktrace running > > * A run consists of doing the following in parallel: > o Make an ext3 FS on each of the 10 volumes > o Mount & unmount each volume > + The unmounting generates a tremendous amount of writes > to the disks - thus stressing the intended storage > devices (10 volumes) plus the separate volume for all > the blktrace data (when blk tracing is enabled). > + Note the times reported below only cover the > make/mount/unmount time - the actual blktrace runs > extended beyond the times measured (took quite a while > for the blk trace data to be output). We're only > concerned with the impact on the "application" > performance in this instance. > > Results are: > > Kernel w/out BT STDDEV w/ BT STDDEV > ------------------------------------- --------- ------ --------- ------ > 2.6.23-rc6 + 2.6.23-rc6-mm1 14.679982 0.34 27.754796 2.09 > 2.6.23-rc6 + 2.6.23-rc6-mm1 + markers 14.993041 0.59 26.694993 3.23 > Interesting results, although we cannot say any of the solutions has much impact due to the std dev. Also, it could be interesting to add the "blktrace compiled out" as a base line. Thanks for running those tests, Mathieu > It looks to be about 2.1% increase in time to do the make/mount/unmount > operations with the marker patches in place and no blktrace operations. > With the blktrace operations in place we see about a 3.8% decrease in > time to do the same ops. > > When our Oracle benchmarking machine frees up, and when the > marker/blktrace patches are more stable, we'll try to get some "real" > Oracle benchmark runs done to gage the impact of the markers changes to > performance... > > Alan D. Brunelle > Hewlett-Packard / Open Source and Linux Organization / Scalability and > Performance Group > -- Mathieu Desnoyers Computer Engineering Ph.D. Student, Ecole Polytechnique de Montreal OpenPGP key fingerprint: 8CD5 52C3 8E3C 4140 715F BA06 3F25 A8FE 3BAE 9A68 -
From: Alan D. Brunelle <Alan.Brunelle@...> Subject: Re: Linux Kernel Markers - performance characterization with large IO load on large-ish system Date: Sep 26, 11:28 am 2007 Mathieu Desnoyers wrote: > * Alan D. Brunelle (Alan.Brunelle@hp.com) wrote: >> Taking Linux 2.6.23-rc6 + 2.6.23-rc6-mm1 as a basis, I took some sample >> runs of the following on both it and after applying Mathieu Desnoyers >> 11-patch sequence (19 September 2007). >> >> * 32-way IA64 + 132GiB + 10 FC adapters + 10 HP MSA 1000s (one 72GiB >> volume per MSA used) >> >> * 10 runs with each configuration, averages shown below >> o 2.6.23-rc6 + 2.6.23-rc6-mm1 without blktrace running >> o 2.6.23-rc6 + 2.6.23-rc6-mm1 with blktrace running >> o 2.6.23-rc6 + 2.6.23-rc6-mm1 + markers without blktrace running >> o 2.6.23-rc6 + 2.6.23-rc6-mm1 + markers with blktrace running >> >> * A run consists of doing the following in parallel: >> o Make an ext3 FS on each of the 10 volumes >> o Mount & unmount each volume >> + The unmounting generates a tremendous amount of writes >> to the disks - thus stressing the intended storage >> devices (10 volumes) plus the separate volume for all >> the blktrace data (when blk tracing is enabled). >> + Note the times reported below only cover the >> make/mount/unmount time - the actual blktrace runs >> extended beyond the times measured (took quite a while >> for the blk trace data to be output). We're only >> concerned with the impact on the "application" >> performance in this instance. >> >> Results are: >> >> Kernel w/out BT STDDEV w/ BT STDDEV >> ------------------------------------- --------- ------ --------- ------ >> 2.6.23-rc6 + 2.6.23-rc6-mm1 14.679982 0.34 27.754796 2.09 >> 2.6.23-rc6 + 2.6.23-rc6-mm1 + markers 14.993041 0.59 26.694993 3.23 >> > > Interesting results, although we cannot say any of the solutions has much > impact due to the std dev. > > Also, it could be interesting to add the "blktrace compiled out" as a > base line. > > Thanks for running those tests, > > Mathieu Mathieu: Here are the results from 6 different kernels (including ones with blktrace not configured in), with now performing 40 runs per kernel. o All kernels start off with Linux 2.6.23-rc6 + 2.6.23-rc6-mm1 o '- bt cfg' or '+ bt cfg' means a kernel without or with blktrace configured respectively. o '- markers' or '+ markers' means a kernel without or with the 11-patch marker series respectively. 38 runs without blk traces being captured (dropped hi/lo value from 40 runs) Kernel Options Min val Avg val Max val Std Dev ------------------ --------- --------- --------- --------- - markers - bt cfg 15.349127 16.169459 16.372980 0.184417 + markers - bt cfg 15.280382 16.202398 16.409257 0.191861 - markers + bt cfg 14.464366 14.754347 16.052306 0.463665 + markers + bt cfg 14.421765 14.644406 15.690871 0.233885 38 runs with blk traces being captured (dropped hi/lo value from 40 runs) Kernel Options Min val Avg val Max val Std Dev ------------------ --------- --------- --------- --------- - markers + bt cfg 24.675859 28.480446 32.571484 1.713603 + markers + bt cfg 18.713280 27.054927 31.684325 2.857186 o It is not at all clear why running without blk trace configured into the kernel runs slower than with blk trace configured in. (9.6 to 10.6% reduction) o The data is still not conclusive with respect to whether the marker patches change performance characteristics when we're not gathering traces. It appears that any change in performance is minimal at worst for this test. o The data so far still doesn't conclusively show a win in this case even when we are capturing traces, although, the average certainly seems to be in its favor. One concern that I should be able to deal easily with is the choice of the IO scheduler being used for both the volume being used to perform the test on, as well as the one used for storing blk traces (when enabled). Right now I was using the default CFQ, when perhaps NOOP or DEADLINE would be a better choice. If there is enough interest in seeing how that changes things I could try to get some runs in later this week. Alan D. Brunelle Hewlett-Packard / Open Source and Linux Organization / Scalability and Performance Group -
From: Jens Axboe <jens.axboe@...> Subject: Re: Linux Kernel Markers - performance characterization with large IO load on large-ish system Date: Oct 2, 8:21 am 2007 On Wed, Sep 26 2007, Alan D. Brunelle wrote: > Mathieu Desnoyers wrote: >> * Alan D. Brunelle (Alan.Brunelle@hp.com) wrote: >>> Taking Linux 2.6.23-rc6 + 2.6.23-rc6-mm1 as a basis, I took some sample >>> runs of the following on both it and after applying Mathieu Desnoyers >>> 11-patch sequence (19 September 2007). >>> >>> * 32-way IA64 + 132GiB + 10 FC adapters + 10 HP MSA 1000s (one 72GiB >>> volume per MSA used) >>> >>> * 10 runs with each configuration, averages shown below >>> o 2.6.23-rc6 + 2.6.23-rc6-mm1 without blktrace running >>> o 2.6.23-rc6 + 2.6.23-rc6-mm1 with blktrace running >>> o 2.6.23-rc6 + 2.6.23-rc6-mm1 + markers without blktrace running >>> o 2.6.23-rc6 + 2.6.23-rc6-mm1 + markers with blktrace running >>> >>> * A run consists of doing the following in parallel: >>> o Make an ext3 FS on each of the 10 volumes >>> o Mount & unmount each volume >>> + The unmounting generates a tremendous amount of writes >>> to the disks - thus stressing the intended storage >>> devices (10 volumes) plus the separate volume for all >>> the blktrace data (when blk tracing is enabled). >>> + Note the times reported below only cover the >>> make/mount/unmount time - the actual blktrace runs >>> extended beyond the times measured (took quite a while >>> for the blk trace data to be output). We're only >>> concerned with the impact on the "application" >>> performance in this instance. >>> >>> Results are: >>> >>> Kernel w/out BT STDDEV w/ BT >>> STDDEV >>> ------------------------------------- --------- ------ --------- >>> ------ >>> 2.6.23-rc6 + 2.6.23-rc6-mm1 14.679982 0.34 27.754796 >>> 2.09 >>> 2.6.23-rc6 + 2.6.23-rc6-mm1 + markers 14.993041 0.59 26.694993 >>> 3.23 >>> >> >> Interesting results, although we cannot say any of the solutions has much >> impact due to the std dev. >> >> Also, it could be interesting to add the "blktrace compiled out" as a >> base line. >> >> Thanks for running those tests, >> >> Mathieu > Mathieu: > > Here are the results from 6 different kernels (including ones with blktrace > not configured in), with now performing 40 runs per kernel. > > o All kernels start off with Linux 2.6.23-rc6 + 2.6.23-rc6-mm1 > > o '- bt cfg' or '+ bt cfg' means a kernel without or with blktrace > configured respectively. > > o '- markers' or '+ markers' means a kernel without or with the 11-patch > marker series respectively. > > 38 runs without blk traces being captured (dropped hi/lo value from 40 > runs) > > Kernel Options Min val Avg val Max val Std Dev > ------------------ --------- --------- --------- --------- > - markers - bt cfg 15.349127 16.169459 16.372980 0.184417 > + markers - bt cfg 15.280382 16.202398 16.409257 0.191861 > > - markers + bt cfg 14.464366 14.754347 16.052306 0.463665 > + markers + bt cfg 14.421765 14.644406 15.690871 0.233885 > > 38 runs with blk traces being captured (dropped hi/lo value from 40 runs) > > Kernel Options Min val Avg val Max val Std Dev > ------------------ --------- --------- --------- --------- > - markers + bt cfg 24.675859 28.480446 32.571484 1.713603 > + markers + bt cfg 18.713280 27.054927 31.684325 2.857186 > > o It is not at all clear why running without blk trace configured into > the kernel runs slower than with blk trace configured in. (9.6 to 10.6% > reduction) > o The data is still not conclusive with respect to whether the marker > patches change performance characteristics when we're not gathering traces. > It appears > that any change in performance is minimal at worst for this test. > o The data so far still doesn't conclusively show a win in this case > even when we are capturing traces, although, the average certainly seems to > be in its favor. > One concern that I should be able to deal easily with is the choice of > the IO scheduler being used for both the volume being used to perform the > test on, as well as the one used for storing blk traces (when enabled). > Right now I was using the default CFQ, when perhaps NOOP or DEADLINE would > be a better choice. If there is enough interest in seeing how that changes > things I could try to get some runs in later this week. Alan, Thanks for running these numbers as well. I don't think you have to bother with it more. My main concern was a performance regression, increasing the overhead of running blktrace. So while we (well, you :-)) could run more tests, I'd say the above is Good Enough for me. Mathieu, you can add my Acked-by: Jens Axboe <jens.axboe@oracle.com> to the blktrace part of your marker series. I do wonder about that performance _increase_ with blktrace enabled. I remember that we have seen and discussed something like this before, it's still a puzzle to me... -- Jens Axboe -
From: Mathieu Desnoyers <mathieu.desnoyers@...> Subject: Re: Linux Kernel Markers - performance characterization with large IO load on large-ish system Date: Oct 2, 8:48 am 2007 * Jens Axboe (jens.axboe@oracle.com) wrote: > On Wed, Sep 26 2007, Alan D. Brunelle wrote: > > Mathieu Desnoyers wrote: > >> * Alan D. Brunelle (Alan.Brunelle@hp.com) wrote: > >>> Taking Linux 2.6.23-rc6 + 2.6.23-rc6-mm1 as a basis, I took some sample > >>> runs of the following on both it and after applying Mathieu Desnoyers > >>> 11-patch sequence (19 September 2007). > >>> > >>> * 32-way IA64 + 132GiB + 10 FC adapters + 10 HP MSA 1000s (one 72GiB > >>> volume per MSA used) > >>> > >>> * 10 runs with each configuration, averages shown below > >>> o 2.6.23-rc6 + 2.6.23-rc6-mm1 without blktrace running > >>> o 2.6.23-rc6 + 2.6.23-rc6-mm1 with blktrace running > >>> o 2.6.23-rc6 + 2.6.23-rc6-mm1 + markers without blktrace running > >>> o 2.6.23-rc6 + 2.6.23-rc6-mm1 + markers with blktrace running > >>> > >>> * A run consists of doing the following in parallel: > >>> o Make an ext3 FS on each of the 10 volumes > >>> o Mount & unmount each volume > >>> + The unmounting generates a tremendous amount of writes > >>> to the disks - thus stressing the intended storage > >>> devices (10 volumes) plus the separate volume for all > >>> the blktrace data (when blk tracing is enabled). > >>> + Note the times reported below only cover the > >>> make/mount/unmount time - the actual blktrace runs > >>> extended beyond the times measured (took quite a while > >>> for the blk trace data to be output). We're only > >>> concerned with the impact on the "application" > >>> performance in this instance. > >>> > >>> Results are: > >>> > >>> Kernel w/out BT STDDEV w/ BT > >>> STDDEV > >>> ------------------------------------- --------- ------ --------- > >>> ------ > >>> 2.6.23-rc6 + 2.6.23-rc6-mm1 14.679982 0.34 27.754796 > >>> 2.09 > >>> 2.6.23-rc6 + 2.6.23-rc6-mm1 + markers 14.993041 0.59 26.694993 > >>> 3.23 > >>> > >> > >> Interesting results, although we cannot say any of the solutions has much > >> impact due to the std dev. > >> > >> Also, it could be interesting to add the "blktrace compiled out" as a > >> base line. > >> > >> Thanks for running those tests, > >> > >> Mathieu > > Mathieu: > > > > Here are the results from 6 different kernels (including ones with blktrace > > not configured in), with now performing 40 runs per kernel. > > > > o All kernels start off with Linux 2.6.23-rc6 + 2.6.23-rc6-mm1 > > > > o '- bt cfg' or '+ bt cfg' means a kernel without or with blktrace > > configured respectively. > > > > o '- markers' or '+ markers' means a kernel without or with the 11-patch > > marker series respectively. > > > > 38 runs without blk traces being captured (dropped hi/lo value from 40 > > runs) > > > > Kernel Options Min val Avg val Max val Std Dev > > ------------------ --------- --------- --------- --------- > > - markers - bt cfg 15.349127 16.169459 16.372980 0.184417 > > + markers - bt cfg 15.280382 16.202398 16.409257 0.191861 > > > > - markers + bt cfg 14.464366 14.754347 16.052306 0.463665 > > + markers + bt cfg 14.421765 14.644406 15.690871 0.233885 > > > > 38 runs with blk traces being captured (dropped hi/lo value from 40 runs) > > > > Kernel Options Min val Avg val Max val Std Dev > > ------------------ --------- --------- --------- --------- > > - markers + bt cfg 24.675859 28.480446 32.571484 1.713603 > > + markers + bt cfg 18.713280 27.054927 31.684325 2.857186 > > > > o It is not at all clear why running without blk trace configured into > > the kernel runs slower than with blk trace configured in. (9.6 to 10.6% > > reduction) > > o The data is still not conclusive with respect to whether the marker > > patches change performance characteristics when we're not gathering traces. > > It appears > > that any change in performance is minimal at worst for this test. > > o The data so far still doesn't conclusively show a win in this case > > even when we are capturing traces, although, the average certainly seems to > > be in its favor. > > One concern that I should be able to deal easily with is the choice of > > the IO scheduler being used for both the volume being used to perform the > > test on, as well as the one used for storing blk traces (when enabled). > > Right now I was using the default CFQ, when perhaps NOOP or DEADLINE would > > be a better choice. If there is enough interest in seeing how that changes > > things I could try to get some runs in later this week. > > Alan, > > Thanks for running these numbers as well. I don't think you have to > bother with it more. My main concern was a performance regression, > increasing the overhead of running blktrace. So while we (well, you :-)) > could run more tests, I'd say the above is Good Enough for me. Mathieu, > you can add my Acked-by: Jens Axboe <jens.axboe@oracle.com> to the > blktrace part of your marker series. > thanks! > I do wonder about that performance _increase_ with blktrace enabled. I > remember that we have seen and discussed something like this before, > it's still a puzzle to me... > Interesting question indeed. In those tests, when blktrace is running, are the relay buffers only written to or they are also read ? Running the tests without consuming the buffers (in overwrite mode) would tell us more about the nature of the disturbance causing the performance increase. Also, a kernel trace could help us understand more thoroughly what is happening there.. is it caused by the scheduler ? memory allocation ? data cache alignment ? I would suggest that you try aligning the block layer data structures accessed by blktrace on L2 cacheline size and compare the results (when blktrace is disabled). Mathieu > -- > Jens Axboe > -- Mathieu Desnoyers Computer Engineering Ph.D. Student, Ecole Polytechnique de Montreal OpenPGP key fingerprint: 8CD5 52C3 8E3C 4140 715F BA06 3F25 A8FE 3BAE 9A68 -
From: Alan D. Brunelle <Alan.Brunelle@...> Subject: Re: Linux Kernel Markers - performance characterization with large IO load on large-ish system Date: Oct 2, 1:51 pm 2007 Mathieu Desnoyers wrote: >> remember that we have seen and discussed something like this before, >> it's still a puzzle to me... >> >> > I do wonder about that performance _increase_ with blktrace enabled. I > > Interesting question indeed. > > In those tests, when blktrace is running, are the relay buffers only > written to or they are also read ? > blktrace (the utility) was running too - so the relay buffere /were/ being read and stored out to disk elsewhere. > Running the tests without consuming the buffers (in overwrite mode) > would tell us more about the nature of the disturbance causing the > performance increase. > I'd have to write a utility to enable the traces, but then not read. Let me think about that. > Also, a kernel trace could help us understand more thoroughly what is > happening there.. is it caused by the scheduler ? memory allocation ? > data cache alignment ? > Yep - when I get some time, I'll look into that. [Clearly not a gating issue for marker support...] > I would suggest that you try aligning the block layer data structures > accessed by blktrace on L2 cacheline size and compare the results (when > blktrace is disabled). > Again, when I get some time! :-) Alan -
Still too much
A 2.1% increase in time todo something, is still 2.1% too long, or 2.1% too much, or an additional 2.1%.
Make it faster, not slower.
I'm sure you'd notice it...
...because you'll only be able mkfs/mount/unmount your filesystems 980 times in the same time you could have done 1000. Never mind the fact that the shift was within 1 standard deviation, so it's practically within the margin of error measurement-wise. That is to say, on a larger set of machines and a larger set of runs, the odds are that the shift is smaller than that.
Now, a 2.1% system wide slowdown on everything? That's rather noticeable. 1% here, 2% there system wide adds up very quickly. A 2.1% slowdown on a micro-benchmark designed to overstate performance shifts on a specific path? That's exceptionally minor. How often do you mkfs / mount / unmount? Gotta keep the context in front of you.
--
Program Intellivision and play Space Patrol!
Please remove it
Newbie:
I prefer a faster kernel.
It slows down performance so please give it a bad review.
Newbie: I prefer a faster
Newbie:
I prefer a faster kernel.
It slows down performance so please give it a bad review.
Brilliant! What a fantastic parody of all the ricer newbie retards that seem to think their opinions are important even though they do not even begin to understand the subject. There sure have been an awful lot of kiddies coming out of the woodwork recently and I'm glad I'm not the only person who is annoyed by this. I used Linux for years before I would have dared to post my opinions about it and expect them to be given any attention, but this attitude seems to have gone right out of fashion. I'm all in favour of being friendly to newbies, but when they think that they should be able to steer the direction in which Linux is moving without having contributed anything, they should be soundly mocked.
The kind of person you are impersonating would definitely be running Reiser4 and using a CK patched kernel on Gentoo with all the unstable packages unmasked. Hopefully they will soon lose all of their data and move back to Windows where they belong.
You completely missed the point
Part of the reason kernel developers are increasing instrumentation actually is making it easier to track down kernel bottlenecks in the exact setup where they occur. I would think that once this framework has stabilized and matured, the improvements will more than make up for the lost 2.3% — in addition to facilitating finding and fixing more fatal bugs.
Of course you are always welcome to compile your own kernel with your own configuration.
It is really naive to believe that any kernel developer would be swayed by a comment on KernelTrap with zero substantiation.
Insignificant
I don't care about 2.1%
If I run into problems caused by "2.1%" i suggest buying new hardware!
I care about a stable kernel.
If kernel markers help to get a even more robust kernel, or help to keep a robust release most of the time:
hell, i want more of it!
Btw. aren't they optional when compiling a new kernel?
Btw. aren't they optional
Yes they are, but they're only useful if a significant portion of users enable them, to track down reported performance problems, so it is expected that they will be compiled into most distribution kernels.
It makes a lot of sense after reading Ingo's LKML rant about kernel instrumentation: http://lwn.net/Articles/252887/
Maybe...
Maybe always have it included/enabled on the Release Candidates, but never have it included/enabled on the stable/final-release?
A 32-way Itanium with 132G
Why the fuck would you use
Why the fuck would you use ext3 on such a system? This is what modern filesystems are for! The benchmark would be massively faster with any other filesystem.
Firstly, this particular micro-benchmark was only testing mount/umount time. So it would probably have been massively slower with more "modern" filesystems. You missed that because you don't really know what you're talking about.
Secondly, in this kind of environment stability and ease of data recovery is vastly more important than a dubious potential performance increase. If you get a job in this environment in the future (and I hope it's a very long way in the future after you've learned some important lessons the hard way) you will understand.
Margin or error
Is it just me or are the reported differences just not statistically significant?
There seems to be a mention of 10 runs and then the following results are given:
Kernel w/out BT STDDEV w/ BT STDDEV
------------------------------------- --------- ------ --------- ------
2.6.23-rc6 + 2.6.23-rc6-mm1 14.679982 0.34 27.754796 2.09
2.6.23-rc6 + 2.6.23-rc6-mm1 + markers 14.993041 0.59 26.694993 3.23
The standard deviation on an average is (standard deviation of samples)/sqrt(number of runs). To have a 95% confidence interval, we need to take 2 standard deviations on each side and when looking at the error on a difference (or sum) we just sum the variances (not the standard deviation). What this means is that the margin or error (95% confidence) for the difference "w/o BT" is 0.43 and the margin for the results "w/ BT" is 2.4. The difference in the averages is much smaller than the error, so it would mean that none of that is statistically significant.
Did I miss something obvious (or not) somewhere?