In April, 2.4 kernel maintainer Willy Tarreau queried the Linux kernel mailing list regarding how the 2.4 kernel is still being used. He followed up summarizing the responses, suggesting that about 5% of 2.4 users run the kernel on old recycled laptops at home or on PDA's and thin clients, running whatever works with no real need to upgrade. Another 5% of the users are on desktop PCs and monitoring stations, not upgrading because "it works". From there, about 50% of the users run the 2.4 kernel on general purpose servers and update regularly, still running the older kernel due to lack of need for new features and lack of time, and possibly due to failed earlier attempts to upgrade. Another 20% use the 2.4 kernel on application specific servers where reliability is of the highest importance. 10% of the users run the older stable kernel on routers, firewalls, and intrusion detections systems, with 1 to 2 year uptimes and limited updates in a business setting, and shorter uptimes and frequent updates in a personal setting. The final 10% or so use the kernel in embedded systems where stability is again very important, and the build tree may be highly modified, causing at least one major network equipment manufacturer to still be shipping devices with the 2.4.2 kernel. Willy continued:
"Based on that and on the workflow people took the time to explain, I realize that the distinction between -pre and -rc is useless (ding! Linus if you read this, don't beat me). In fact, either people want absolute reliability and they pick one kernel from the stable 2.4.X.Y branch, or they want more recent updates and they simply do their shopping in the -master branch, which is fairly easy thanks to the Gitweb interface. But there are almost no testers in 2.4, just users. That means that I don't have to expect immediate feedback when posting a pre-release. And it has happened several times that I got a build error report several weeks after the release.
"Also, since most people do not update more than 1 - 2 times a year, it's not very useful to have more than 1 - 2 new versions a year, especially since we have the stable release. For this reason, I think I will issue stable releases a bit more often for users to quickly get their fixes, but progressively increase the delay between major releases. Those ones will only be issued with new PCI IDs, major driver updates, compiler support, etc..."
From: Willy Tarreau
Subject: Linux 2.4 usage statistics (results)
Date: Jun 1, 4:50 pm 2008
Hi all,
I've tried to summarize the usage reports I got for kernel 2.4.
[ Note: some of the responders asked to be notified when I send the
results, so I've Bcc'd the responders so that they can get the
results without being polluted by potential responses. ]
About 22 useful responses. It is not possible to report accurate numbers
because most participants did not themselves have accurate numbers. I
would say that based on the responses (and a few cases I know), 2.4 usage
approximately follows this distribution :
- old recycled laptops at home, or PDA/thin clients - ~5%
They rely on whatever kernel managed to boot on them, then they never
got updated anymore. Security generally not too much of an issue, no
plan to update to newer 2.4, let alone 2.6. The hardware will die first.
Sometimes a few patches were added to support one particular component
or enable one feature specific to the use case. These boxes have the
lowest level of criticity.
- desktop PCs, monitoring stations - about the same as previous ones, with
an increased level of criticity. Generally users have not upgraded simply
because "it works". Those are systems with known fixed hardware and usage
pattern. Most often they serve as X11 terminals and/or browsers, and do
not require many updates. They cause trouble to the user(s) when they
die and no benefit is expected from an upgrade to 2.6. Future installation
(same or replaced hardware) will generally be on 2.6.
- general purpose servers : regularly updated - about 50%.
The pattern is most always the same. A lot of services are installed,
among which WWW, Mail, DNS, Samba, NFS. The server never experiences
any outage, and a moderate amount of people would be affected if a
problem happened. Almost always up to date with latest 2.4. Reason
for not upgrading to 2.6 generally is lack of need and time, as well
as risks of regressions which would get a lot of users angry. Users
per server are in the range of 10 to 1000. Generally some migration
tests are in progress. Early attempts at 2.6 failed due to stability
problems (GRE tunnels freezing, sparc images limited in size).
- application specific servers : about 20%. Often mission-critical.
Generally deployed with latest 2.4, and almost never updated.
Long stress-tests before production. Generally the number of users
does not mean anything, it's the process the system is involved in
which makes sense. Pre-press workflow applications for daily
newspapers and weather broadcasting for TV channels were reported.
Reliability is the #1 requirement, and generally recent tests in
2.6 have not been much satisfying. One reported DRBD 0.6 very reliable
in 2.4, while DRBD 0.7 not as much in 2.6. Another one reported that
the application needed to be ported to correctly work in 2.6.
Interestingly, in all cases, the high level of criticity implies
that 2.6 is still being evaluated, in the event that one day there
is no choice (eg: due to hardware compatibility).
- routers/firewalls/VPN/IDS : about 10%. Longest uptimes up to 5-600 days
in business environments, implying kernel not often updated (avg once a
year). For personal use, it's where the updates are the most common
(probably due to the quick reboot and low impact of a short outage).
One user reported a number of these boxes deployed in finance/business/train
environments with high level of criticity, receiving occasional updates
during dead hours. Generally no upgrade to 2.6 is planned (though I think
it is where it might be the least painful). One user reported missing
devfs in 2.6, too big kernel size, and no clear upgrade path. We might
help them switch by enumerating the small number of changes in such a
networked environment. What may be the most obscure in business environment
is the lack of sight of long term reliability. I personally know about
several firewalls I have deployed 5 years ago in a finance environment
which see 1 million unique clients a day. It has been discussed about
migrating them to 2.6 but it seems like the only way to know how long
they will survive is to test in production, which is generally not
acceptable. So leaving them on 2.4 is often the solution.
- embedded systems : about 10% of the reports, probably more in terms of
number of systems. They often expect very long uptimes (two of them
reported rebooting less than once a year). This is either dictated by
the fact that consumer electronics get bad reputation when failures or
updates happen too often, or because the big customer does not want
to update something which works. Reported usages include outdoor
information displays for trains and busses, various consumer electronic
devices including TV sets, monitoring systems and building security
devices (eg door openers). Upgrade to 2.6 not needed nor planned at
all for some, and planned for others to gain better hardware support.
Some are still testing but experiencing regressions (eg: parport). In
all cases, the build process has to be massively redesigned and this
costs a lot of time and money. Not surprizing that some major network
equipement manufacturer still ships device with and old 2.4.2 in them.
It is worth noting that several users still using 2.4 roll their own distro,
which will need to be substantially updated to support 2.6. This seems to be
one showstopper too, especially when the distro was designed for easy remote
and centralized upgrades. In this case, what seems to happen is that the old
version continues to live at existing locations, and a new version appears
with support for 2.6 for newer systems. But from an organisational point of
view, it's not always easy to manage two branches of a distro when only one
has been enough for years.
Surprizingly, it appears that very few users patch their kernels. Those who
do so have their own distros or build kernels for large amounts of machines
(eg: consultants doing so for their customers). Many of the reported patches
have found their way in mainline 2.6, but some have definitely vanished (eg:
devfs,linux-abi, kstat, umsdos, lvm). Among the patches, GRsecurity and PaX
were reported several times. Some special-purpose patches are included in
appliance products.
Vendor drivers are the only ones installed by most users who patch (eg:
3Ware 9xxx, sk98lin, bnx2). One user reported an annoying problem with USB
for which I realized I've been having a patch in my own tree for years, to
the point I completely forgot the problem existed. It is related to
USB-storage, where unplugging then replugging a device assigns it a new
SCSI drive name. I may merge it into 2.4.37.
Based on that and on the workflow people took the time to explain, I
realize that the distinction between -pre and -rc is useless (ding! Linus
if you read this, don't beat me). In fact, either people want absolute
reliability and they pick one kernel from the stable 2.4.X.Y branch, or
they want more recent updates and they simply do their shopping in the
-master branch, which is fairly easy thanks to the Gitweb interface.
But there are almost no testers in 2.4, just users. That means that I
don't have to expect immediate feedback when posting a pre-release. And
it has happened several times that I got a build error report several
weeks after the release.
Also, since most people do not update more than 1/2 times a year, it's
not very useful to have more than 1/2 new versions a year, expecially
since we have the stable release. For this reason, I think I will issue
stable releases a bit more often for users to quickly get their fixes,
but progressively increase the delay between major releases. Those ones
will only be issued with new PCI IDs, major driver updates, compiler
support, etc...
Last, I think users would benefit from an inventory of functional changes
between 2.4 and recent 2.6, with a reversible upgrade path, so that they
can experiment with 2.6 on their production machine and immediately go
back to 2.4 in case of trouble.
Well, I hope it was not too much boring!
Regards,
Willy
--
From: Willy Tarreau
Subject: Linux 2.4 usage statistics
Date: Apr 28, 1:24 pm 2008
Hi all,
I would be very interested in getting feedback about kernel 2.4 usage.
Please, please, I do not want this thread to degenerate into a useless
flamewar or "mine is bigger" discussion. For this reason, I would like
that participants reply privately to me so that this mail does not turn
into a thread.
I know 2.4 is still used in a lot of sensible areas, although far less
than 2.6 nowadays. At least I know that several appliance makers/sellers
still use it in new products, and even very old versions sometimes. What
I'd like to estimate is the way it is used, if it is systematically
patched with local add-ons, frequency of updates (if any), etc... so
that I can adapt my process if needed. For instance, I don't release
any -rc for stable releases because I'm convinced that nobody will ever
test them, but I have no problem being proved wrong.
If I get enough replies to build up a statistic synthesis, I'll try to
summarise it (eventhough 73% of statistics are all made up :-) ).
This "poll" is not meant to dictate what will get merged next (we're in
feature freeze anyway), but it helps me know your usage better, to try
to serve you better. Also, if a majority of people report the same
problem or lacking feature for which a trivial fix is know, it can be
studied.
Basically what I'd like to know is the following. You can use the proposed
responses as a guide, but they're not mandatory. Also if possible and adapted,
please report number of concerned units :
1) where do you use it ?
- PC turned into network equipment (router, LB, BGP route reflector, ...)
- security equipment (firewall, vpn, ids/ips, traffic analyzer, ...)
- proxy services (proxy, smtp relay, reverse-proxy, anti-virus/spam, ...)
- storage/directory server (NFS, LDAP, logging, ...)
- multi-function server (proxy/fw/nfs/mail/...)
- monitoring / remote access
- desktop/workstation
- laptop (if recent, what model ?)
- dedicated appliances (that you may be designing/selling)
- soho embedded systems (eg: wifi routers, ...)
- PDA ?
- other ?
2) how critical are the systems it runs on ?
- medical (people's health depend on them) ?
- finance (massive amounts of money pass through them) ?
- business-critical (you may go bankrupt if it fails too often) ?
- mission-critical (you may loose your job if it fails too often) ?
- security-critical (security gets degraded when it fails) ?
- users get angry at you when it fails (admin, or do this for a living) ?
- not that important (you may reboot when you decide to do so) ?
- no importance at all (eg: home usage, experimentation, ...) ?
3) how many users are you serving with it approximately, all systems
included ? Or better, how many people will notice the failure at once
if any ? Note: do not count web population as users, but use maximum
concurrent users instead.
- < 10
- < 100
- < 1000
- < 10000
- more ?
4) what version are you running ?
- latest or close to that (eg: 2.4.36.x)
- latest was available last time you upgraded
- somewhat old mainline (2.4.32-2.4.34)
- quite old mainline (2.4.21..2.4.31)
- very old mainline (2.4.2, 2.4.17, ...)
- debian's 2.4.27
- rhel 3's 2.4.21
- my other standard distros's 2.4 kernel
- an embedded distro's heavily patched 2.4 (port to other archs, ...)
5) have you applied any patches ?
- provided with the distro
- security: pax/grsec/ow, capabilities, ...
- performance: epoll, variable_hz, jiffies_64, preempt, O(1), ...
- network: tcp md5, transparent proxy, netfilter tcp_window_tracking...
- drivers: very specific to your platform, self-added PCI-IDs, vendors'
- filesystems: updated jffs2+mtd, squashfs, cifs, fuse, ...
- security/stability fixes backported from later 2.4 versions
- other security/stability fixes or improvements
- drivers backported from 2.6
- other ?
6) how often do you update (may translate into avg uptime) ?
- every release
- every major release after a few dot-versions
- only when you see an important (to you) security fix
- same as well as with reliability fixes
- at least once a year
- only if you need to re-install
- after a lot of careful revalidation
- never
7) how do you test ?
- pre-production environment in equivalent conditions
- long-term reliability stress-tests (days to weeks)
- quick regression testing before production
- you test in production
- no test at all, blind upgrade
- other ?
8) why have you not upgraded to 2.6 yet (including Adrian's 2.6.16.X ?)
- not needed, all features OK and hardware still 100% compatible
- you have to change user-space tools
- you don't feel comfortable with 2.6's patch pace (but 2.6.16 is quite
comparable to 2.4)
- you do not trust 2.6 enough yet ? (why ?)
- missing features you had in 2.4 mainline that are not in 2.6 anymore ?
- missing features you had in your 2.4 patches that do not exist in 2.6 ?
- drivers not existing anymore in 2.6 ?
- drivers not working anymore in 2.6 ?
- performance regression in 2.6 ?
- you find 2.6 more complicated ?
- other ?
9) when do you plan to switch to 2.6 ?
- it has already begun (and as a supplement, if you had to do so because
2.4 did not work anymore for you for other reasons than hardware
support and features, please indicate which ones)
- very soon (<1 year)
- not before 1 year
- next installations
- next batch of major upgrades
- when your hardware is not compatible anymore
- when you can't build 2.4 with your distro's gcc
- when 2.6 supports feature/driver XXX
- when 2.6 is more reliable (example ?)
- when 2.6 development slows down
- the latest possible (why ???)
- never (why ??? and what would you switch to then ?)
10) any comments or suggestions about current release process, minor issues
that you are facing each time such as external patches that do not apply,
recurrent problems on all of your setups, etc... ?
Well, that's all. You do not need to respond precisely, and once again,
I'm just trying to get the figure.
And please do not forget! Reply to me directly, do not pollute the list,
that way you will refrain the random geeks from comparing their setups
to other ones.
Thanks!
Willy
--
2.6 is too unstable for us.
2.6 is too unstable for us. Plus it has several securities issues that always take a few weeks or months to be discovered. this is inherent to the way 2.6 is developed.
Also developing an invasive patch on 2.6 means rewriting 50% of it at each single kernel upgrade. What a mess! How unconformable.
thus we still use 2.4 even if we'd like to move on 2.6 for some reasons sometimes.
Just think if the kernel
Just think if the kernel devs had stuck with the old model. We'd be up to Linux 11.5 by now!
Also developing an invasive
Also developing an invasive patch on 2.6 means rewriting 50% of it at each single kernel upgrade.
then submit it to mainline!
Also developing an invasive
Also developing an invasive patch on 2.6 means rewriting 50% of it at each single kernel upgrade. What a mess! How unconformable.
If your code would be in the kernel, you wouldn't have this problem in the first place. GPL means you can do a lot, but it still gives you the responsibility to give back to the community.
Assuming the mainline wanted
Assuming the mainline wanted the patchset and didn't determine it to be a "distribution level" issue. I don't see how that would make the problem go away? Main difference is that you'd be expected to update in a timely fashion. Maybe it's by assuming that you could push your code off on someone else to maintain, which in the general case may or may not be reasonable.
Does a git-rebase not save
Does a git-rebase not save you a huge load of the work involved with a significant non-mainline patchset? If your work is copied to a git tree which follows Linus, then you can rebase the branch as the kernel.org tree is updated.
K3n.
Hah?
2.6 is too unstable for us. Plus it has several securities issues that always take a few weeks or months to be discovered. this is inherent to the way 2.6 is developed.
See, if you actually helped testing and developing it, it would have had less problems. It all comes back to you.
We use the linux kernel in
We use the linux kernel in an embedded communications product, and we use 2.4 because a hardware vendor for one of our major connectivity options (T1 card or something, I'm not sure on the specifics) only has a driver for the 2.4 kernels.
Choice of card is dictated by the client, and having it available is a big boost, so we're kind of stuck with it.
2.4-only driver? Surely not!
I don't understand. Why didn't they submit the driver for inclusion in the mainline kernel? Then they would have got an upgrade to 2.6 for free.
2.4-only driver? Sure.
Some drivers are closed-source. Don't ask me why. That's just the way things are. Sometimes hardware vendors insist on monopolizing the right to develop and maintain drivers for their products. Then they shirk the responsibility they put on themselves, thereby making their products unusable. It's baffling to me.
At work I have a bunch of systems that are stuck at 2.4 because the driver for the RAID controller is proprietary, closed-source, and 2.4-only. It's also buggy. For example, once I mount a filesystem, it gets pinned forever, and I have to reboot to unmount it. Usually this means that the process for unmounting also includes editing /etc/fstab to prevent automatic remounting on reboot. :^(
Why 2.4-only? The vendor says that only the "stable" release of Linux is supported. They consider 2.4 to be stable and 2.6 to be unstable.
which RAID vendor is it?
Could you share with us which crappy RAID vendor it is? Just to avoid it in the future.
RAID vendor
Adaptec. The computers have 39320A-R Ultra320 SCSI RAID controllers in them. The driver is a320raid.o for Linux 2.4.21, or some Red Hat variant thereof.
What?
Adaptec's support of Linux is actually pretty good, which is why the drivers for this card are in fact in the 2.6 kernel. You can even download the source from Adaptec's website. The following release notes confirm that it is supported :
http://www.adaptec.com/en-US/speed/scsi/linux/aic7Yxx-2_0_15-6_3_11-linu...
So how did you come to the conclusion that they weren't?
I didn't come to that
I didn't come to that conclusion. The subcontractor who built the machines did. At the time they put it all together, they found (right or wrong) that the RAID controllers could only be made to work with certain older Red Hat releases using Linux 2.4. And the a320raid.o driver they got from Adaptec is certainly buggy.
2.4 only binary blob
To perhaps shed a little light, I've run into similar issues, where the question is actually the RedHat installation media, and not directly the kernel at all, RedHat seems to go out of their way to support a certain range of server hardware "out of the box", on their install cd, even if it meant including the blob that your are now stuck with, and when that particular card was no longer in their "target" set of hardware, they dropped the blob, and their default kernel on their install disk has no support for the particular raid controller, thus they report that "New versions of Linux do not support that particular device". Anyone treating RedHat as a drop-in replacement for their previous proprietary vendor is subject to this type of silliness.
Amazing
The only thing baffling to me is why customers continue to put up with this nonsense. If you didn't, it would stop immediately.
If allowed to design the
If allowed to design the systems in question, I would have just gone with LVM. (RAID level 0 is used, which is hardly RAID at all. But for the specific application, that's fine.) You're right; I greatly prefer to avoid hardware with closed-source drivers, because I want my dollar vote to encourage open source.