Linux: Cleaning the Kernel Headers

Submitted by Jeremy
on April 28, 2006 - 11:58am

David Woodhouse offered a collection of patches aimed at cleaning up Linux kernel headers, explaining that he maintains the Fedora glibc-kernheaders package and has found the process of syncing with the latest kernel to be unnecessarily tedious. Linus Torvalds acknowledged the effort, noting that he preferred to wait until early in the 2.6.18 development cycle before merging the patches as things tend to break, "yeah, people shouldn't include kernel headers, but if they didn't, this patch wouldn't matter. And when they do, patches like this tends to show some strange app that depends on the current header layout.. Gaah."

David responded, "Well, yes, but we all know that people _have_ to include kernel headers. We can't just bury our head in the sand and say 'they mustn't do that'. The kernel headers contain all the juicy stuff like structure definitions and ioctls which you _need_ in order to communicate with the kernel. The problem is that we don't actually have any _discipline_ about how we throw our kernel headers over the wall. We never even _think_ about how usable they are in userspace, or how what we're doing will affect userspace."

Linus was quick to retort, "if this is work to try to make kernel headers generally palatable to user space, I'm not going to apply it at all. Not now, not early in the 2.6.18 sequence, not EVER. Because that's not a goal I believe in for a moment." He then went on to explain, "in contrast, if this is work to make it eventually _easier_ for _library_ people to decide to upgrade their kernel-related information, I'm ok with it. But that 'target audience' part really is very very important. The target audience should NOT be applications including kernel headers. The target audience should be distributions and library maintainers."


From: David Woodhouse [email blocked]
To:  torvalds, [email blocked]
Subject: Simple header cleanups
Date:	Thu, 27 Apr 2006 03:13:42 +0100

Andrew (or preferably Linus since these are fairly simple and
unintrusive bug fixes), please pull from my tree at 
git://git.infradead.org/hdrcleanup-2.6.git

(Gitweb at http://git.infradead.org/?p=hdrcleanup-2.6.git;a=summary)

This tree contains a number of simple fixes for kernel headers which are
currently exposing things to userspace that should not be present
outside #ifdef __KERNEL__.

Since ifdefs are horrid, these patches tend not to add new ones --
mostly it's a case of moving things inside existing ifdefs, rather than
adding new ones.

Most of the patches are small and self-contained, except for the one
which removes #include <linux/config.h> from everything under include/

The individual commits are as follows:

      Remove user-visible references to PAGE_SIZE in include/asm-powerpc/elf.h
      Include <linux/jiffies.h> from linux/acct.h only in kernel-private part.
      Don't include agp_backend.h in user-visible part of agpgart.h
      Use __KERNEL__ to hide kernel-private bits of linux/gameport.h
      Export only the appropriate GS_xxx flags to userspace from generic_serial.h
      Include various private files only from within __KERNEL__ in genhd.h
      Sanitise linux/i2c-algo-ite.h for userspace consumption
      Sanitise linux/i2c.h for userspace consumption
      Don't include <linux/device.h> from user-visible part of linux/ipmi.h
      Remove gratuitous inclusion of <linux/pci.h> from linux/isdn/tpam.h
      Sanitise linux/mman.h for userspace consumption
      Don't include private files from user-visible part of linux/ncp_fs.h
      Don't include <linux/list.h> from user-visible part of linux/msg.h
      Don't include <linux/stringify> from user-visible part of linux/net.h
      Don't include private headers from user-visible parts of include/linux/nfs*.h
      Don't include private headers from user-visible parts of linux/quota.h
      Don't include <linux/list.h> from user-visible part of reiserfs_xattr.h
      Partially sanitise linux/sched.h for userspace consumption
      Don't include <asm/atomic.h> from user-visible part of linux/sem.h
      Don't include private headers from user-visible part of linux/signal.h
      Move comment in mtd-abi.h to stop confusing unifdef
      Don't include <linux/spinlock.h> from user-visible part of linux/wanrouter.h
      Don't export CONFIG_COMPAT stuff in linux/usbdevice_fs.h to userspace
      Sanitise linux/sunrpc/debug.h for userspace consumption
      Don't include private headers from user-visible part of linux/smb_fs.h
      Don't include private headers from user-visible part of linux/ext2_fs.h
      Don't include private headers from user-visible part of linux/ext3_fs.h
      Don't include <linux/config.h> and <linux/linkage.h> from linux/socket.h
      Don't include linux/config.h from anywhere else in include/
      Sanitise linux/audit.h for userspace consumption, split elf-em.h from elf.h
      Sanitise linux/sched.h for userspace consumption

Excluding the one-line files which have just <linux/config.h> removed,
the diffstat looks like this:

 asm-m32r/mmu_context.h                       |    2
 asm-powerpc/elf.h                            |    7 --
 asm-ppc/page.h                               |    2
 asm-sparc/system.h                           |    2
 linux/acct.h                                 |    3
 linux/agpgart.h                              |    3
 linux/audit.h                                |    4 -
 linux/elf-em.h                               |   44 +++++++++++++
 linux/elf.h                                  |   59 -----------------
 linux/ext2_fs.h                              |    2
 linux/ext3_fs.h                              |    7 --
 linux/gameport.h                             |    6 +
 linux/generic_serial.h                       |    6 +
 linux/genhd.h                                |   12 +--
 linux/i2c-algo-ite.h                         |    7 +-
 linux/i2c.h                                  |    9 +-
 linux/ipmi.h                                 |    2
 linux/mman.h                                 |   12 ++-
 linux/msg.h                                  |    2
 linux/ncp_fs.h                               |    5 -
 linux/net.h                                  |    3
 linux/nfs.h                                  |    8 +-
 linux/nfs4.h                                 |    6 -
 linux/nfs_fs.h                               |   39 +++++------
 linux/quota.h                                |    4 -
 linux/reiserfs_xattr.h                       |    3
 linux/sched.h                                |   90 +++++++++++++--------------
 linux/sem.h                                  |    2
 linux/signal.h                               |    4 -
 linux/smb_fs.h                               |    4 -
 linux/socket.h                               |    2
 linux/sunrpc/debug.h                         |   24 +++----
 linux/usbdevice_fs.h                         |    2
 linux/wanrouter.h                            |    4 -
 mtd/mtd-abi.h                                |    5 -
 864 files changed, 191 insertions(+), 1034 deletions(-)



-- 
dwmw2


From: Linus Torvalds [email blocked] Subject: Re: Simple header cleanups Date: Wed, 26 Apr 2006 19:18:52 -0700 (PDT) On Thu, 27 Apr 2006, David Woodhouse wrote: > > Andrew (or preferably Linus since these are fairly simple and > unintrusive bug fixes), please pull from my tree at > git://git.infradead.org/hdrcleanup-2.6.git Hmm. Every time we've done this in the past, something has broken, so I'd actually _much_ rather wait until early in the 2.6.18 cycle than do it now. Yeah, people shouldn't include kernel headers, but if they didn't, this patch wouldn't matter. And when they do, patches like this tends to show some strange app that depends on the current header layout.. Gaah. Linus
From: Gerrit Huizenga [email blocked] Subject: Re: Simple header cleanups Date: Wed, 26 Apr 2006 19:27:27 -0700 On Wed, 26 Apr 2006 19:18:52 PDT, Linus Torvalds wrote: > > > On Thu, 27 Apr 2006, David Woodhouse wrote: > > > > Andrew (or preferably Linus since these are fairly simple and > > unintrusive bug fixes), please pull from my tree at > > git://git.infradead.org/hdrcleanup-2.6.git > > Hmm. Every time we've done this in the past, something has broken, so I'd > actually _much_ rather wait until early in the 2.6.18 cycle than do it > now. > > Yeah, people shouldn't include kernel headers, but if they didn't, this > patch wouldn't matter. And when they do, patches like this tends to show > some strange app that depends on the current header layout.. Gaah. > > Linus Except there are a few things you can't use in user space now, like connectors, without including a kernel header file. So, should apps duplicate the structures and definitions in each and every application and hopefully notice when the kernel API changes? Seems painful. Plus, do we really want Apache or other licensed software directly including GPL header files? Oh what a tangled web we weave when we suck GPL headers into other applications. Damned if you do, damned if you don't. gerrit
From: Linus Torvalds [email blocked] Subject: Re: Simple header cleanups Date: Wed, 26 Apr 2006 19:46:29 -0700 (PDT) On Wed, 26 Apr 2006, Gerrit Huizenga wrote: > > Damned if you do, damned if you don't. Right. Which is why I'm ok with the cleanups, I'm just _not_ ok with them happening this late in the game (ie I'm hoping to get 2.6.17 out in a couple of weeks..) Linus
From: David Woodhouse [email blocked] Subject: Re: Simple header cleanups Date: Thu, 27 Apr 2006 03:37:37 +0100 On Wed, 2006-04-26 at 19:18 -0700, Linus Torvalds wrote: > Hmm. Every time we've done this in the past, something has broken, so I'd > actually _much_ rather wait until early in the 2.6.18 cycle than do it > now. OK. I've tried to be _very_ conservative with this set -- it mostly is just a case of moving #includes from one part of a header file to another so they aren't exposed to userspace. I figured the more interesting stuff could come later. But if you'd rather have it only in -mm only for now, that's fine. > Yeah, people shouldn't include kernel headers, but if they didn't, this > patch wouldn't matter. And when they do, patches like this tends to show > some strange app that depends on the current header layout.. Gaah. Well, yes, but we all know that people _have_ to include kernel headers. We can't just bury our head in the sand and say "they mustn't do that". The kernel headers contain all the juicy stuff like structure definitions and ioctls which you _need_ in order to communicate with the kernel. The problem is that we don't actually have any _discipline_ about how we throw our kernel headers over the wall. We never even _think_ about how usable they are in userspace, or how what we're doing will affect userspace. That's why there are so many silly little cleanups which I had to make just to be able to use some recent kernel headers sanely again in userspace, instead of the ancient fork we'd taken in Fedora from a 2.4 kernel. The answer to this lies in my other git tree, at git://git.infradead.org/hdrinstall-2.6.git -- again in gitweb, at http://git.infradead.org/?p=hdrinstall-2.6.git;a=summary Based on an original implementation by Arnd Bergmann, this implements new 'make headers_install' target for the kernel Makefiles, which exports a selection of the contents of our include/ directories to userspace, having run them through unifdef where appropriate. This gives us a pristine set of headers which _are_ suitable for userspace inclusion. I've been talking to maintainers of the 'kernel-headers' or similar packages in other distributions, and we seem to have a consensus that this is going to be useful. All the distributions can ship basically the _same_ set of headers, instead of all doing their own thing to clean them up, and getting it inconsistent across distros. As well as achieving that laudable goal, the export step also allows us (and in particular our janitor-types) to _read_ through the headers which userspace gets, and start to clean them up a bit more, cleaning up namespace pollution and removing things which are still visible but which shouldn't be. It allows us to take a diff of the _exported_ headers between one release and the next, and see the user-visible changes in isolation. -- dwmw2
From: Linus Torvalds [email blocked] Subject: Re: Simple header cleanups Date: Wed, 26 Apr 2006 19:59:10 -0700 (PDT) On Thu, 27 Apr 2006, David Woodhouse wrote: > > Well, yes, but we all know that people _have_ to include kernel headers. > We can't just bury our head in the sand and say "they mustn't do that". > The kernel headers contain all the juicy stuff like structure > definitions and ioctls which you _need_ in order to communicate with the > kernel. .. and that's generally the job of library maintainers. I disagree violently with the notion that any normal system should _ever_ have the regular headers have anything to do with "what kernel source is installed right now". It is untenable to expect kernel developers to have to worry about user-level header problems. > The problem is that we don't actually have any _discipline_ about how we > throw our kernel headers over the wall. We never even _think_ about how > usable they are in userspace, or how what we're doing will affect > userspace. AND WE SHOULD NOT HAVE TO! Linkages are bad. We _will_ break kernel headers for user space, and that's just a undeniable fact. If this is work to try to make kernel headers generally palatable to user space, I'm not going to apply it at all. Not now, not early in the 2.6.18 sequence, not EVER. Because that's not a goal I believe in for a moment. In contrast, if this is work to make it eventually _easier_ for _library_ people to decide to upgrade their kernel-related information, I'm ok with it. But that "target audience" part really is very very important. The target audience should NOT be applications including kernel headers. The target audience should be distributions and library maintainers. Linus
From: David Woodhouse [email blocked] Subject: Re: Simple header cleanups Date: Thu, 27 Apr 2006 04:17:51 +0100 On Wed, 2006-04-26 at 19:59 -0700, Linus Torvalds wrote: > Linkages are bad. We _will_ break kernel headers for user space, and > that's just a undeniable fact. Agreed. And distributions and library maintainers _will_ fix them. Are we to deny those people the tools which will help them to keep track of our breakage and submit patches to fix it? The current situation is that they all go off and do their own thing to work around it, instead of submitting those fixes. We need to do better than that. > If this is work to try to make kernel headers generally palatable to user > space, I'm not going to apply it at all. Not now, not early in the 2.6.18 > sequence, not EVER. Because that's not a goal I believe in for a moment. Kernel headers are never going to be generally palatable to user space. That's always been clear, and it's a large part of the reason why we wanted the 'export' step -- so that a _subset_ of headers could be copied out for use by userspace, rather than a complete tarball as has been used in the past. > In contrast, if this is work to make it eventually _easier_ for _library_ > people to decide to upgrade their kernel-related information, I'm ok with > it. But that "target audience" part really is very very important. The > target audience should NOT be applications including kernel headers. > > The target audience should be distributions and library maintainers. And those are the people I've been talking to over the last week or so. All the distributions have their own hacks and patches and scripts which mangle the kernel headers. Some from a vaguely current copy, some from a copy which was forked years ago and kept vaguely up to date with syscalls and new ioctls by hand. The results differ wildly from distribution to distribution, and the end result for the _user_ is a steaming inconsistent pile of crap. What I'm trying to do here is centralise that process and let people work on it together -- and yes, it still needs to go through distributions and library maintainers. The point is not to go back to making symlinks from /usr/include/{linux,asm} to the kernel directories again -- that would be a retrograde step. The point is that distro and library maintainers can work from what is provided by the kernel to achieve a consistent and useful result. That _is_ needed. It's a quality of implementation issue. -- dwmw2
From: Linus Torvalds [email blocked] Subject: Re: Simple header cleanups Date: Wed, 26 Apr 2006 20:31:00 -0700 (PDT) On Thu, 27 Apr 2006, David Woodhouse wrote: > > Agreed. And distributions and library maintainers _will_ fix them. Are > we to deny those people the tools which will help them to keep track of > our breakage and submit patches to fix it? No. As mentioned, as long as the target audience is distributions and library maintainers, I definitely think we should do help them as much as possible. Our problems have historically been "random people" who have /usr/include/linux being the symlink to "kernel source of the day", which is an unsupportable situation. (And yes, for a short while back in the early nineties, that symlink was even the proper thing to do. But exactly because it's unsupportable, it pretty quickly got to be a "don't do that then", but I still occasionally hear from people who use bad distributions). Linus
From: David Woodhouse [email blocked] Subject: Re: Simple header cleanups Date: Thu, 27 Apr 2006 04:41:34 +0100 On Wed, 2006-04-26 at 20:31 -0700, Linus Torvalds wrote: > No. As mentioned, as long as the target audience is distributions and > library maintainers, I definitely think we should do help them as much as > possible. Our problems have historically been "random people" who have > /usr/include/linux being the symlink to "kernel source of the day", which > is an unsupportable situation. Right. I'm not interested in helping random people who want the kernel source of the day; my point here is to just help the distributions get their collective act together w.r.t. the kernel headers which we _have_ to ship in some form or other. In particular, my reason for getting started on it was because for my sins I own the glibc-kernheaders package in Fedora, and it's a complete pain in my wossname. We _do_ need to update periodically. I'd been doing it by cherry-picking stuff like new ioctls and syscalls manually, and it was horrid. The answer was obviously to have some way to at least partially automate the process of updating from the real kernel... and it's fairly obvious how we got from that requirement to the second of the git trees I'm showing you now, having consulted the people who maintain equivalent packages in other distributions along the way. -- dwmw2
From: Adrian Bunk [email blocked] Subject: Re: Simple header cleanups Date: Thu, 27 Apr 2006 23:37:54 +0200 On Wed, Apr 26, 2006 at 08:31:00PM -0700, Linus Torvalds wrote: > > On Thu, 27 Apr 2006, David Woodhouse wrote: > > > > Agreed. And distributions and library maintainers _will_ fix them. Are > > we to deny those people the tools which will help them to keep track of > > our breakage and submit patches to fix it? > > No. As mentioned, as long as the target audience is distributions and > library maintainers, I definitely think we should do help them as much as > possible. Our problems have historically been "random people" who have > /usr/include/linux being the symlink to "kernel source of the day", which > is an unsupportable situation. >... A definition of the kernel <-> userspace ABI is required. Currently, each distribution does it slightly different. Currently we might accidentially touch the userspace ABI of the kernel without even noticing it. I'd like to do the following: Create an include/kabi/linux/ with the following properties: - the goal is that include/kabi/linux an be copied verbatim to /usr/include/ (by distributions and library maintainers, normal users should use the copy installed by their distribution in /usr/include/) - kernel code outside include/ wouldn't notice, since for each ABI header include/kabi/linux/foo.h there's a header include/linux/foo.h that does #include <kabi/linux/foo.h> - moving stuff to kabi/linux can happen incrementially - once the move is complete, we can announce this as the official ABI headers - moving to the kabi/ headers shouldn't cause compile time or runtime breakages for userspace software [1] I'd be willing to work on this, but if you have any problems with this approach, I'd like to understand them > Linus cu Adrian [1] with the exception of header abuses like the MySQL asm/atomic.h usage -- "Is there not promise of rain?" Ling Tan asked suddenly out of the darkness. There had been need of rain for many days. "Only a promise," Lao Er said. Pearl S. Buck - Dragon Seed
From: Linus Torvalds [email blocked] Subject: Re: Simple header cleanups Date: Thu, 27 Apr 2006 15:00:16 -0700 (PDT) On Thu, 27 Apr 2006, Adrian Bunk wrote: > > A definition of the kernel <-> userspace ABI is required. Well, we can get certain hints by just looking at every single type that is used as a __user pointer. That should give us a lot of the type information. The other big piece ends up being argument values passed in to system calls, most notably ioctl numbers, but there are certainly others too. And then there are the system call numbers themselves, and their calling conventions (fairly small part). > Create an include/kabi/linux/ with the following properties: I do hate your naming. Why is that "linux" there? We're not going to have FreeBSD kabi files. And what about the (pretty common) architecture-specific ones? The dependency chain is also quite often nontrivial. The ABI's all end up depending on the basic types, and often on each other (eg the ioctl numbers depend on the sizes of all the structures, which in turn depend on the architecture-specific structure layout and low-level types). So it's _not_ usually possible to just do one file that does one thing, because they do actually have linkages. And the linkages can be nasty, because they can easily be linkages that POSIX - and other standards - forbid them from being visible (you cannot expose certain typenames if they weren't _explicitly_ included, regardless of whether you need the type defines). This is one reason why we shouldn't even _plan_ on having header files that can just be _directly_ used by the C libraries etc, even if it's just a "small" kernel ABI header. Selling it as that kind of idea will inevitably mean that we then get blamed for not knowing magic rule #579 for SuS v2.1.6 subsection 8(a). And if we say "you can use these headers unmodified", that _is_ what we're going to get blamed for. I'm so _not_ interested in having to care or worry. So I seriously think we should aim for making it _easier_ for system libraries to get the information, but we should at the same time make it clear that we make it easier for them to get the basic info, BUT WE DO NOT CARE ABOUT THE RANDOM USER STANDARD OF THE DAY. Have you looked in /usr/include lately? Have you _looked_ at the "expose BSD names" vs "GNU extended source" vs "strict POSIX" vs "_XOPEN_SOURCE==600" bs "_USE_MISC" vs a million random and strange things? The day I see somebody adding crap like that to the kernel headers is the day I pull the plug on any "KABI" interfaces. And don't tell me this has got nothing to do with the kernel constants. Go look in something like /usr/include/bits/fcntl.h, and cry. See how it's using _exactly_ the kernel constants, but it has added all the random standard-of-the-day #ifdef (whether real standards, or the "GNU standards" or just "legacy BSD-like" etc). And THAT is why I don't think the simplistic "kabi" directory approach that people have brought up many times over many years is actually realistic. People don't realize that glibcs makes "struct flock" actually look different in user space depending on whether "__USE_FILE_OFFSET64" is defined or not. You just haven't seen just how NASTY those user-space headers are. They can't use _any_ kernel headers directly, because even when they want a _raw_ kernel data structure, they actually end up doing things differently in the _middle_ of that data structure. Really. So we should try to help those system libc people perhaps _find_ the values and structures they need, but no, I will _never_ allow the kernel headers to be used directly. And it doesn't _matter_ if they've been moved to a "kabi" subdirectory. That's not the issue. The issue is that user space does insane things that aren't acceptable in kernel space. Linus
From: David Woodhouse [email blocked] Subject: Re: Simple header cleanups Date: Fri, 28 Apr 2006 00:11:01 +0100 On Thu, 2006-04-27 at 15:00 -0700, Linus Torvalds wrote: > This is one reason why we shouldn't even _plan_ on having header files > that can just be _directly_ used by the C libraries etc, even if it's just > a "small" kernel ABI header. > > Selling it as that kind of idea will inevitably mean that we then get > blamed for not knowing magic rule #579 for SuS v2.1.6 subsection 8(a). > > And if we say "you can use these headers unmodified", that _is_ what we're > going to get blamed for. I'm so _not_ interested in having to care or > worry. OK... I wasn't going to go here, because I don't want a potentially contentious discussion to get in the way of the other cleanups which I've been doing. We made that mistake at least year's Kernel Summit, and we've made no progress since then despite having an almost unanimous consensus on where we need to go. But since Adrian's drawn you onto the topic, and since you've given me an insight into what your objection is, I'll respond... I don't think anyone imagines that a potential 'kabi' directory would be used to specify anything like a _library_ interface, or that random userspace programs would use stuff like 'struct stat' directly from it. That would be like going back to the dark old days of libc5, and makes me shudder as much as it does you. Glibc has all that crap to define the interface between it and _its_ users. Its version of stuff like 'struct stat' may indeed vary according to __USE_FILEOFFSET64 and the random standard of the day. We certainly don't want to get involved with that kind of mess, I agree. Our headers describe only _our_ interface to userspace, which often bears no relation to glibc's interface to its users. We don't need to care about our headers being usable in general by userspace applications; only that they're useful for glibc. (With the exception of stuff which defines ioctls, sockopts, etc.; obviously those are used by system libraries and utilities rather than just glibc, but still -- it's not just "random cross-platform POSIX userspace".) Nobody _wants_ the hellish things you seem concerned about. All we want to do is _think_ a little bit about avoiding gratuitous namespace pollution in the headers -- for example, we should use only types like __u32 instead of uint32_t and u32, we should refrain from exposing private helper inlines, etc. That's precisely what my cleanups have been fixing. It's not about conforming to POSIX and other random standards so that our headers are 'clean' for direct inclusion as part of standard headers. This is _our_ interface, and our definition of it. It's just a case of being a little bit disciplined about it rather than polluting it with internal crap. Distinct from the cleanups, my existing 'make headers_install' takes the limited set of headers which actually contain user-visible stuff, and runs them through unifdef to get a consistent set of headers which all distributions can use. The cleanups are just what was necessary to get the resulting exported headers to the point where I can build glibc against them on ppc, ppc64, i386, x86_64, s390, s390x and ia64. That's a massive step forward, and it's all I really want to concentrate on right now rather than talking about any potential 'next step'. What I'm doing now stands on its own and is necessary. However, there does seem to be a consensus that ifdefs are best avoided, and that we should avoid them by splitting private stuff into one file, public stuff into another, then including the latter from the former. I haven't done that in the hdrcleanup-2.6.git tree; I've mostly just moved stuff inside _existing_ '#ifdef __KERNEL__' guards within the same file. But I _have_ done it in the past for include/linux/mtd/ (and put the user-visible bits into linux/mtd/) and it's _much_ nicer that way. _If_ we do that kind of thing, then it also makes some sense for the user-visible files to be in one directory, and the private files to be in another directory. That's just good practice. That's all Adrian and others are talking about when they talk of a 'kabi directory', I believe -- not your dystopian vision of trying to make our headers conform to random standards-du-jour, but just treating them the the same as we try to do in the actual C files through out the kernel -- avoiding ifdefs by splitting stuff into separate files, and grouping those files into appropriate directories. I think they're right, but I'm not arguing that point right now because it's secondary to my task at hand, which is identifying the private vs. public stuff in the first place and reducing the pollution so that our headers are actually usable to build glibc again. In particular, I don't want that discussion to get in the way of what I'm doing right now. -- dwmw2

Related Links:

nvidia-installer doesn't work with it.

Anonymous (not verified)
on
May 3, 2006 - 4:07am

NVIDIA-Linux-x86-1.0-8756-pkg1.run complain
it has conflict type with "pm_message_t"..

Comment viewing options

Select your preferred way to display the comments and click "Save settings" to activate your changes.