there's a new SLOB regression - the attached config crashes with:
[   61.245190] rc.sysinit used greatest stack depth: 1680 bytes left

[   61.386859] list_add corruption. prev->next should be next (407d973c), but was 418cf818. (prev=41877098).

[   61.396328] ------------[ cut here ]------------

[   61.400910] kernel BUG at lib/list_debug.c:33!

[   61.405330] invalid opcode: 0000 [#1] DEBUG_PAGEALLOC
looks like memory corruption of some sort and it's reproducible. Picking

CONFIG_SLUB makes the crash go away. Booting v2.6.23 with the same

.config works fine.
bisection is difficult due to networking releated Kconfig problems: if i

put my .24 .config into .23 then the network drivers get lost. (we

should be more Kconfig-compatible between kernel releases, to ease

bisection efforts)
(full bootlog and config attached)
	Ingo

Hmmm, the changes in SLOB since v2.6.23 are all trivial. I'll try to

reproduce it with your config, but it doesn't seem promising.
--

Mathematics is the supreme nostalgia of our time.

-

Couldn't reproduce it here, let me know if you get anywhere with your bisect.
--

Mathematics is the supreme nostalgia of our time.

-

the bug went away - and the only thing i did was a networking config

tweak. So maybe something in networking corrupts memory? I'm not sure i

can restore the old state. (i had lots of problems with net interface

renaming not working in .24)
	Ingo

-

From: Ingo Molnar <mingo@elte.hu>
This wouldn't surprise me at all.
I think we can make some headway on this bug, the next time

you trigger it, if the list debugging was a little less terse.
For example, a backtrace and perhaps even feeding the bad list

pointers in question to the SLAB/SLUB debug helpers that can

identify a kmem cache from a given pointer would help.
Thanks.

-

He hit the bug using SLOB and there are no kmem (or any other) caches

in SLOB.
--

Mathematics is the supreme nostalgia of our time.

-

From: Matt Mackall <mpm@selenic.com>
That's unfortunate, is there any user tracking facility at

all?

-

No, the usual strategy for debugging problems -outside- SLOB is to

switch to another allocator with more extensive debugging facilities.
It is of course possible to add redzoning, last user, etc., but there

aren't many advantages to implementing these in SLOB compared to

switching allocators, unless the bug disappears in those other

allocators. In the case of random pointer fandango, such bugs are

likely to disappear when you turn on debugging anyway.
The most likely thing you'll hit in SLOB vs SLUB/SLAB is that SLOB

doesn't hand back power-of-two allocations for kmalloc. Instead, it

has 2-byte granularity on most machines. So small pointer overruns on

kmalloced objects will be somewhat more visible in SLOB than

SLAB/SLUB. I don't think SLAB/SLUB debugging can detect overruns

inside the not-requested-but-still-allocated region of objects.
I've implemented redzoning and various other debugging checks for

earlier versions of SLOB to find problems -in- the allocator, but

those won't apply to current SLOB (which can be considered v2).
--

Mathematics is the supreme nostalgia of our time.

-

From: Matt Mackall <mpm@selenic.com>
Ok, so the thing we still can do is do a dump_stack() at the

list debugging assertion trigger points.

-

ok, i'll first try to trigger it again.
it's a bzImage kernel with fixed order of eth0 and eth1 detection. What

i did was to twiddle the /etc/sysconfig/network-scripts/ifcfg-eth*

configs to address a network-does-not-show-up bug that .24 introduced.

The crash logs contain this:
 VFS: Mounted root (ext3 filesystem) readonly.

 Freeing unused kernel memory: 396k freed

 Write protecting the kernel read-only data: 2056k

 udev: renamed network interface eth1 to eth0

 udev: renamed network interface eth0_rename to eth1

 eth0: link down

 ADDRCONF(NETDEV_UP): eth0: link is not ready

 EXT3 FS on sda6, internal journal

 kjournald starting.  Commit interval 5 seconds
followed by the crash shortly afterwards (but not immediately). With the

non-crashing kernel i dont get those "renamed network interface"

messages.
network interface renaming has been a historic source of pain for me so

i frequently have to 'twiddle' the networking config to make it work

again on new kernels. Perhaps because i'm using bzImage kernels.

User-space is Fedora 8, so fairly recent.
	Ingo

-

I had implemented SLOB in userspace, so I resynched and think I

found your problem. Sorry for the attachment format -- this mailer

isn't the best. I'm really computer illiterate when it comes to

userspace...
Anyway, I'm really happy to see you're testing and using SLOB

upstream :) Is there any particular reason that you're using it?
Thanks,

Nick

i sometimes test SLOB for -rt, but this time it's the result of my

"automated random QA" effort, as part of arch/x86 maintainance/QA.
the main trick is to build and booting random "make randconfig"

bzImages. That finds build bugs and a good deal of boot hang and crash

bugs as well. (it also found a compiler bug already) I can build and

boot about 1000 random kernels in 24 hours, and it's all fully

automated. I usually run it overnight - when a kernel does not come up

due to a bootup hang or crash (or the kernel log signals any exception

condition) then the script stops and i can fix it in the morning.
The first step towards this was to get allyesconfig bzImage kernels to

build and boot fine. That effort took months (we had many problems in

this area) - i think you saw bugreports and fixes from me about that on

lkml.
Once that worked reasonably well i made a small Kconfig patch that

forcibly selects a "minimum set" of drivers and kernel subsystems that

are needed to boot up a testsystem. Once a "make allnoconfig" and a

"make allyesconfig" bzImage kernel boots up fine on the testbox all

randconfig configs "inbetween" are supposed to build and boot fine as

well.
I also have a patch that adds all the x86 boot options like nosmp,

maxcpus=1, nohz=off, hpet=disable to be selectable as .config options -

so those boot options are randomized as well.
I also have a small patch that disables half a dozen drivers/features

that are not expected to work out of box in a bzImage kernel. (such as

ISA drivers that assume the presence of hardware, or root filesystem

features such as NFSROOT)
the resulting make randconfig kernel still has 99% of the degrees of

freedom that a stock make randconfig kernel has, so by all practical

purposes it's a fully random kernel - it just happens to boot on my

testsystem all the time.
A successful bootup means the test system is able to boot up into a

stock Fedora 8 userspace and is able to bring up its network interfaces

a...
[ message continues ]

How complete is the QA testing?  I was reading this interesting thread

and it occurred to me that this sounds like a useful distributed

computing application.  ie a central server with all valid Kconfig

combinations (how many are there?) for a particular release (-rc or

otherwise) across all architectures.  These are allocated to clients on

request to be built / booted etc.  Any errors are fed back to the

central server.  I guess this would be a useful resource for

developers.  More importantly (and I don't know if this is the case

already!) a new Linux release (2.6.x) could be "certified" with some

level of testing on known hardware / architectures.
  tbh, I feel sorry for Ingo's machine compiling 1000 random kernels in

24h!  I'm surprised it hasn't called the Samaritans...
Dave.
-

Well, my hat's off to you. Actually I was more wondering how it is

that you're catching SLOB bugs ;) so it seems your test setup is much

more useful than just to test the x86 arch code...

-

that did the trick! Nick, find an updated patch below. (reference to the

bugzilla added.)
	Ingo
-------------------->

Subject: slob: fix memory corruption

From: Nick Piggin <npiggin@suse.de>
Previously, it would be possible for prev->next to point to

&free_slob_pages, and thus we would try to move a list onto itself, and

bad things would happen.
It seems a bit hairy to be doing list operations with the list marker as

an entry, rather than a head, but...
this resolves the following crash:
  http://bugzilla.kernel.org/show_bug.cgi?id=9379
Signed-off-by: Nick Piggin <npiggin@suse.de>

Signed-off-by: Ingo Molnar <mingo@elte.hu>

---

 mm/slob.c |    3 ++-

 1 file changed, 2 insertions(+), 1 deletion(-)
Index: linux/mm/slob.c

===================================================================

--- linux.orig/mm/slob.c

+++ linux/mm/slob.c

@@ -321,7 +321,8 @@ static void *slob_alloc(size_t size, gfp

 		/* Improve fragment distribution and reduce our average

 		 * search time by starting our next search here. (see

 		 * Knuth vol 1, sec 2.5, pg 449) */

-		if (free_slob_pages.next != prev->next)

+		if (prev != free_slob_pages.prev &&

+				free_slob_pages.next != prev->next)

 			list_move_tail(&free_slob_pages, prev->next);

 		break;

 	}

-

Signed-off-by: Matt Mackall <mpm@selenic.com>
--

Mathematics is the supreme nostalgia of our time.

-

btw., exactly how did you find this bug? User-space simulation of SLOB?
	Ingo

-

Yes. It was very useful in developing the improvements to the freelist

handling. The only reason why I don't release/run the code more often

is that my test harness work is pretty ugly (ie. it isn't just a simple

cp mm/slob.c ../blah/).
After that, just a loop of N iterations, within each iteration, there is

a chance of a single allocation of a random size, a single free of a

random outstanding allocation, a run of allocating MAX allocations, or

a run of freeing all previously allocated memory. It's a bit crude, but

it showed up your list head corruption in a second or two.

-

From: Ingo Molnar <mingo@elte.hu>
Yeah I wish udev would just leave the damn devices alone.
It even does things like try to rename a network device to the same

name it already has, and other strange stuff.
But that log difference is a good clue.
Because udev can try to rename a network device stupidly to a name the

device already has we added a patch to just short circuit this case in

the networking.  We did this because otherwise the generic device

layer gives an ugly stack backtrace via dev_rename().
Therefore, you might want to see if reverting that patch (attached

below) has some effect, once you are able to trigger it again.
Thanks Ingo.
commit c8d90dca3211966ba5189e0f3d4bccd558d9ae08

Author: Stephen Hemminger <shemminger@linux-foundation.org>

Date:   Fri Oct 26 03:53:42 2007 -0700
    [NET] dev_change_name: ignore changes to same name
    Prevent error/backtrace from dev_rename() when changing

    name of network device to the same name. This is a common

    situation with udev and other scripts that bind addr to device.
    Signed-off-by: Stephen Hemminger <shemminger@linux-foundation.org>

    Signed-off-by: David S. Miller <davem@davemloft.net>
diff --git a/net/core/dev.c b/net/core/dev.c

index f1647d7..ddfef3b 100644

--- a/net/core/dev.c

+++ b/net/core/dev.c

@@ -883,6 +883,9 @@ int dev_change_name(struct net_device *dev, char *newname)

 	if (!dev_valid_name(newname))

 		return -EINVAL;
+	if (strncmp(newname, dev->name, IFNAMSIZ) == 0)

+		return 0;

+

 	memcpy(oldname, dev->name, IFNAMSIZ);
 	if (strchr(newname, '%')) {

-

just to confuse things, i just got a crash with the twiddled network

setup :-/ I have reverted the same-name optimization and have got a

similar crash again. So this angle is a red herring.
now that it's reproducible again i'll try more direct debugging.

(Networking might not even be the cause of this - that was just a quick

first impression that i had.)
Btw., the .config is the result of automated "make randconfig" x86

bootup testing QA, so there might be weird combinations in the .config.

That's how SLOB got randomly enabled in the first place, i dont normally

use SLOB kernels.
	Ingo

-

From: Ingo Molnar <mingo@elte.hu>
Check out Nick Piggin's SLOB bug fix, I think it is a good

lead :-)

-

It's also pretty easy to add some debugging code to make SLOB walk all

its lists at alloc/free time.
--

Mathematics is the supreme nostalgia of our time.

-

Quite possible. SLOB is more sensitive to off by one bugs because it

doesn't have the power-of-two buckets that SLAB/SLUB have. IIRC,

SLAB/SLUB's debugging features won't detect when you request 28 bytes,

get 32, then overwrite byte 29. But that will damage other objects or

the free list in SLOB.
But this isn't the per-page SLOB list that's getting clobbered, this

is the list of pages held in struct page.
--

Mathematics is the supreme nostalgia of our time.

-

Fixed and merged in Linus's tree as:

- 50d84c2dc00e48ff9ba018ed0dd23276cf79e566

- b9d04e2401bf308df921d3bbbdacab40fadc27bb
--

Ueimor

-

should be fixed by acpi-make-acpi_procfs-default-to-y.patch in -mm. (not

yet upstream i think, but should before v2.6.24)
	Ingo

-

As I said before pata_acpi was not present in 2.6.23 -> Not a regression.

WONTFIX for 2.6.24. Not actually clear it is even a bug, the interactions

between using pata_acpi and simplex controllers are not documented
Tejun is looking into this - its not trivial so may be 2.6.25 material.
Not sure who is handling this now - seems to be an IRQ routing bug

introduced in -rc2. I've got a pile of similar breakage reports for

random ATA controllers.
Thanks
Alan

-

http://lkml.org/lkml/2007/10/12/537
The regression itself has been foreseen a month ago and it is quite
We may fix the regression in a bit different way for now and give Tejun

some more time for the complete rework of the cable detection code.
[PATCH] pata_amd/pata_via: de-couple programming of PIO/MWDMA and UDMA timings
* Don't program UDMA timings when programming PIO or MWDMA modes.
  This has also a nice side-effect of fixing regression added by commit

  681c80b5d96076f447e8101ac4325c82d8dce508 ("libata: correct handling of

  SRST reset sequences") (->set_piomode method for PIO0 is called before

  ->cable_detect method which checks UDMA timings to get the cable type).
* Bump driver version.
Signed-off-by: Bartlomiej Zolnierkiewicz <bzolnier@gmail.com>

---

Untested, please don't merge until it is confirmed to fix the problem.
 drivers/ata/pata_amd.c |    5 +++--

 drivers/ata/pata_via.c |    4 ++--

 2 files changed, 5 insertions(+), 4 deletions(-)
Index: b/drivers/ata/pata_amd.c

===================================================================

--- a/drivers/ata/pata_amd.c

+++ b/drivers/ata/pata_amd.c

@@ -25,7 +25,7 @@

 #include <linux/libata.h>
 #define DRV_NAME "pata_amd"

-#define DRV_VERSION "0.3.9"

+#define DRV_VERSION "0.3.10"
 /**

  *	timing_setup		-	shared timing computation and load

@@ -115,7 +115,8 @@ static void timing_setup(struct ata_port

 	}
 	/* UDMA timing */

-	pci_write_config_byte(pdev, offset + 0x10 + (3 - dn), t);

+	if (at.udma)

+		pci_write_config_byte(pdev, offset + 0x10 + (3 - dn), t);

 }
 /**

Index: b/drivers/ata/pata_via.c

===================================================================

--- a/drivers/ata/pata_via.c

+++ b/drivers/ata/pata_via.c

@@ -63,7 +63,7 @@

 #include <linux/dmi.h>
 #define DRV_NAME "pata_via"

-#define DRV_VERSION "0.3.2"

+#define DRV_VERSION "0.3.3"
 /*

  *	The following comes directly from Vojtech Pavlik's ide/pci/via82cxxx

@@ -296,7 +296,7 @@ static void...
[ message continues ]

I'm not sure this helps as if the ACPI _GTF method is looking at the

flags and stuff but it has to be worth a try.
Works for me as a 2.6.24 band aid

-

I'm looking at that "Untested, please don't merge until it is confirmed to

fix the problem." comment..
Thomas, can you please give it a try, let us know?
Thanks

-

I can confirm that the patch "pata_amd/pata_via: de-couple programming

of PIO/MWDMA and UDMA timings" does fix my issue "pata_amd fails to

detect 80-pin wire".

-

OK, dropped.

-