Home » Mailing list archives » freebsd-current » 2007 » October » 15

Re: panic: ffs_blkfree: freeing free block

Previous thread: Re: JMicron RAID problems by Andrey V. Elsukov on Sunday, September 30, 2007 - 6:53 am. (4 messages)

Next thread: patch: change in acpi taskq behavior by Nate Lawson on Sunday, September 30, 2007 - 1:10 pm. (6 messages)
[view original message]
From: Joe Marcus Clarke
Subject: panic: ffs_blkfree: freeing free block
Date: Sunday, September 30, 2007 - 11:03 am

I just upgraded my i386 Tinderbox machine last night, and a package
build triggered this panic.  I've been using this machine for Tinderbox
for a year or so now, and this is the first such panic I received.  The
disk in question is 61% used, and I am building locally using loopback
NFS mounts.

FreeBSD fugu.marcuscom.com 7.0-CURRENT FreeBSD 7.0-CURRENT #12: Sun Sep
30 02:14:27 EDT 2007
gnome@fugu.marcuscom.com:/space2/obj/usr/src/sys/FUGU  i386

I was previously running a kernel from September 19 without any
problems.

dev =3D ad6s1e, block =3D 22287592, fs =3D /space
panic: ffs_blkfree: freeing free block
cpuid =3D 0
KDB: enter: panic
Physical memory: 2038 MB
Dumping 351 MB: 336 320 304 288 272 256 240 224 208 192 176 160 144 128
112 96 80 64 48 32 16

#0  doadump () at pcpu.h:195
195     pcpu.h: No such file or directory.
        in pcpu.h
(kgdb) bt
#0  doadump () at pcpu.h:195
#1  0xc045ed89 in db_fncall (dummy1=3D-236168880, dummy2=3D0, dummy3=3D70,=20
    dummy4=3D0xf1ec58bc "=F0\022F=C0") at /usr/src/sys/ddb/db_command.c:486
#2  0xc045f2f5 in db_command_loop ()
at /usr/src/sys/ddb/db_command.c:401
#3  0xc0460d25 in db_trap (type=3D3, code=3D0)
at /usr/src/sys/ddb/db_main.c:222
#4  0xc0595316 in kdb_trap (type=3D3, code=3D0, tf=3D0xf1ec5a64)
    at /usr/src/sys/kern/subr_kdb.c:502
#5  0xc072d32f in trap (frame=3D0xf1ec5a64)
at /usr/src/sys/i386/i386/trap.c:621
#6  0xc07131fb in calltrap () at /usr/src/sys/i386/i386/exception.s:139
#7  0xc0595492 in kdb_enter (msg=3D0xc07706ef "panic") at cpufunc.h:60
#8  0xc056dda4 in panic (fmt=3D0xc077ee64 "ffs_blkfree: freeing free
block")
    at /usr/src/sys/kern/kern_shutdown.c:547
#9  0xc0698954 in ffs_blkfree (ump=3D0xc5140300, fs=3D0xc508a800,=20
    devvp=3D0xc5167990, bno=3D22287592, size=3D16384, inum=3D5566594)
    at /usr/src/sys/ufs/ffs/ffs_alloc.c:1893
#10 0xc06ac318 in indir_trunc (freeblks=3D0xcf6d4000, dbn=3D89113600,
level=3D0,=20
    lbn=3D12, countp=3D0xf1ec5c4c)
at ...
[ message continues ]
[view original message]
From: Rolf Witt
Subject: Re: panic: ffs_blkfree: freeing free block
Date: Sunday, September 30, 2007 - 12:16 pm

block

The kernel on my PC panics by reading the UPDATING-File with less after
executing a portsnap update.

Sorry, i know thats are not very helpful informations, but i lost the
vmcore and all debug output.

rowi
_______________________________________________
freebsd-current@freebsd.org mailing list
http://lists.freebsd.org/mailman/listinfo/freebsd-current
To unsubscribe, send any mail to "freebsd-current-unsubscribe@freebsd.org"

[ message continues ]
[view original message]
From: Arjan van Leeuwen
Subject: Re: panic: ffs_blkfree: freeing free block
Date: Monday, October 1, 2007 - 6:28 am

Same here:

#0 doadump () at pcpu.h:194
#1 0x0000000000000004 in ?? ()
#2 0xffffffff80472c09 in boot (howto=260)
  at /usr/src/sys/kern/kern_shutdown.c:409
#3 0xffffffff8047300d in panic (fmt=0x104 <Address 0x104 out of bounds>)
  at /usr/src/sys/kern/kern_shutdown.c:563
#4 0xffffffff80642135 in ffs_blkfree (ump=Variable "ump" is not available.
)
  at /usr/src/sys/ufs/ffs/ffs_alloc.c:1893
#5 0xffffffff80651dc3 in indir_trunc (freeblks=0xffffff001544d500,
dbn=Variable "dbn" is not available.
)
  at /usr/src/sys/ufs/ffs/ffs_softdep.c:2896
#6 0xffffffff8065227f in handle_workitem_freeblocks (
  freeblks=0xffffff001544d500, flags=0)
  at /usr/src/sys/ufs/ffs/ffs_softdep.c:2746
#7 0xffffffff80653853 in process_worklist_item (mp=0xffffff0001478978,
flags=Variable "flags" is not available.
)
  at /usr/src/sys/ufs/ffs/ffs_softdep.c:963
#8 0xffffffff80654867 in softdep_process_worklist (mp=0xffffff0001478978,
  full=0) at /usr/src/sys/ufs/ffs/ffs_softdep.c:847
#9 0xffffffff806567d7 in softdep_flush ()
  at /usr/src/sys/ufs/ffs/ffs_softdep.c:758
#10 0xffffffff80454723 in fork_exit (
  callout=0xffffffff806566b0 <softdep_flush>, arg=0x0,
  frame=0xffffffffac3a8c80) at /usr/src/sys/kern/kern_fork.c:796
#11 0xffffffff806dbb8e in fork_trampoline ()
  at /usr/src/sys/amd64/amd64/exception.S:397


_______________________________________________
freebsd-current@freebsd.org mailing list
http://lists.freebsd.org/mailman/listinfo/freebsd-current
To unsubscribe, send any mail to "freebsd-current-unsubscribe@freebsd.org"

[ message continues ]
[view original message]
From: Adam McDougall
Subject: Re: panic: ffs_blkfree: freeing free block
Date: Monday, October 1, 2007 - 2:32 pm

I was running rsync, got about half way done copying 100G from a remote server
to a local gjournal ffs however it looks like the panic did NOT occur on the
drive I was running rsync to:

Unread portion of the kernel message buffer:
dev = da1s1g, block = 27232072, fs = /data
panic: ffs_blkfree: freeing free block
cpuid = 0
Uptime: 4d5h39m49s
Physical memory: 2030 MB
Dumping 352 MB: 337 321 305 289 273 257 241 225 209 193 177 161 145 129 113 97 81 65 49 33 17 1

#0  doadump () at pcpu.h:195
195     pcpu.h: No such file or directory.
        in pcpu.h
(kgdb) bt
#0  doadump () at pcpu.h:195
#1  0xc0589b47 in boot (howto=260) at /usr/src/sys/kern/kern_shutdown.c:409
#2  0xc0589e3b in panic (fmt=Variable "fmt" is not available.
) at /usr/src/sys/kern/kern_shutdown.c:563
#3  0xc0651fa4 in ffs_blkfree (ump=0xc5a6e600, fs=0xc5974000, devvp=0xc59ab220, bno=27232072, 
    size=16384, inum=6810668) at /usr/src/sys/ufs/ffs/ffs_alloc.c:1893
#4  0xc0665968 in indir_trunc (freeblks=0xcc171500, dbn=108840064, level=0, lbn=12, countp=0xe631cc4c)
    at /usr/src/sys/ufs/ffs/ffs_softdep.c:2896
#5  0xc0665c47 in handle_workitem_freeblocks (freeblks=0xcc171500, flags=0)
    at /usr/src/sys/ufs/ffs/ffs_softdep.c:2746
#6  0xc06673de in process_worklist_item (mp=0xc5a74000, flags=Variable "flags" is not available.
)
    at /usr/src/sys/ufs/ffs/ffs_softdep.c:963
#7  0xc0668402 in softdep_process_worklist (mp=0xc5a74000, full=0)
    at /usr/src/sys/ufs/ffs/ffs_softdep.c:847
#8  0xc066a8cd in softdep_flush () at /usr/src/sys/ufs/ffs/ffs_softdep.c:758
#9  0xc0568369 in fork_exit (callout=0xc066a450 <softdep_flush>, arg=0x0, frame=0xe631cd38)
    at /usr/src/sys/kern/kern_fork.c:796
#10 0xc06cff90 in fork_trampoline () at /usr/src/sys/i386/i386/exception.s:205


On Mon, Oct 01, 2007 at 03:28:57PM +0200, Arjan van Leeuwen wrote:

  Same here:
  
  #0 doadump () at pcpu.h:194
  #1 0x0000000000000004 in ?? ()
  #2 0xffffffff80472c09 in boot (howto=260)
    at /usr/src/sys/kern/kern_shutdown.c:409
 ...
[ message continues ]
[view original message]
From: Arjan van Leeuwen
Subject: Re: panic: ffs_blkfree: freeing free block
Date: Wednesday, October 3, 2007 - 6:21 am

I'm getting this panic several times a day now. How can I help in further
debugging this?
Also, I note that everytime I panic, my currently opened files are reduced
to 0 bytes. Is that expected?
Arjan

_______________________________________________
freebsd-current@freebsd.org mailing list
http://lists.freebsd.org/mailman/listinfo/freebsd-current
To unsubscribe, send any mail to "freebsd-current-unsubscribe@freebsd.org"

[ message continues ]
[view original message]
From: Peter Jeremy
Subject: Re: panic: ffs_blkfree: freeing free block
Date: Wednesday, October 3, 2007 - 12:56 pm

On 2007-Oct-03 15:21:15 +0200, Arjan van Leeuwen <avleeuwen@gmail.com> wrot=

It depends, are you talking about files being read or only files being
written?  If this is just affecting writes, then this is a side-effect
of the stdio buffering, together with the write-back nature of the UFS
buffer cache in conjunction with soft-updates: Data on disk is
typically about 30 seconds behind reality and the file contents will
always be behind the file itself.  It is quite normal for recently
written files (or files currently being written) to be truncated on
disk following a crash.

--=20
Peter Jeremy

[ message continues ]
[view original message]
From: Arjan van Leeuwen
Subject: Re: panic: ffs_blkfree: freeing free block
Date: Wednesday, October 3, 2007 - 1:06 pm

Yep, these are recently written files indeed. Usually the files I had open
in my editor while it paniced, files that I save often.
Oh well... I'm setting my hopes on this panic being resolved soon then :).
Thanks for the explanation.
_______________________________________________
freebsd-current@freebsd.org mailing list
http://lists.freebsd.org/mailman/listinfo/freebsd-current
To unsubscribe, send any mail to "freebsd-current-unsubscribe@freebsd.org"

[ message continues ]
[view original message]
From: Eric Anderson
Subject: Re: panic: ffs_blkfree: freeing free block
Date: Thursday, October 4, 2007 - 6:48 pm

Can anyone provide access to the core dumps?

Eric


_______________________________________________
freebsd-current@freebsd.org mailing list
http://lists.freebsd.org/mailman/listinfo/freebsd-current
To unsubscribe, send any mail to "freebsd-current-unsubscribe@freebsd.org"

[ message continues ]
[view original message]
From: Pieter de Goeje
Subject: Re: panic: ffs_blkfree: freeing free block
Date: Wednesday, October 10, 2007 - 5:15 pm

Hi Eric,

I've put a coredump and kernel.debug at 
http://lux.student.utwente.nl/~pyotr/panic/ along with the dmesg and kernel 
configuration file.

uname -a:
FreeBSD unforgiven.student.utwente.nl 7.0-CURRENT FreeBSD 7.0-CURRENT #8: Mon 
Oct  8 01:48:17 CEST 2007     
pyotr@unforgiven.student.utwente.nl:/usr/obj/usr/src/sys/UNFORGIVEN  amd64

I am not sure about the security impact of putting a coredump in a public 
place, so I didn't cc current. ("Somewhat" less public this way...)

The panic often occurs while doing
# portsnap fetch update
# portversion -vl \<

If you need anything else please let me know.

With kind regards,
Pieter de Goeje
_______________________________________________
freebsd-current@freebsd.org mailing list
http://lists.freebsd.org/mailman/listinfo/freebsd-current
To unsubscribe, send any mail to "freebsd-current-unsubscribe@freebsd.org"

[ message continues ]
[view original message]
From: Pieter de Goeje
Subject: Re: panic: ffs_blkfree: freeing free block
Date: Thursday, October 11, 2007 - 3:25 am

This address is no longer valid. Eric, if you are reading this, I've sent you 

_______________________________________________
freebsd-current@freebsd.org mailing list
http://lists.freebsd.org/mailman/listinfo/freebsd-current
To unsubscribe, send any mail to "freebsd-current-unsubscribe@freebsd.org"

[ message continues ]
[view original message]
From: Pieter de Goeje
Subject: Re: panic: ffs_blkfree: freeing free block
Date: Monday, October 1, 2007 - 7:04 am

"me too": I have seen this panic on two different machines. Both times it was 
triggered by doing a partial restore of a filesystem and then doing some 
reading of the extracted files. Both machines are SMP amd64.
-
Pieter de Goeje
_______________________________________________
freebsd-current@freebsd.org mailing list
http://lists.freebsd.org/mailman/listinfo/freebsd-current
To unsubscribe, send any mail to "freebsd-current-unsubscribe@freebsd.org"

[ message continues ]
[view original message]
From: Rolf Witt
Subject: Re: panic: ffs_blkfree: freeing free block
Date: Wednesday, October 3, 2007 - 6:55 pm

Here the backtrace from the last panic:

# kgdb kernel.debug /var/crash/vmcore.0
kgdb: kvm_nlist(_stopped_cpus):
kgdb: kvm_nlist(_stoppcbs):
[GDB will not be able to debug user-mode threads:
/usr/lib/libthread_db.so: Undefined symbol "ps_pglobal_lookup"]
GNU gdb 6.1.1 [FreeBSD]
Copyright 2004 Free Software Foundation, Inc.
GDB is free software, covered by the GNU General Public License, and you are
welcome to change it and/or distribute copies of it under certain
conditions.
Type "show copying" to see the conditions.
There is absolutely no warranty for GDB.  Type "show warranty" for details.
This GDB was configured as "i386-marcel-freebsd".

Unread portion of the kernel message buffer:
dev = ad4s1f, block = 26012752, fs = /usr
panic: ffs_blkfree: freeing free block
Uptime: 1d5h6m39s
Physical memory: 631 MB
Dumping 152 MB: 137 121 105 89 73 57 41 25 9

#0  doadump () at pcpu.h:195
195             __asm __volatile("movl %%fs:0,%0" : "=r" (td));
(kgdb) backtrace
#0  doadump () at pcpu.h:195
#1  0xc06328d3 in boot (howto=260) at /usr/src/sys/kern/kern_shutdown.c:409
#2  0xc0632ad4 in panic (fmt=Variable "fmt" is not available.
) at /usr/src/sys/kern/kern_shutdown.c:563
#3  0xc07d48f0 in ffs_blkfree (ump=0xc323ab00, fs=0xc321c800,
    devvp=0xc328add0, bno=26012752, size=16384, inum=6506870)
    at /usr/src/sys/ufs/ffs/ffs_alloc.c:1893
#4  0xc07e8168 in indir_trunc (freeblks=0xc3c1bd00, dbn=103948864, level=0,
    lbn=12, countp=0xd9177c4c) at /usr/src/sys/ufs/ffs/ffs_softdep.c:2896
#5  0xc07e8430 in handle_workitem_freeblocks (freeblks=0xc3c1bd00, flags=0)
    at /usr/src/sys/ufs/ffs/ffs_softdep.c:2746
#6  0xc07e9cb8 in process_worklist_item (mp=0xc324b7d4, flags=Variable
"flags" is not available.
)
    at /usr/src/sys/ufs/ffs/ffs_softdep.c:963
#7  0xc07ead71 in softdep_process_worklist (mp=0xc324b7d4, full=0)
    at /usr/src/sys/ufs/ffs/ffs_softdep.c:847
#8  0xc07ed2ea in softdep_flush () at /usr/src/sys/ufs/ffs/ffs_softdep.c:758
#9  0xc0612211 in fork_exit ...
[ message continues ]
[view original message]
From: Rolf Witt
Subject: Re: panic: ffs_blkfree: freeing free block
Date: Saturday, October 13, 2007 - 7:57 am

Backtrace from crash tonight (after portsnap cron):

# kgdb kernel.debug /var/crash/vmcore.1
kgdb: kvm_nlist(_stopped_cpus):
kgdb: kvm_nlist(_stoppcbs):
[GDB will not be able to debug user-mode threads:
/usr/lib/libthread_db.so: Unde                       fined symbol
"ps_pglobal_lookup"]
GNU gdb 6.1.1 [FreeBSD]
Copyright 2004 Free Software Foundation, Inc.
GDB is free software, covered by the GNU General Public License, and you are
welcome to change it and/or distribute copies of it under certain
conditions.
Type "show copying" to see the conditions.
There is absolutely no warranty for GDB.  Type "show warranty" for details.
This GDB was configured as "i386-marcel-freebsd".

Unread portion of the kernel message buffer:
dev = ad4s1f, block = 26098960, fs = /usr
panic: ffs_blkfree: freeing free block
Uptime: 1d10h32m17s
Physical memory: 631 MB
Dumping 150 MB: 135 119 103 87 71 55 39 23 7

#0  doadump () at pcpu.h:195
195             __asm __volatile("movl %%fs:0,%0" : "=r" (td));
(kgdb) backtrace
#0  doadump () at pcpu.h:195
#1  0xc06328a3 in boot (howto=260) at /usr/src/sys/kern/kern_shutdown.c:409
#2  0xc0632aa4 in panic (fmt=Variable "fmt" is not available.
) at /usr/src/sys/kern/kern_shutdown.c:563
#3  0xc07d4d10 in ffs_blkfree (ump=0xc323ab00, fs=0xc321c800,
    devvp=0xc328add0, bno=26098960, size=16384, inum=6500370)
    at /usr/src/sys/ufs/ffs/ffs_alloc.c:1893
#4  0xc07e8588 in indir_trunc (freeblks=0xc3e12500, dbn=103954304, level=0,
    lbn=12, countp=0xd9177c4c) at /usr/src/sys/ufs/ffs/ffs_softdep.c:2896
#5  0xc07e8850 in handle_workitem_freeblocks (freeblks=0xc3e12500, flags=0)
    at /usr/src/sys/ufs/ffs/ffs_softdep.c:2746
#6  0xc07ea0d8 in process_worklist_item (mp=0xc324b7d4, flags=Variable
"flags" is not available.
)
    at /usr/src/sys/ufs/ffs/ffs_softdep.c:963
#7  0xc07eb191 in softdep_process_worklist (mp=0xc324b7d4, full=0)
    at /usr/src/sys/ufs/ffs/ffs_softdep.c:847
#8  0xc07ed70a in softdep_flush () at ...
[ message continues ]
[view original message]
From: Pieter de Goeje
Subject: Re: panic: ffs_blkfree: freeing free block
Date: Saturday, October 13, 2007 - 9:57 am

Eric sent me this patch: 
http://www.googlebit.com/freebsd/patches/ffs_softdep.c-patch
which seems to be working great so far. I am still testing it, but I think it 
fixed the problem.
To apply, cd /usr/src; patch < /path/to/patch and then rebuild the kernel.

Cheers,
Pieter de Goeje
_______________________________________________
freebsd-current@freebsd.org mailing list
http://lists.freebsd.org/mailman/listinfo/freebsd-current
To unsubscribe, send any mail to "freebsd-current-unsubscribe@freebsd.org"

[ message continues ]
[view original message]
From: Oliver Herold
Subject: Re: panic: ffs_blkfree: freeing free block
Date: Saturday, October 13, 2007 - 10:20 am

Thanks for the patch, I'll try it.

Cheers, Oliver


-- 
I was in accord with the system so long as it
permitted me to function effectively.
		-- Albert Speer
_______________________________________________
freebsd-current@freebsd.org mailing list
http://lists.freebsd.org/mailman/listinfo/freebsd-current
To unsubscribe, send any mail to "freebsd-current-unsubscribe@freebsd.org"

[ message continues ]
[view original message]
From: Rolf Witt
Subject: Re: panic: ffs_blkfree: freeing free block
Date: Saturday, October 13, 2007 - 1:14 pm

Thank you. I will try this patch.

cu
Rowi

_______________________________________________
freebsd-current@freebsd.org mailing list
http://lists.freebsd.org/mailman/listinfo/freebsd-current
To unsubscribe, send any mail to "freebsd-current-unsubscribe@freebsd.org"

[ message continues ]
[view original message]
From: Eric Anderson
Subject: Re: panic: ffs_blkfree: freeing free block
Date: Saturday, October 13, 2007 - 6:29 pm

It doesn't actually 'fix' the problem, but I think it helps identify it.
I'm not 100% certain if this is the right fix our not, but so far 
feedback has been good when running with this patch.

Can somebody confirm that this patch is ok?

Eric

_______________________________________________
freebsd-current@freebsd.org mailing list
http://lists.freebsd.org/mailman/listinfo/freebsd-current
To unsubscribe, send any mail to "freebsd-current-unsubscribe@freebsd.org"

[ message continues ]
[view original message]
From: Pieter de Goeje
Subject: Re: panic: ffs_blkfree: freeing free block
Date: Saturday, October 13, 2007 - 7:08 pm

Can you elaborate on what this patch exactly does / what the problem is?

Pieter
_______________________________________________
freebsd-current@freebsd.org mailing list
http://lists.freebsd.org/mailman/listinfo/freebsd-current
To unsubscribe, send any mail to "freebsd-current-unsubscribe@freebsd.org"

[ message continues ]
[view original message]
From: Rako
Subject: Re: panic: ffs_blkfree: freeing free block
Date: Monday, October 15, 2007 - 9:11 am

Hello, I apply the patch 
(http://www.googlebit.com/freebsd/patches/ffs_softdep.c-patch) but I got 
the panic again. (with portsnap update)

How can I help for this bug (?) fixed?
Thanks,
Javier

FreeBSD odin 7.0-PRERELEASE FreeBSD 7.0-PRERELEASE #0: Sat Oct 13 
15:42:34 ART 2007

kgdb: kvm_nlist(_stopped_cpus):
kgdb: kvm_nlist(_stoppcbs):
[GDB will not be able to debug user-mode threads: 
/usr/lib/libthread_db.so: Undefined symbol "ps_pglobal_lookup"]
GNU gdb 6.1.1 [FreeBSD]
Copyright 2004 Free Software Foundation, Inc.
GDB is free software, covered by the GNU General Public License, and you are
welcome to change it and/or distribute copies of it under certain 
conditions.
Type "show copying" to see the conditions.
There is absolutely no warranty for GDB.  Type "show warranty" for details.
This GDB was configured as "i386-marcel-freebsd".

Unread portion of the kernel message buffer:
dev = ad4s1f, block = 5515224, fs = /usr
panic: ffs_blkfree: freeing free block
KDB: stack backtrace:
db_trace_self_wrapper(c0a5d4aa,d301aad0,c0787e3a,c0a5b8b4,c0b59b40,...) 
at db_trace_self_wrapper+0x26
kdb_backtrace(c0a5b8b4,c0b59b40,c0a6c5bb,d301aadc,d301aadc,...) at 
kdb_backtrace+0x29
panic(c0a6c5bb,c2cc2b78,5427d8,0,c2d528d4,...) at panic+0xaa
ffs_blkfree(c2db3a00,c2d52800,c2d76bb0,5427d8,0,...) at ffs_blkfree+0x433
indir_trunc(0,c,0,d301ac4c,c1455718,...) at indir_trunc+0x3a8
handle_workitem_freeblocks(0,d301ac84,2,c07b5200,c2d2e210,...) at 
handle_workitem_freeblocks+0x1e0
process_worklist_item(d301acc8,c07fe20e,c2d3b538,2001,c2d3b568,...) at 
process_worklist_item+0x22e
softdep_process_worklist(c2d3b538,0,c0b5eae0,c2d2e210,3e8,...) at 
softdep_process_worklist+0x81
softdep_flush(0,d301ad38,c9e1dfe1,a9fd7d80,262e286a,...) at 
softdep_flush+0x46a
fork_exit(c08f7d70,0,d301ad38) at fork_exit+0x94
fork_trampoline() at fork_trampoline+0x8
--- trap 0, eip = 0, esp = 0xd301ad70, ebp = 0 ---
Uptime: 1d15h29m16s
Physical memory: 435 ...
[ message continues ]
[view original message]
From: Eric Anderson
Subject: Re: panic: ffs_blkfree: freeing free block
Date: Monday, October 15, 2007 - 7:53 pm

I think the problem is that blocks are being put in the worklist twice, 
but I'm not certain why yet.  The patch reduces the chance of this 
happening by more often removing the block from the worklist instead of 
leaving it on.  I think actually the patch is hiding the real issue more 
than anything, which is why I said it isn't fixing the problem, but 
because of the reports I've seen, I think we're sniffing in the right 
area.  I'll be looking more into this soon, when I get another few 
minutes of free time and my debugger..

Eric

_______________________________________________
freebsd-current@freebsd.org mailing list
http://lists.freebsd.org/mailman/listinfo/freebsd-current
To unsubscribe, send any mail to "freebsd-current-unsubscribe@freebsd.org"

[ message continues ]
[view original message]
From: Arjan van Leeuwen
Subject: Re: panic: ffs_blkfree: freeing free block
Date: Tuesday, October 16, 2007 - 3:20 am

I actually seem to be getting the panic more often with the patch (i.e.
twice in just 30 minutes with the patch, about twice per day without the
patch), while doing the same things as usual. The panic usually happens when
I'm compiling or linking. Is that strange?
Arjan
_______________________________________________
freebsd-current@freebsd.org mailing list
http://lists.freebsd.org/mailman/listinfo/freebsd-current
To unsubscribe, send any mail to "freebsd-current-unsubscribe@freebsd.org"

[ message continues ]
[view original message]
From: Kostik Belousov
Subject: Re: panic: ffs_blkfree: freeing free block
Date: Tuesday, October 16, 2007 - 4:29 am

Patch is wrong. Actually, it does put the dirrem to the proccessing twice
when xp !=3D NULL.

[ message continues ]
[view original message]
From: Eric Anderson
Subject: Re: panic: ffs_blkfree: freeing free block
Date: Wednesday, October 17, 2007 - 5:05 am

Yes, exactly why I said 'it doesn't fix the problem'.  In fact, I 
actually did not post it to the list, and didn't intend for it to be 
posted.  I was hoping for feedback for further examination.

Kostik - do you have any ideas on what is going on here?  Did you see 
the message in another thread that looked similar (it was regarding 
gjournal)?

Eric

_______________________________________________
freebsd-current@freebsd.org mailing list
http://lists.freebsd.org/mailman/listinfo/freebsd-current
To unsubscribe, send any mail to "freebsd-current-unsubscribe@freebsd.org"

[ message continues ]
[view original message]
From: Pieter de Goeje
Subject: Re: panic: ffs_blkfree: freeing free block
Date: Wednesday, October 17, 2007 - 5:43 am

I had (and still have) trouble reproducing the panic with the patch. I tried 
to write a program which would trigger the panic, but I failed (hence my 
question on what problem the patch masks so my test tool could focus on the 
problem area more). I ended up trying to write something similar to portsnap 
followed by portsdb -u. Now that portsnap works again for -CURRENT, I am able 
to reproduce the panic much more reliably.

I posted the patch to current@ because of my feedback being useless for the 
aforementioned reasons and I hoped someone else had a better testcase.

Regards,
_______________________________________________
freebsd-current@freebsd.org mailing list
http://lists.freebsd.org/mailman/listinfo/freebsd-current
To unsubscribe, send any mail to "freebsd-current-unsubscribe@freebsd.org"

[ message continues ]
[view original message]
From: Kostik Belousov
Subject: Re: panic: ffs_blkfree: freeing free block
Date: Thursday, October 18, 2007 - 2:39 am

No, I have no idea. BTW, some further information from the people that
experience the problem could be helpful. For instance,
UFS1/UFS2
size of the problematic fs/% of the space used
are quotas enabled in kernel/active for the fs ?
are softupdates on ?
what block number is reported as being freed twice ? is it always the same ?
what is the full fsck diagnostic after reboot ?

The range of questions is so wide because I try to somewhat localize the
search field down from "anywhere".

[ message continues ]
[view original message]
From: Arjan van Leeuwen
Subject: Re: panic: ffs_blkfree: freeing free block
Date: Thursday, October 18, 2007 - 2:59 am

Hey Kostik,


UFS2, filesystem created with FreeBSD 6-STABLE

size of the problematic fs/% of the space used

 /dev/ad4s1g 90613368 22619950 60744350 27% /home

are quotas enabled in kernel/active for the fs ?

No to both.

are softupdates on ?

Yes. Turning them off seems to fix the problem (fingers crossed - I only
turned it off this morning, didn't have a panic yet)


I'm in X when it happens, so I can't see the message. It doesn't say which
block is being freed on reboot.
I haven't saved the fsck output either. I can turn softupdates back on
tonight and reproduce the panic to get a full fsck log, if you want me to.

np :)

Arjan
_______________________________________________
freebsd-current@freebsd.org mailing list
http://lists.freebsd.org/mailman/listinfo/freebsd-current
To unsubscribe, send any mail to "freebsd-current-unsubscribe@freebsd.org"

[ message continues ]
[view original message]
From: Arjan van Leeuwen
Subject: Re: panic: ffs_blkfree: freeing free block
Date: Thursday, October 18, 2007 - 9:40 am

After working for a whole day without softupdates, I can say that turning
them off at least causes less panics to happen than with them turned on,
maybe they even don't happen at all without softupdates; I haven't had a
panic all day, while I have one every few hours with softupdates turned on.
Arjan
_______________________________________________
freebsd-current@freebsd.org mailing list
http://lists.freebsd.org/mailman/listinfo/freebsd-current
To unsubscribe, send any mail to "freebsd-current-unsubscribe@freebsd.org"

[ message continues ]
[view original message]
From: Tim Bishop
Subject: Re: panic: ffs_blkfree: freeing free block
Date: Friday, October 19, 2007 - 2:12 am

I've been running for the best part of a day now with softupdates turned
off and so far no panics. I'm running tinderbox on the host, and it
would quite reliably crash it before.

Of course it's hard to say if this has fixed the problem... maybe it
doesn't happen as often, or maybe my data is being slowly chewed up
instead ;-)

Tim.

-- 
Tim Bishop
http://www.bishnet.net/tim/
PGP Key: 0x5AE7D984
_______________________________________________
freebsd-current@freebsd.org mailing list
http://lists.freebsd.org/mailman/listinfo/freebsd-current
To unsubscribe, send any mail to "freebsd-current-unsubscribe@freebsd.org"

[ message continues ]
[view original message]
From: Arjan van Leeuwen
Subject: Re: panic: ffs_blkfree: freeing free block
Date: Tuesday, October 23, 2007 - 2:35 am

It looks like on the same system, I'm able to reliably panic zfs as well,
under the exact same conditions (i.e. linking a particularly big piece of
software). Maybe this is not a problem in the filesystem at all. I've not
been able to get a coredump yet from the zfs panic.
This is in 7.0-PRERELEASE, btw (I switched to RELENG_7 when the branching
happened).
_______________________________________________
freebsd-current@freebsd.org mailing list
http://lists.freebsd.org/mailman/listinfo/freebsd-current
To unsubscribe, send any mail to "freebsd-current-unsubscribe@freebsd.org"

[ message continues ]
[view original message]
From: Ken Smith
Subject: Re: panic: ffs_blkfree: freeing free block
Date: Wednesday, November 7, 2007 - 10:36 am

Is anyone who had been able to trigger this panic still having problems
with recent kernels (and soft updates turned on)?  I've checked with a
few people who had been experiencing the panic and they can no longer
trigger it.

It's at least a tiny bit possible some of the VM fixes that have gone in
addressed this problem.  We'd like to find out if anyone can still
trigger this.

Thanks.

--=20
                                                Ken Smith
- From there to here, from here to      |       kensmith@cse.buffalo.edu
  there, funny things are everywhere.   |
                      - Theodore Geisel |


[ message continues ]
[view original message]
From: Joe Marcus Clarke
Subject: Re: panic: ffs_blkfree: freeing free block
Date: Wednesday, November 7, 2007 - 12:16 pm

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1


I have been running unpatched for a while now with multiple GNOME 2.20
TB builds running, and I have not seen this panic.  It's probably been
three weeks now.

Joe

- --
Joe Marcus Clarke
FreeBSD GNOME Team	::	gnome@FreeBSD.org
FreeNode / #freebsd-gnome
http://www.FreeBSD.org/gnome
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.6 (Darwin)
Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org

iD8DBQFHMg76b2iPiv4Uz4cRAuOZAJ9RO6gj29dd7mIliFtPOV0Hv+YeyACfe7Tf
Urd5kQKv1qCzE1FJTFxdndc=
=9cYK
-----END PGP SIGNATURE-----
_______________________________________________
freebsd-current@freebsd.org mailing list
http://lists.freebsd.org/mailman/listinfo/freebsd-current
To unsubscribe, send any mail to "freebsd-current-unsubscribe@freebsd.org"

[ message continues ]
[view original message]
From: Peter Holm
Subject: Re: panic: ffs_blkfree: freeing free block
Date: Thursday, November 8, 2007 - 11:30 am

I have been hunting this problem for a while, but not been able to
reproduce it lately.

-- 
Peter Holm
_______________________________________________
freebsd-current@freebsd.org mailing list
http://lists.freebsd.org/mailman/listinfo/freebsd-current
To unsubscribe, send any mail to "freebsd-current-unsubscribe@freebsd.org"

[ message continues ]
[view original message]
From: Tim Bishop
Subject: Re: panic: ffs_blkfree: freeing free block
Date: Friday, November 9, 2007 - 4:50 am

I've been trying for over a week to reproduce this and have so far been
unable to do so.

Tim.

-- 
Tim Bishop
http://www.bishnet.net/tim/
PGP Key: 0x5AE7D984
_______________________________________________
freebsd-current@freebsd.org mailing list
http://lists.freebsd.org/mailman/listinfo/freebsd-current
To unsubscribe, send any mail to "freebsd-current-unsubscribe@freebsd.org"

[ message continues ]
[view original message]
From: Pieter de Goeje
Subject: Re: panic: ffs_blkfree: freeing free block
Date: Friday, November 9, 2007 - 5:04 am

I can no longer reproduce it any more.

-
Pieter
_______________________________________________
freebsd-current@freebsd.org mailing list
http://lists.freebsd.org/mailman/listinfo/freebsd-current
To unsubscribe, send any mail to "freebsd-current-unsubscribe@freebsd.org"

[ message continues ]
[view original message]
From: Kostik Belousov
Subject: Re: panic: ffs_blkfree: freeing free block
Date: Friday, November 9, 2007 - 5:26 am

Could anybody who was able to trigger the panic with relative ease, do
the binary search for the dates that
A. started the problem
B. eliminated it

There is uneasy feeling for the bug that did such appearance and still
not tracked.

[ message continues ]
[view original message]
From: Pieter de Goeje
Subject: Re: panic: ffs_blkfree: freeing free block
Date: Saturday, November 10, 2007 - 9:07 am

Doing a binary search now. Might take some time though.

Pieter de Goeje
_______________________________________________
freebsd-current@freebsd.org mailing list
http://lists.freebsd.org/mailman/listinfo/freebsd-current
To unsubscribe, send any mail to "freebsd-current-unsubscribe@freebsd.org"

[ message continues ]
[view original message]
From: Pieter de Goeje
Subject: Re: panic: ffs_blkfree: freeing free block
Date: Sunday, November 11, 2007 - 7:33 am

This commit to sys/vm/vm_object.c fixed it:

revision 1.386
date: 2007/10/18 23:02:18;  author: alc;  state: Exp;  lines: +2 -1
The previous revision, updating vm_object_page_remove() for the new page
cache, did not account for the case where the vm object has nothing but
cached pages.

Reported by:    kris, tegge
Reviewed by:    tegge

If you want I can still do a search for the commit that introduced the 
problem, but I think this gives a rather strong clue on what caused it :)

Pieter de Goeje
_______________________________________________
freebsd-current@freebsd.org mailing list
http://lists.freebsd.org/mailman/listinfo/freebsd-current
To unsubscribe, send any mail to "freebsd-current-unsubscribe@freebsd.org"

[ message continues ]
[view original message]
From: Ken Smith
Subject: Re: panic: ffs_blkfree: freeing free block
Date: Sunday, November 11, 2007 - 10:26 am

We really really really appreciate your help with this.  Thank you VERY
much!

--=20
                                                Ken Smith
- From there to here, from here to      |       kensmith@cse.buffalo.edu
  there, funny things are everywhere.   |
                      - Theodore Geisel |


[ message continues ]
[view original message]
From: Tim Bishop
Subject: Re: panic: ffs_blkfree: freeing free block
Date: Thursday, October 18, 2007 - 3:05 am

Filesystem         1K-blocks     Used    Avail Capacity  Mounted on
/dev/mirror/gm0s1f 137110242 37650508 88490916    30%    /u1

The problems only occur under load, and this is the filesystem that has
all the activity on it.




It's not always the same. Recently I've had:

block = 2752640
block = 2754352

/dev/mirror/gm0s1f: INCORRECT BLOCK COUNT I=615883 (7328 should be 0) (CORRECTED)
/dev/mirror/gm0s1f: INCORRECT BLOCK COUNT I=615921 (416 should be 0) (CORRECTED)
/dev/mirror/gm0s1f: INCORRECT BLOCK COUNT I=616086 (7328 should be 0) (CORRECTED)
/dev/mirror/gm0s1f: INCORRECT BLOCK COUNT I=616102 (4 should be 0) (CORRECTED)
/dev/mirror/gm0s1f: LINK COUNT DIR I=612441  OWNER=root MODE=40755
/dev/mirror/gm0s1f: SIZE=512 MTIME=Oct 15 18:57 2007  COUNT 20 SHOULD BE 19 (ADJUSTED)
/dev/mirror/gm0s1f: LINK COUNT DIR I=612454  OWNER=root MODE=41777
/dev/mirror/gm0s1f: SIZE=512 MTIME=Oct 15 18:57 2007  COUNT 5 SHOULD BE 2 (ADJUSTED)
/dev/mirror/gm0s1f: UNREF FILE I=616131  OWNER=root MODE=120755
/dev/mirror/gm0s1f: SIZE=0 MTIME=Oct 15 18:57 2007  (CLEARED)
/dev/mirror/gm0s1f: UNREF DIR I=4057971  OWNER=root MODE=40755
/dev/mirror/gm0s1f: SIZE=512 MTIME=Oct 15 18:57 2007  (CLEARED)
/dev/mirror/gm0s1f: UNREF DIR I=4057972  OWNER=root MODE=40755
/dev/mirror/gm0s1f: SIZE=512 MTIME=Oct 15 18:57 2007  (CLEARED)
/dev/mirror/gm0s1f: UNREF DIR I=4057973  OWNER=root MODE=40755
/dev/mirror/gm0s1f: SIZE=512 MTIME=Oct 15 18:57 2007  (CLEARED)
/dev/mirror/gm0s1f: UNREF DIR I=4057974  OWNER=root MODE=40755
/dev/mirror/gm0s1f: SIZE=512 MTIME=Oct 15 18:57 2007  (CLEARED)
/dev/mirror/gm0s1f: LINK COUNT DIR I=4199271  OWNER=root MODE=40755
/dev/mirror/gm0s1f: SIZE=512 MTIME=Oct 15 18:57 2007  COUNT 20 SHOULD BE 19 (ADJUSTED)
/dev/mirror/gm0s1f: UNREF DIR I=4429003  OWNER=root MODE=40755
/dev/mirror/gm0s1f: SIZE=512 MTIME=Oct 15 18:57 2007  (CLEARED)
/dev/mirror/gm0s1f: UNREF DIR I=4429004  OWNER=root MODE=40755
/dev/mirror/gm0s1f: SIZE=512 MTIME=Oct 15 18:57 2007  (CLEARED)
/dev/mirror/gm0s1f: ...
[ message continues ]
[view original message]
From: Kostik Belousov
Subject: Re: panic: ffs_blkfree: freeing free block
Date: Thursday, October 18, 2007 - 3:25 am

Oh, and another one:
was there a message "handle_workitem_freeblocks: block count"
from the kernel before the panic ?


[ message continues ]
[view original message]
From: Tim Bishop
Subject: Re: panic: ffs_blkfree: freeing free block
Date: Thursday, October 18, 2007 - 3:42 am

In my case, no.

Tim.

-- 
Tim Bishop
http://www.bishnet.net/tim/
PGP Key: 0x5AE7D984
_______________________________________________
freebsd-current@freebsd.org mailing list
http://lists.freebsd.org/mailman/listinfo/freebsd-current
To unsubscribe, send any mail to "freebsd-current-unsubscribe@freebsd.org"

[ message continues ]
[view original message]
From: Eric Anderson
Subject: Re: panic: ffs_blkfree: freeing free block
Date: Thursday, October 18, 2007 - 4:17 am

Of all the reports recently, this one has only come up once, in another 
thread, but he did not get the blkfree panic with it.


Eric

_______________________________________________
freebsd-current@freebsd.org mailing list
http://lists.freebsd.org/mailman/listinfo/freebsd-current
To unsubscribe, send any mail to "freebsd-current-unsubscribe@freebsd.org"

[ message continues ]
[view original message]
From: Pieter de Goeje
Subject: Re: panic: ffs_blkfree: freeing free block
Date: Thursday, October 18, 2007 - 6:57 am

Filesystem  1K-blocks    Used     Avail Capacity  Mounted on
To me this problem happened on both SMP and UP systems, all running amd64.
_______________________________________________
freebsd-current@freebsd.org mailing list
http://lists.freebsd.org/mailman/listinfo/freebsd-current
To unsubscribe, send any mail to "freebsd-current-unsubscribe@freebsd.org"

[ message continues ]
[view original message]
From: Rong-en Fan
Subject: Re: panic: ffs_blkfree: freeing free block
Date: Thursday, October 18, 2007 - 8:04 am

Just want to add one more instance ;-)

I have seen this kind of panic since 4.x, it does not happen very often
but it often (not always) happens when the filesystem is full. The server
in question is an i386 with local disk exported via nfs to ~20 clients. The
filesystem was UFS1 and now UFS2 (but the underlying disk was changed
to another raid box). Softupdate is turned on all the time, but not quota.
The block in panic message does not the same. And I do full fsck after
the panic (no background fsck). Some previous ddb and kgdb sessions
can be found at:

http://www.rafan.org/FreeBSD/panic/freeing_free_block/

Regards,
_______________________________________________
freebsd-current@freebsd.org mailing list
http://lists.freebsd.org/mailman/listinfo/freebsd-current
To unsubscribe, send any mail to "freebsd-current-unsubscribe@freebsd.org"

[ message continues ]
[view original message]
From: Tim Bishop
Subject: Re: panic: ffs_blkfree: freeing free block
Date: Monday, October 15, 2007 - 7:56 am

I'll helpfully add a "me too" here. But my system can't even manage a
dump:

dev = mirror/gm0s1f, block = 2754352, fs = /u1
panic: ffs_blkfree: freeing free block
cpuid = 1
Uptime: 3h24m58s
Physical memory: 1013 MB
Dumping 202 MB:
Fatal double fault:
eip = 0xc073e8b9
esp = 0xe415dfd4
ebp = 0xe415e048
cpuid = 0; apic id = 00

Has anyone found a fix yet?

Tim.

-- 
Tim Bishop
http://www.bishnet.net/tim/
PGP Key: 0x5AE7D984
_______________________________________________
freebsd-current@freebsd.org mailing list
http://lists.freebsd.org/mailman/listinfo/freebsd-current
To unsubscribe, send any mail to "freebsd-current-unsubscribe@freebsd.org"

[ message continues ]
[view original message]
From: Marcus Alves Grando
Subject: Re: panic: ffs_blkfree: freeing free block
Date: Monday, October 15, 2007 - 9:54 am

Ok. +1

My backtrace:

(kgdb) bt
#0  doadump () at pcpu.h:195
#1  0xc0608a87 in boot (howto=260) at /usr/src/sys/kern/kern_shutdown.c:409
#2  0xc0608d49 in panic (fmt=Variable "fmt" is not available.
) at /usr/src/sys/kern/kern_shutdown.c:563
#3  0xc07b3bb4 in ffs_blkfree (ump=0xc3edca00, fs=0xc3f17000, 
devvp=0xc3f8abb0, bno=7785608, size=16384, inum=1931884)
     at /usr/src/sys/ufs/ffs/ffs_alloc.c:1893
#4  0xc07c7578 in indir_trunc (freeblks=0xc4e0a500, dbn=30913952, 
level=0, lbn=12, countp=0xe46e9c4c) at 
/usr/src/sys/ufs/ffs/ffs_softdep.c:2896
#5  0xc07c7857 in handle_workitem_freeblocks (freeblks=0xc4e0a500, 
flags=0) at /usr/src/sys/ufs/ffs/ffs_softdep.c:2746
#6  0xc07c8fee in process_worklist_item (mp=0xc3fee538, flags=Variable 
"flags" is not available.
) at /usr/src/sys/ufs/ffs/ffs_softdep.c:963
#7  0xc07ca012 in softdep_process_worklist (mp=0xc3fee538, full=0) at 
/usr/src/sys/ufs/ffs/ffs_softdep.c:847
#8  0xc07cc4dd in softdep_flush () at /usr/src/sys/ufs/ffs/ffs_softdep.c:758
#9  0xc05e7fa9 in fork_exit (callout=0xc07cc060 <softdep_flush>, 
arg=0x0, frame=0xe46e9d38) at /usr/src/sys/kern/kern_fork.c:796
#10 0xc0843b50 in fork_trampoline () at 
/usr/src/sys/i386/i386/exception.s:205

Regards


-- 
Marcus Alves Grando
marcus(at)sbh.eng.br | Personal
mnag(at)FreeBSD.org  | FreeBSD.org
_______________________________________________
freebsd-current@freebsd.org mailing list
http://lists.freebsd.org/mailman/listinfo/freebsd-current
To unsubscribe, send any mail to "freebsd-current-unsubscribe@freebsd.org"

[ message continues ]
[view original message]
From: Tim Bishop
Subject: Re: panic: ffs_blkfree: freeing free block
Date: Monday, October 15, 2007 - 11:56 am

After changing my dump device to a spare disk rather than a mirror I
have the following dump:

Unread portion of the kernel message buffer:
dev = mirror/gm0s1f, block = 2752640, fs = /u1
panic: ffs_blkfree: freeing free block
cpuid = 0
Uptime: 55m54s
Physical memory: 1013 MB
Dumping 161 MB: 146 130 114 98 82 66 50 34 18 2

#0  doadump () at pcpu.h:195
195             __asm __volatile("movl %%fs:0,%0" : "=r" (td));
(kgdb) bt
#0  doadump () at pcpu.h:195
#1  0xc05167a7 in boot (howto=260) at
/usr/src/sys/kern/kern_shutdown.c:409
#2  0xc0516a69 in panic (fmt=Variable "fmt" is not available.
) at /usr/src/sys/kern/kern_shutdown.c:563
#3  0xc06a2124 in ffs_blkfree (ump=0xc3d81200, fs=0xc3da9800,
    devvp=0xc3e4dcc0, bno=2752640, size=16384, inum=615921)
    at /usr/src/sys/ufs/ffs/ffs_alloc.c:1893
#4  0xc06b6321 in handle_workitem_freefrag (freefrag=0xc7b2f1a0)
    at /usr/src/sys/ufs/ffs/ffs_softdep.c:1825
#5  0xc06b7569 in process_worklist_item (mp=0xc3dc6d0c, flags=Variable "flags" is not available.
)
    at /usr/src/sys/ufs/ffs/ffs_softdep.c:968
#6  0xc06b8582 in softdep_process_worklist (mp=0xc3dc6d0c, full=0)
    at /usr/src/sys/ufs/ffs/ffs_softdep.c:847
#7  0xc06baa4d in softdep_flush () at /usr/src/sys/ufs/ffs/ffs_softdep.c:758
#8  0xc04f6d89 in fork_exit (callout=0xc06ba5d0 <softdep_flush>, arg=0x0,
    frame=0xe415fd38) at /usr/src/sys/kern/kern_fork.c:796
#9  0xc071a2b0 in fork_trampoline () at /usr/src/sys/i386/i386/exception.s:205

Tim.

-- 
Tim Bishop
http://www.bishnet.net/tim/
PGP Key: 0x5AE7D984
_______________________________________________
freebsd-current@freebsd.org mailing list
http://lists.freebsd.org/mailman/listinfo/freebsd-current
To unsubscribe, send any mail to "freebsd-current-unsubscribe@freebsd.org"

[ message continues ]
Previous thread: Re: JMicron RAID problems by Andrey V. Elsukov on Sunday, September 30, 2007 - 6:53 am. (4 messages)

Next thread: patch: change in acpi taskq behavior by Nate Lawson on Sunday, September 30, 2007 - 1:10 pm. (6 messages)