Re: very poor ext3 write performance on big filesystems?

Score:

Previous message: [thread] [date] [author]
Next message: [thread] [date] [author]

To: Theodore Tso <tytso@...>, Andi Kleen <andi@...>, LKML <linux-fsdevel@...>, LKML <linux-kernel@...>

Subject: Re: very poor ext3 write performance on big filesystems?

Date: Wednesday, February 27, 2008 - 7:20 am

Theodore Tso schrieb:
quoted text> On Mon, Feb 18, 2008 at 03:03:44PM +0100, Andi Kleen wrote:
>> Tomasz Chmielewski <mangoo@wpkg.org> writes:
>>> Is it normal to expect the write speed go down to only few dozens of
>>> kilobytes/s? Is it because of that many seeks? Can it be somehow
>>> optimized? 
>> I have similar problems on my linux source partition which also
>> has a lot of hard linked files (although probably not quite
>> as many as you do). It seems like hard linking prevents
>> some of the heuristics ext* uses to generate non fragmented
>> disk layouts and the resulting seeking makes things slow.

A follow-up to this thread.
Using small optimizations like playing with /proc/sys/vm/* didn't help 
much, increasing "commit=" ext3 mount option helped only a tiny bit.

What *did* help a lot was... disabling the internal bitmap of the RAID-5 
array. "rm -rf" doesn't "pause" for several seconds any more.

If md and dm supported barriers, it would be even better I guess (I 
could enable write cache with some degree of confidence).


This is "iostat sda -d 10" output without the internal bitmap.
The system mostly tries to read (Blk_read/s), and once in a while it
does a big commit (Blk_wrtn/s):

Device:            tps   Blk_read/s   Blk_wrtn/s   Blk_read   Blk_wrtn
sda             164,67      2088,62         0,00      20928          0

Device:            tps   Blk_read/s   Blk_wrtn/s   Blk_read   Blk_wrtn
sda             180,12      1999,60         0,00      20016          0

Device:            tps   Blk_read/s   Blk_wrtn/s   Blk_read   Blk_wrtn
sda             172,63      2587,01         0,00      25896          0

Device:            tps   Blk_read/s   Blk_wrtn/s   Blk_read   Blk_wrtn
sda             156,53      2054,64         0,00      20608          0

Device:            tps   Blk_read/s   Blk_wrtn/s   Blk_read   Blk_wrtn
sda             170,20      3013,60         0,00      30136          0

Device:            tps   Blk_read/s   Blk_wrtn/s   Blk_read   Blk_wrtn
sda             119,46      1377,25      5264,67      13800      52752

Device:            tps   Blk_read/s   Blk_wrtn/s   Blk_read   Blk_wrtn
sda             154,05      1897,10         0,00      18952          0

Device:            tps   Blk_read/s   Blk_wrtn/s   Blk_read   Blk_wrtn
sda             197,70      2177,02         0,00      21792          0

Device:            tps   Blk_read/s   Blk_wrtn/s   Blk_read   Blk_wrtn
sda             166,47      1805,19         0,00      18088          0

Device:            tps   Blk_read/s   Blk_wrtn/s   Blk_read   Blk_wrtn
sda             150,95      1552,05         0,00      15536          0

Device:            tps   Blk_read/s   Blk_wrtn/s   Blk_read   Blk_wrtn
sda             158,44      1792,61         0,00      17944          0

Device:            tps   Blk_read/s   Blk_wrtn/s   Blk_read   Blk_wrtn
sda             132,47      1399,40      3781,82      14008      37856



With the bitmap enabled, it sometimes behave similarly, but mostly, I
can see as reads compete with writes, and both have very low numbers then:

Device:            tps   Blk_read/s   Blk_wrtn/s   Blk_read   Blk_wrtn
sda             112,57       946,11      5837,13       9480      58488

Device:            tps   Blk_read/s   Blk_wrtn/s   Blk_read   Blk_wrtn
sda             157,24      1858,94         0,00      18608          0

Device:            tps   Blk_read/s   Blk_wrtn/s   Blk_read   Blk_wrtn
sda             116,90      1173,60        44,00      11736        440

Device:            tps   Blk_read/s   Blk_wrtn/s   Blk_read   Blk_wrtn
sda              24,05        85,43       172,46        856       1728

Device:            tps   Blk_read/s   Blk_wrtn/s   Blk_read   Blk_wrtn
sda              25,60        90,40       165,60        904       1656

Device:            tps   Blk_read/s   Blk_wrtn/s   Blk_read   Blk_wrtn
sda              25,05       276,25       180,44       2768       1808

Device:            tps   Blk_read/s   Blk_wrtn/s   Blk_read   Blk_wrtn
sda              22,70        65,60       229,60        656       2296

Device:            tps   Blk_read/s   Blk_wrtn/s   Blk_read   Blk_wrtn
sda              21,66       202,79       786,43       2032       7880

Device:            tps   Blk_read/s   Blk_wrtn/s   Blk_read   Blk_wrtn
sda              20,90        83,20      1800,00        832      18000

Device:            tps   Blk_read/s   Blk_wrtn/s   Blk_read   Blk_wrtn
sda              51,75       237,36       479,52       2376       4800

Device:            tps   Blk_read/s   Blk_wrtn/s   Blk_read   Blk_wrtn
sda              35,43       129,34       245,91       1296       2464

Device:            tps   Blk_read/s   Blk_wrtn/s   Blk_read   Blk_wrtn
sda              34,50        88,00       270,40        880       2704


Now, let's disable the bitmap in the RAID-5 array:

Device:            tps   Blk_read/s   Blk_wrtn/s   Blk_read   Blk_wrtn
sda             110,59       536,26       973,43       5368       9744

Device:            tps   Blk_read/s   Blk_wrtn/s   Blk_read   Blk_wrtn
sda             119,68       533,07      1574,43       5336      15760

Device:            tps   Blk_read/s   Blk_wrtn/s   Blk_read   Blk_wrtn
sda             123,78       368,43      2335,26       3688      23376

Device:            tps   Blk_read/s   Blk_wrtn/s   Blk_read   Blk_wrtn
sda             122,48       315,68      1990,01       3160      19920

Device:            tps   Blk_read/s   Blk_wrtn/s   Blk_read   Blk_wrtn
sda             117,08       580,22      1009,39       5808      10104

Device:            tps   Blk_read/s   Blk_wrtn/s   Blk_read   Blk_wrtn
sda             119,50       324,00      1080,80       3240      10808

Device:            tps   Blk_read/s   Blk_wrtn/s   Blk_read   Blk_wrtn
sda             118,36       353,69      1926,55       3544      19304


And let's enable it again - after a while, it degrades again, and I can
see "rm -rf" stops for longer periods:

Device:            tps   Blk_read/s   Blk_wrtn/s   Blk_read   Blk_wrtn
sda             162,70      2213,60         0,00      22136          0

Device:            tps   Blk_read/s   Blk_wrtn/s   Blk_read   Blk_wrtn
sda             165,73      1639,16         0,00      16408          0

Device:            tps   Blk_read/s   Blk_wrtn/s   Blk_read   Blk_wrtn
sda             119,76      1192,81      3722,16      11952      37296

Device:            tps   Blk_read/s   Blk_wrtn/s   Blk_read   Blk_wrtn
sda             178,70      1855,20         0,00      18552          0

Device:            tps   Blk_read/s   Blk_wrtn/s   Blk_read   Blk_wrtn
sda             162,64      1528,07         0,80      15296          8

Device:            tps   Blk_read/s   Blk_wrtn/s   Blk_read   Blk_wrtn
sda             182,87      2082,07         0,00      20904          0

Device:            tps   Blk_read/s   Blk_wrtn/s   Blk_read   Blk_wrtn
sda             168,93      1692,71         0,00      16944          0

Device:            tps   Blk_read/s   Blk_wrtn/s   Blk_read   Blk_wrtn
sda             177,45      1572,06         0,00      15752          0

Device:            tps   Blk_read/s   Blk_wrtn/s   Blk_read   Blk_wrtn
sda             123,10      1436,00      4941,60      14360      49416

Device:            tps   Blk_read/s   Blk_wrtn/s   Blk_read   Blk_wrtn
sda             201,30      1984,03         0,00      19880          0

Device:            tps   Blk_read/s   Blk_wrtn/s   Blk_read   Blk_wrtn
sda             165,50      1555,20        22,40      15552        224

Device:            tps   Blk_read/s   Blk_wrtn/s   Blk_read   Blk_wrtn
sda              25,35       273,05       189,22       2736       1896

Device:            tps   Blk_read/s   Blk_wrtn/s   Blk_read   Blk_wrtn
sda              22,58        63,94       165,43        640       1656

Device:            tps   Blk_read/s   Blk_wrtn/s   Blk_read   Blk_wrtn
sda              69,40       435,20       262,40       4352       2624



There is a related thread (although not much kernel-related) on a 
BackupPC mailing list:

http://thread.gmane.org/gmane.comp.sysutils.backup.backuppc.general/14009

As it's BackupPC software which makes this amount of hardlinks (but hey, 
I can keep ~14 TB of data on a 1.2 TB filesystem which is not even 65% 
full).


-- 
Tomasz Chmielewski
http://wpkg.org
--
unsubscribe noticeTo unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html
Please read the FAQ at  http://www.tux.org/lkml/

Previous message: [thread] [date] [author]
Next message: [thread] [date] [author]

Messages in current thread:

very poor ext3 write performance on big filesystems?, Tomasz Chmielewski, (Mon Feb 18, 8:57 am)

Re: very poor ext3 write performance on big filesystems?, Vladislav Bolkhovitin, (Tue Feb 19, 5:24 am)

Re: very poor ext3 write performance on big filesystems?, Andi Kleen, (Mon Feb 18, 10:03 am)

Re: very poor ext3 write performance on big filesystems?, Theodore Tso, (Mon Feb 18, 10:16 am)

Re: very poor ext3 write performance on big filesystems?, Tomasz Chmielewski, (Wed Feb 27, 7:20 am)

Re: very poor ext3 write performance on big filesystems?, Andreas Dilger, (Wed Feb 27, 4:03 pm)

Re: very poor ext3 write performance on big filesystems?, Bill Davidsen, (Sat Mar 1, 4:04 pm)

Re: very poor ext3 write performance on big filesystems?, Tomasz Chmielewski, (Wed Feb 27, 4:25 pm)

Re: very poor ext3 write performance on big filesystems?, Mark Lord, (Tue Feb 19, 2:29 pm)

Re: very poor ext3 write performance on big filesystems?, Paulo Marques, (Tue Feb 19, 2:58 pm)

Re: very poor ext3 write performance on big filesystems?, Mark Lord, (Tue Feb 19, 6:33 pm)

Re: very poor ext3 write performance on big filesystems?, Mark Lord, (Tue Feb 19, 2:41 pm)

Re: very poor ext3 write performance on big filesystems?, Tomasz Chmielewski, (Tue Feb 19, 10:54 am)

Re: very poor ext3 write performance on big filesystems?, Chris Mason, (Tue Feb 19, 11:06 am)

Re: very poor ext3 write performance on big filesystems?, Tomasz Chmielewski, (Tue Feb 19, 11:21 am)

Re: very poor ext3 write performance on big filesystems?, Chris Mason, (Tue Feb 19, 12:04 pm)

Re: very poor ext3 write performance on big filesystems?, Tomasz Chmielewski, (Mon Feb 18, 11:02 am)

Re: very poor ext3 write performance on big filesystems?, Theodore Tso, (Mon Feb 18, 11:16 am)

Re: very poor ext3 write performance on big filesystems?, Tomasz Chmielewski, (Mon Feb 18, 12:16 pm)

Re: very poor ext3 write performance on big filesystems?, Theodore Tso, (Mon Feb 18, 2:45 pm)

Re: very poor ext3 write performance on big filesystems?, Andi Kleen, (Mon Feb 18, 11:57 am)

Re: very poor ext3 write performance on big filesystems?, Theodore Tso, (Mon Feb 18, 11:35 am)

Re: very poor ext3 write performance on big filesystems?, Jan Engelhardt, (Wed Feb 20, 6:57 am)

Re: very poor ext3 write performance on big filesystems?, David Rees, (Wed Feb 20, 1:44 pm)

Re: very poor ext3 write performance on big filesystems?, Jan Engelhardt, (Wed Feb 20, 2:08 pm)

Re: very poor ext3 write performance on big filesystems?, Andi Kleen, (Mon Feb 18, 11:18 am)

Re: very poor ext3 write performance on big filesystems?, Theodore Tso, (Mon Feb 18, 11:03 am)


linux-kernel:
Arnd Bergmann	SCHED_IDLE documentation
david	Re: limits on raid
Jan Engelhardt	Re: [PATCH] CodingStyle: multiple updates
Ingo Molnar	Re: Rescheduling interrupts
git:
Russ Brown	git-svn: Branching clarifications
Sam Song	Fwd: [OT] Re: Git via a proxy server?
Junio C Hamano	Re: More precise tag following
Pierre Habouzit	Re: People unaware of the importance of "git gc"?
openbsd-misc:
Michael	Virtual interface
Stijn	Re: libiconv problem
Stefan Beke	mail dovecot: pipe() failed: Too many open files
Amaury De Ganseman	"ping: sendto: No buffer space available" when using bittorrent or another p2p
linux-activists:
Jim Winstead Jr.	Re: Root Disk/Book Disk Compatibility
Darren Senn	Re: Elm
Seung-Chul Woo	Is it possible to mount GNU HURD file system as DOS in SLS?
David Willmore	Re: Intel, the Pentium and Linux


spam	2 hours ago	KernelTrap Suggestions and Feedback
sis190/191 gigabit ethernet driver	3 hours ago	Linux general
read /dev/mem not working in 2.6	1 day ago	Linux general
Fedora with Windows vista : windows explorer restarts	1 day ago	Windows
Intel Graphics Drivers (IEGD 9.0.2) - help in patching for kernels > 2.6.24	1 day ago	Linux kernel
Porting from Windows to Linux	1 day ago	Linux general
Serial Driver Implementation Help	2 days ago	Linux kernel
aacraid bad	2 days ago	Linux general
Tools: GCC 3.4.6, Final GCC 3 Release	3 days ago	Applications and Utilities
GNU/Hurd is so awful	4 days ago	GNU/Hurd

Re: very poor ext3 write performance on big filesystems?

Online users