On Sun, Mar 29, 2009 at 09:55:59PM -0600, Trenton D. Adams wrote:
Well, I believe it should solve it for most realistic workloads (where
I don't think "dd if=/dev/zero of=bigzero.img" is realistic).
Looking more closely at the statistics, the delays aren't coming from
trying to flush the data blocks in data=ordered mode. If we disable
delayed allocation (mount -o nodelalloc), you'll see this when you
look at /proc/fs/jbd2/<dev>/history:
R/C tid wait run lock flush log hndls block inlog ctime write drop close
R 12 23 3836 0 1460 2563 50129 56 57
R 13 0 5023 0 1056 2100 64436 70 71
R 14 0 3156 0 1433 1803 40816 47 48
R 15 0 4250 0 1206 2473 57623 63 64
R 16 0 5000 0 1516 1136 61087 67 68
Note the amount of time in milliseconds in the flush column. That's
time spent flusing the allocated data blocks to disk. This goes away
once you enable delayed allocation:
R/C tid wait run lock flush log hndls block inlog ctime write drop close
R 56 0 2283 0 10 1250 32735 37 38
R 57 0 2463 0 13 1126 31297 38 39
R 58 0 2413 0 13 1243 35340 40 41
R 59 3 2383 0 20 1270 30760 38 39
R 60 0 2316 0 23 1176 33696 38 39
R 61 0 2266 0 23 1150 29888 37 38
R 62 0 2490 0 26 1140 35661 39 40
You may see slightly worse times since I'm running with a patch (which
will be pushed for 2.6.30) that makes sure that the blocks we are
writing during the "log" phase are written using WRITE_SYNC instead of
WRITE. (Without this patch, the huge amount of writes caused by the
VM trying to keep up with pages being dirtied at CPU speeds via "dd
if=/dev/zero..." will interfere with writes to the journal.)
During the log phase (which is averaging around 2 seconds for
nodealloc, and 1 seconds with delayed allocation enabled), we write
the metadata to the journal. The number of blocks that we are
actually writing to the journal is small (around 40 per transaction)
so I suspect we're seeing some lock contention or some accounting
overhead caused by the metadata blocks constantly getting dirtied by
dd if=/dev/zero task. We can look to see if this can be improved,
possibly by changing how we handle the locking, but it's no longer
being caused by the data=ordered flushing behaviour.
And that's a good thing from a development point of view when trying
to fix performance problems. When making statements about what people
are likely to find in real life, it's less useful.
- Ted
--
