Re: [PATCH] remove throttle_vm_writeout()

Previous thread: [PATCH 2/2] implement new notifier function to panic_notifier_list by Takenori Nagano on Thursday, October 4, 2007 - 4:38 am. (5 messages)

Next thread: video resume stuff by Pavel Machek on Thursday, October 4, 2007 - 6:05 am. (13 messages)
From: Miklos Szeredi
Date: Thursday, October 4, 2007 - 5:25 am

This in preparation for the writable mmap patches for fuse.  I know it
conflicts with

  writeback-remove-unnecessary-wait-in-throttle_vm_writeout.patch

but if this function is to be removed, it doesn't make much sense to
fix it first ;)
---

From: Miklos Szeredi <mszeredi@suse.cz>

By relying on the global diry limits, this can cause a deadlock when
devices are stacked.

If the stacking is done through a fuse filesystem, the __GFP_FS,
__GFP_IO tests won't help: the process doing the allocation doesn't
have any special flag.

So why exactly does this function exist?

Direct reclaim does not _increase_ the number of dirty pages in the
system, so rate limiting it seems somewhat pointless.

There are two cases:

1) File backed pages -> file

  dirty + writeback count remains constant

2) Anonymous pages -> swap

  writeback count increases, dirty balancing will hold back file
  writeback in favor of swap

So the real question is: does case 2 need rate limiting, or is it OK
to let the device queue fill with swap pages as fast as possible?

Signed-off-by: Miklos Szeredi <mszeredi@suse.cz>
---

Index: linux/include/linux/writeback.h
===================================================================
--- linux.orig/include/linux/writeback.h	2007-10-02 16:55:03.000000000 +0200
+++ linux/include/linux/writeback.h	2007-10-04 13:40:33.000000000 +0200
@@ -94,7 +94,6 @@ static inline void inode_sync_wait(struc
 int wakeup_pdflush(long nr_pages);
 void laptop_io_completion(void);
 void laptop_sync_completion(void);
-void throttle_vm_writeout(gfp_t gfp_mask);
 
 /* These are exported to sysctl. */
 extern int dirty_background_ratio;
Index: linux/mm/page-writeback.c
===================================================================
--- linux.orig/mm/page-writeback.c	2007-10-02 16:55:03.000000000 +0200
+++ linux/mm/page-writeback.c	2007-10-04 13:40:33.000000000 +0200
@@ -497,37 +497,6 @@ void balance_dirty_pages_ratelimited_nr(
 }
 ...
From: Peter Zijlstra
Date: Thursday, October 4, 2007 - 5:40 am

Because balance_dirty_pages() maintains:

 nr_dirty + nr_unstable + nr_writeback <=20
	total_dirty + nr_cpus * ratelimit_pages

throttle_vm_writeout() _should_ not deadlock on that, unless you're
caught in the error term: nr_cpus * ratelimit_pages.=20

Which can only happen when it is larger than 10% of dirty_thresh.

Which is even more unlikely since it doesn't account nr_dirty (as I
think it should).

As for 2), yes I think having a limit on the total number of pages in
flight is a good thing. But that said, there might be better ways to do
that.



From: Miklos Szeredi
Date: Thursday, October 4, 2007 - 6:00 am

And it does get caught on that in small memory machines.  This
deadlock is easily reproducable on a 32MB UML instance.  I haven't yet
tested with the per-bdi patches, but I don't think they make a

I think nr_dirty is totally irrelevant.  Since we don't care about


Sure, if we do need to globally limit the number of under-writeback
pages, then I think we need to do it independently of the dirty
accounting.

Miklos
-

From: Peter Zijlstra
Date: Thursday, October 4, 2007 - 6:23 am

Ah, but its correct to have since we compare against dirty_thresh, which
is defined to be a unit of nr_dirty + nr_unstable + nr_writeback. if we

for my swapping over network thingies I need to put a bound on the
amount of outgoing traffic in flight because that bounds the amount of

It need not be global, it could be per BDI as well, but yes.
From: Miklos Szeredi
Date: Thursday, October 4, 2007 - 6:49 am

Yeah, I guess the point of the function was to limit nr_write to

I guess you will have some request queue with limited length, no?

The main problem seems to be if devices use up all the reserved memory
for queuing write requests.  Limiting the in-flight pages is a very
crude way to solve this, the assumptions are:

O: overhead as a fraction of the request size
T: total memory
R: reserved memory
T-R: may be full of anon pages

so if (T-R)*O > R  we are in trouble.

if we limit the writeback memory to L and L*O < R we are OK.  But we
don't know O (it's device dependent).  We can make an estimate
calculate L based on that, but that will be a number totally

For per-bdi limits we have the queue length.

Miklod
-

From: Peter Zijlstra
Date: Thursday, October 4, 2007 - 9:47 am

Yeah, I'm guestimating O on a per device basis, but I agree that the
current ratio limiting is quite crude. I'm not at all sorry to see
throttle_vm_writeback() go, I just wanted to make a point that what it
does is not quite without merrit - we agree that it can be done better

Agreed, except for:

static int may_write_to_queue(struct backing_dev_info *bdi)
{
	if (current->flags & PF_SWAPWRITE)
		return 1;
	if (!bdi_write_congested(bdi))
		return 1;
	if (bdi == current->backing_dev_info)
		return 1;
	return 0;
}

Which will write to congested queues. Anybody know why?

-

From: Andrew Morton
Date: Thursday, October 4, 2007 - 10:46 am

commit c4e2d7ddde9693a4c05da7afd485db02c27a7a09
Author: akpm <akpm>
Date:   Sun Dec 22 01:07:33 2002 +0000

    [PATCH] Give kswapd writeback higher priority than pdflush
    
    The `low latency page reclaim' design works by preventing page
    allocators from blocking on request queues (and by preventing them from
    blocking against writeback of individual pages, but that is immaterial
    here).
    
    This has a problem under some situations.  pdflush (or a write(2)
    caller) could be saturating the queue with highmem pages.  This
    prevents anyone from writing back ZONE_NORMAL pages.  We end up doing
    enormous amounts of scenning.
    
    A test case is to mmap(MAP_SHARED) almost all of a 4G machine's memory,
    then kill the mmapping applications.  The machine instantly goes from
    0% of memory dirty to 95% or more.  pdflush kicks in and starts writing
    the least-recently-dirtied pages, which are all highmem.  The queue is
    congested so nobody will write back ZONE_NORMAL pages.  kswapd chews
    50% of the CPU scanning past dirty ZONE_NORMAL pages and page reclaim
    efficiency (pages_reclaimed/pages_scanned) falls to 2%.
    
    So this patch changes the policy for kswapd.  kswapd may use all of a
    request queue, and is prepared to block on request queues.
    
    What will now happen in the above scenario is:
    
    1: The page alloctor scans some pages, fails to reclaim enough
       memory and takes a nap in blk_congetion_wait().
    
    2: kswapd() will scan the ZONE_NORMAL LRU and will start writing
       back pages.  (These pages will be rotated to the tail of the
       inactive list at IO-completion interrupt time).
    
       This writeback will saturate the queue with ZONE_NORMAL pages.
       Conveniently, pdflush will avoid the congested queues.  So we end up
       writing the correct pages.
    
    In this test, kswapd CPU utilisation falls from 50% to 2%, page reclaim
    efficiency rises from 2% to 40% and ...
From: Peter Zijlstra
Date: Thursday, October 4, 2007 - 11:10 am

with highmem >> normal, and user pages preferring highmem, this will


So request queue's have a limit above the congestion level on which they
will block?


I could do that.

-

From: Andrew Morton
Date: Thursday, October 4, 2007 - 11:54 am

On Thu, 04 Oct 2007 20:10:10 +0200


I guess first you'd need to be able to reproduce the problem which that
patch fixed, then check that it remains fixed.

Sigh.  That problem was fairly subtle.  We could re-break reclaim in
this way and not find out about it for six months.  There's a lesson here. 
Several.  
-

From: Fengguang Wu
Date: Friday, October 5, 2007 - 5:30 am

Sorry, I cannot not understand it. We now have balanced aging between
zones. So the page allocations are expected to distribute proportionally

We may see the same problem and improvement in the absent of 'all
writeback goes to one zone' assumption.

The problem could be:
- dirty_thresh is exceeded, so balance_dirty_pages() starts syncing
  data and quickly _congests_ the queue;
- dirty pages are slowly but continuously turned into clean pages by
  balance_dirty_pages(), but they still stay in the same place in LRU;
- the zones are mostly dirty/writeback pages, kswapd has a hard time
  finding the randomly distributed clean pages;
- kswapd cannot do the writeout because the queue is congested!

The improvement could be:
- kswapd is now explicitly preferred to do the writeout;
- the pages written by kswapd will be rotated and easy for kswapd to reclaim;
- it becomes possible for kswapd to wait for the congested queue,
  instead of doing the vmscan like mad.

The congestion wait looks like a pretty natural way to throttle the kswapd.
Instead of doing the vmscan at 1000MB/s and actually freeing pages at
60MB/s(about the write throughput), kswapd will be relaxed to do vmscan at
maybe 150MB/s.

Fengguang

-

From: Andrew Morton
Date: Friday, October 5, 2007 - 10:20 am

On Fri, 5 Oct 2007 20:30:28 +0800

Sure, but we don't have one disk queue per disk per zone!  The queue is
shared by all the zones.  So if writeback from one zone has filled the
queue up, the kernel can't write back data from another zone.

(Well, it can, by blocking in get_request_wait(), but that causes long and

Or someone ran fsync(), or pdflush is writing back data because it exceeded

Yeah.  In 2.4 and early 2.5, page-reclaim (both direct reclaim and kswapd,
iirc) would throttle by waiting on writeout of a particular page.  This was
a poor design, because writeback against a *particular* page can take
anywhere from one millisecond to thirty seconds to complete, depending upon
where the disk head is and all that stuff.

The critical change I made was to switch the throttling algorithm from
"wait for one page to get written" to "wait for _any_ page to get written".
 Becaue reclaim really doesn't care _which_ page got written: we want to
wake up and start scanning again when _any_ page got written.

That's what congestion_wait() does.

It is pretty crude.  It could be that writeback completed against pages which
aren't in the correct zone, or it could be that some other task went and
allocated the just-cleaned pages before this task can get running and
reclaim them, or it could be that the just-written-back pages weren't
reclaimable after all, etc.

It would take a mind-boggling amount of logic and locking to make all this
100% accurate and the need has never been demonstrated.  So page reclaim
presently should be viewed as a polling algorithm, where the rate of

Something like that.

The critical numbers to watch are /proc/vmstat's *scan* and *steal*.  Look:

akpm:/usr/src/25> uptime
 10:08:14 up 10 days, 16:46, 15 users,  load average: 0.02, 0.05, 0.04
akpm:/usr/src/25> grep steal /proc/vmstat
pgsteal_dma 0
pgsteal_dma32 0
pgsteal_normal 0
pgsteal_high 0
pginodesteal 0
kswapd_steal 1218698
kswapd_inodesteal 266847
akpm:/usr/src/25> grep scan /proc/vmstat     ...
From: Fengguang Wu
Date: Friday, October 5, 2007 - 7:32 pm

Hmm, that's a problem. But I guess when one zone is full, other zones

I guess PF_SWAPWRITE processes still have good probability to stuck in
get_request_wait().  Because balance_dirty_pages() are allowed to
disregard the congestion.  It will be exhausting the available request
slots all the time.

Signed-off-by: Fengguang Wu <wfg@mail.ustc.edu.cn>
---
 mm/page-writeback.c |    1 +
 1 file changed, 1 insertion(+)

--- linux-2.6.23-rc8-mm2.orig/mm/page-writeback.c
+++ linux-2.6.23-rc8-mm2/mm/page-writeback.c
@@ -400,6 +400,7 @@ static void balance_dirty_pages(struct a
 			.sync_mode	= WB_SYNC_NONE,
 			.older_than_this = NULL,
 			.nr_to_write	= write_chunk,
+			.nonblocking	= 1,
 			.range_cyclic	= 1,




Nice tool!
I've been watching the raw numbers by writing scripts. This one can be
written as:

#!/bin/bash

while true; do
	while read a b
	do
		eval $a=$b
	done < /proc/vmstat

	uptime=$(</proc/uptime)

	scan=$((
		pgscan_kswapd_dma +
		pgscan_kswapd_dma32 +
		pgscan_kswapd_normal +
		pgscan_kswapd_high +
		pgscan_direct_dma +
		pgscan_direct_dma32 +
		pgscan_direct_normal +
		pgscan_direct_high
		))

	steal=$((
		pgsteal_dma +
		pgsteal_dma32 +
		pgsteal_normal +
		pgsteal_high
		))

	ratio=$((100*steal/(scan+1)))

	echo -e "$uptime\t$ratio%\t$steal\t$scan"
        sleep 1;
done

Not surprisingly, I see nice numbers on my desktop:


Thank you for the nice tip :-)

Fengguang

-

From: Miklos Szeredi
Date: Thursday, October 4, 2007 - 2:07 pm

Yes.  So what is it to be?

Is limiting by device queues enough?

Or do we need some global limit?

If so, the cleanest way I see is to separately account and limit
swap-writeback pages, so the global counters don't interfere with the
limiting.

This shouldn't be hard to do, as we have the per-bdi writeback
counting infrastructure already, and also a pseudo bdi for swap in
swapper_space.backing_dev_info.

Miklos
-

From: Andrew Morton
Date: Thursday, October 4, 2007 - 2:56 pm

On Thu, 04 Oct 2007 14:25:22 +0200

This description of the bug-which-is-being-fixed is nowhere near adequate
enough for a reviewer to understand the problem.  This makes it hard to

That's described in the changelog for the patch which added

None of the above.

    [PATCH] vm: pageout throttling
    
    With silly pageout testcases it is possible to place huge amounts of memory
    under I/O.  With a large request queue (CFQ uses 8192 requests) it is
    possible to place _all_ memory under I/O at the same time.
    
    This means that all memory is pinned and unreclaimable and the VM gets
    upset and goes oom.
    
    The patch limits the amount of memory which is under pageout writeout to be
    a little more than the amount of memory at which balance_dirty_pages()
    callers will synchronously throttle.
    
    This means that heavy pageout activity can starve heavy writeback activity
    completely, but heavy writeback activity will not cause starvation of
    pageout.  Because we don't want a simple `dd' to be causing excessive
    latencies in page reclaim.

afaict that problem is still there.  It is possible to get all of
ZONE_NORMAL dirty on a highmem machine.  With a large queue (or lots of
queues), vmscan can them place all of ZONE_NORMAL under IO.

It could be that we've fixed this problem via other means in the interrim,
but from a quick peek to seems to me that the scanner will still do a 100%
CPU burn when all of a zone's pages are under writeback.

throttle_vm_writeout() should be a per-zone thing, I guess.  Perhaps fixing
that would fix your deadlock.  That's doubtful, but I don't know anything
about your deadlock so I cannot say.

-

From: Miklos Szeredi
Date: Thursday, October 4, 2007 - 3:39 pm

Ah, OK.


No, doing the throttling per-zone won't in itself fix the deadlock.

Here's a deadlock example:

Total memory = 32M
/proc/sys/vm/dirty_ratio = 10
dirty_threshold = 3M
ratelimit_pages = 1M

Some program dirties 4M (dirty_threshold + ratelimit_pages) of mmap on
a fuse fs.  Page balancing is called which turns all these into
writeback pages.

Then userspace filesystem gets a write request, and tries to allocate
memory needed to complete the writeout.

That will possibly trigger direct reclaim, and throttle_vm_writeout()
will be called.  That will block until nr_writeback goes below 3.3M
(dirty_threshold + 10%).  But since all 4M of writeback is from the
fuse fs, that will never happen.

Does that explain it better?

Miklos
-

From: Andrew Morton
Date: Thursday, October 4, 2007 - 4:09 pm

On Fri, 05 Oct 2007 00:39:16 +0200

yup, thanks.

This is a somewhat general problem: a userspace process is in the IO path. 
Userspace block drivers, for example - pretty much anything which involves
kernel->userspace upcalls for storage applications.

I solved it once in the past by marking the userspace process as
PF_MEMALLOC and I beleive that others have implemented the same hack.

I suspect that what we need is a general solution, and that the solution
will involve explicitly telling the kernel that this process is one which
actually cleans memory and needs special treatment.

Because I bet there will be other corner-cases where such a process needs
kernel help, and there might be optimisation opportunities as well.

Problem is, any such mark-me-as-special syscall would need to be
privileged, and FUSE servers presently don't require special perms (do
they?)

-

From: Miklos Szeredi
Date: Thursday, October 4, 2007 - 4:26 pm

No, and that's a rather important feature, that I'd rather not give
up.  But with the dirty limiting, the memory cleaning really shouldn't
be a problem, as there is plenty of memory _not_ used for dirty file
data, that the filesystem can use during the writeback.

So the only thing the kernel should be careful about, is not to block
on an allocation if not strictly necessary.

Actually a trivial fix for this problem could be to just tweak the
thresholds, so to make the above scenario impossible.  Although I'm
still not convinced, this patch is perfect, because the dirty
threshold can actually change in time...

Index: linux/mm/page-writeback.c
===================================================================
--- linux.orig/mm/page-writeback.c      2007-10-05 00:31:01.000000000 +0200
+++ linux/mm/page-writeback.c   2007-10-05 00:50:11.000000000 +0200
@@ -515,6 +515,12 @@ void throttle_vm_writeout(gfp_t gfp_mask
         for ( ; ; ) {
                get_dirty_limits(&background_thresh, &dirty_thresh, NULL, NULL);

+               /*
+                * Make sure the theshold is over the hard limit of
+                * dirty_thresh + ratelimit_pages * nr_cpus
+                */
+               dirty_thresh += ratelimit_pages * num_online_cpus();
+
                 /*
                  * Boost the allowable dirty threshold a bit for page
                  * allocators so they don't get DoS'ed by heavy writers


-

From: Andrew Morton
Date: Thursday, October 4, 2007 - 4:48 pm

On Fri, 05 Oct 2007 01:26:12 +0200

Can fuse do it?  Perhaps the fs can diddle the server's task_struct at

I don't think I understand that.  Sure, it _shouldn't_ be a problem.  But it

I can probably kind of guess what you're trying to do here.  But if
ratelimit_pages * num_online_cpus() exceeds the size of the offending zone
then things might go bad.

-

From: Miklos Szeredi
Date: Thursday, October 4, 2007 - 5:12 pm

No, it's futile.  What if another process is involved (ssh in case of

The problem, I believe is in the memory allocation code, not in fuse.

In the example, memory allocation may be blocking indefinitely,
because we have 4MB under writeback, even though 28MB can still be

I think the admin can do quite a bit of other damage, by setting
dirty_ratio too high.

Maybe this writeback throttling should just have a fixed limit of 80%
ZONE_NORMAL, and limit dirty_ratio to something like 50%.

Miklos
-

From: Andrew Morton
Date: Thursday, October 4, 2007 - 5:48 pm

fuse is trying to do something which page reclaim was not designed for. 

Well yes.  But we need to work out how, without re-breaking the thing which

Bear in mind that the same problem will occur for the 16MB ZONE_DMA, and
we cannot limit the system-wide dirty-memory threshold to 12MB.

iow, throttle_vm_writeout() needs to become zone-aware.  Then it only
throttles when, say, 80% of ZONE_FOO is under writeback.

Except I don't think that'll fix the problem 100%: if your fuse kernel
component somehow manages to put 80% of ZONE_FOO under writeback (and
remmeber this might be only 12MB on a 16GB machine) then we get stuck again
- the fuse server process (is that the correct terminology, btw?) ends up
waiting upon itself.

I'll think about it a bit.
-

From: Peter Zijlstra
Date: Friday, October 5, 2007 - 1:22 am

I'm thinking the really_congested thing will also fix this. By only
=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=

As it stand 110% of dirty limit can already be larger than say zone_dma
(and likely is), so that is not a new bug - and I don't think its the
thing Miklos runs into.

The problem Miklos is seeing (and I, just in a different form), is that
throttle_vm_writeout() gets stuck because balance_dirty_pages() gets
called once every ratelimit_pages (per cpu). So we can have nr_cpus *
ratelimit_pages extra.....

/me thinks

ok I confused myself.

by calling balance_dirty_pages() once every ratelimit_pages (per cpu)
allows for nr_cpus() * ratelimit_pages extra _dirty_ pages. But
balance_dirty_pages() will make it:
  nr_dirty + nr_unstable + nr_writeback < thresh

So even if it writes out all of the dirty pages, we still have:
  nr_unstable + nr_writeback < thresh

So at any one time nr_writeback should not exceed thresh. But it does!?

So how do we end up with more writeback pages than that? should we teach
pdflush about these limits as well?

From: Miklos Szeredi
Date: Friday, October 5, 2007 - 2:22 am

Ugh.

I think we should rather fix vmscan to not spin when all pages of a
zone are already under writeout.  Which is the _real_ problem,
according to Andrew.

Miklos
-

From: Peter Zijlstra
Date: Friday, October 5, 2007 - 2:47 am

diff --git a/include/linux/writeback.h b/include/linux/writeback.h
index 4ef4d22..eff2438 100644
--- a/include/linux/writeback.h
+++ b/include/linux/writeback.h
@@ -88,7 +88,7 @@ static inline void wait_on_inode(struct inode *inode)
 int wakeup_pdflush(long nr_pages);
 void laptop_io_completion(void);
 void laptop_sync_completion(void);
-void throttle_vm_writeout(gfp_t gfp_mask);
+void throttle_vm_writeout(struct zone *zone, gfp_t gfp_mask);
=20
 /* These are exported to sysctl. */
 extern int dirty_background_ratio;
diff --git a/mm/page-writeback.c b/mm/page-writeback.c
index eec1481..f949997 100644
--- a/mm/page-writeback.c
+++ b/mm/page-writeback.c
@@ -326,11 +326,8 @@ void balance_dirty_pages_ratelimited_nr(struct address=
_space *mapping,
 }
 EXPORT_SYMBOL(balance_dirty_pages_ratelimited_nr);
=20
-void throttle_vm_writeout(gfp_t gfp_mask)
+void throttle_vm_writeout(struct zone *zone, gfp_t gfp_mask)
 {
-	long background_thresh;
-	long dirty_thresh;
-
 	if ((gfp_mask & (__GFP_FS|__GFP_IO)) !=3D (__GFP_FS|__GFP_IO)) {
 		/*
 		 * The caller might hold locks which can prevent IO completion
@@ -342,17 +339,16 @@ void throttle_vm_writeout(gfp_t gfp_mask)
 	}
=20
         for ( ; ; ) {
-		get_dirty_limits(&background_thresh, &dirty_thresh, NULL);
+		unsigned long thresh =3D zone_page_state(zone, NR_ACTIVE) +
+			zone_page_state(zone, NR_INACTIVE);
=20
-                /*
-                 * Boost the allowable dirty threshold a bit for page
-                 * allocators so they don't get DoS'ed by heavy writers
-                 */
-                dirty_thresh +=3D dirty_thresh / 10;      /* wheeee... */
+		/*
+		 * wait when 75% of the zone's pages are under writeback
+		 */
+		thresh -=3D thresh >> 2;
+		if (zone_page_state(zone, NR_WRITEBACK) < thresh)
+			break;
=20
-                if (global_page_state(NR_UNSTABLE_NFS) +
-			global_page_state(NR_WRITEBACK) <=3D dirty_thresh)
-                        	break;
                 ...
From: Miklos Szeredi
Date: Friday, October 5, 2007 - 3:27 am

I think that's an improvement in all respects.

However it still does not generally address the deadlock scenario: if
there's a small DMA zone, and fuse manages to put all of those pages
under writeout, then there's trouble.

But it's not really fuse specific.  If it was a normal filesystem that
did that, and it needed a GFP_DMA allocation for writeout, it is in
trouble also, as that allocation would fail (at least no deadlock).

Or is GFP_DMA never used by fs/io writeout paths?

Miklos
-

From: Miklos Szeredi
Date: Friday, October 5, 2007 - 3:32 am

And the only way to solve that AFAICS, is to make sure fuse never uses
more than e.g. 50% of _any_ zone for page cache.  And that may need
some tweaking in the allocator...

Miklos
-

From: John Stoffel
Date: Friday, October 5, 2007 - 8:43 am

Miklos> And the only way to solve that AFAICS, is to make sure fuse
Miklos> never uses more than e.g. 50% of _any_ zone for page cache.
Miklos> And that may need some tweaking in the allocator...

So what happens if I have three different FUSE mounts, all under heavy
write pressure?  It's not a FUSE problem, it's a VM problem as far as
I can see.   All I did was extrapolate from the 50% number (where did
that come from?) and triple it to go over 100%, since we obviously
shouldn't take 100% of any zone, right?

So the real cure is to have some way to rate limit Zone usage, making
it harder and harder to allocate in a zone as the zone gets more and
more full.  But how do you do this in a non-deadlocky way?  

Buy hey, I'm not that knowledgeable about the VM.  
-

From: Peter Zijlstra
Date: Friday, October 5, 2007 - 3:57 am

I agree that its not complete. (hence the lack of sign-off etc.)

'normally' writeback pages just need an interrupt to signal the stuff is
written back. ie. the writeback completion path is atomic.=20

 [ result of the thinking below -- the above is esp. true for swap pages
   - so maybe we should ensure that !swap traffic can never exceed this
   75% - that way, swap can always us get out of a tight spot ]

Maybe we should make that a requirement, although I see how that becomes
rather hard for FUSE (which is where Andrews PF_MEMALLOC suggestion
comes from - but I really dislike PF_MEMALLOC exposed to userspace).

Limiting FUSE to say 50% (suggestion from your other email) sounds like
a horrible hack to me. - Need more time to think on this.

.. o O [ process of thinking ]

While I think, I might as well tell the problem I had with
throttle_vm_writeout(), which is nfs unstable pages. Those don't go away
by waiting for it - as throttle_vm_writeout() does. So once you have an
excess of those, you're stuck as well.

In this patch I totally ignored unstable, but I'm not sure that's the
proper thing to do, I'd need to figure out what happens to an unstable
page when passed into pageout() - or if its passed to pageout at all.

If unstable pages would be passed to pageout(), and it would properly
convert them to writeback and clean them, then there is nothing wrong.

(Trond?)


From: Miklos Szeredi
Date: Friday, October 5, 2007 - 4:27 am

I don't really understand all that page balancing stuff, but I think
this will probably never or very rarely happen, because the allocator
will prefer the bigger zones, and the dirty page limiting will not let
the bigger zones get too full of dirty pages.

And even it can happen, it's not necessarily a fuse-only thing.

It makes tons of sense to make sure, that we don't fully dirty _any_
specialized zone.  One special zone group are the low-memory pages.
And currently balance_dirty_pages() makes sure we don't fill that up
with dirty file backed pages.  So something like that should make
sense for other special zones like DMA as well.

I'm not saying it's trivial, or even possible to implement, just
thinking...

Miklos
-

From: Trond Myklebust
Date: Friday, October 5, 2007 - 10:50 am

Why would we want to do that? That would be a hell of a lot of work
(locking pages, setting flags, unlocking pages, ...) for absolutely no
reason.

Unstable writes are writes which have been sent to the server, but which
haven't been written to disk on the server. A single RPC command is then
sent (COMMIT) which basically tells the server to call fsync(). After
that is successful, we can free up the pages, but we do that with no
extra manipulation of the pages themselves: no page locks, just removal
from the NFS private radix tree, and freeing up of the NFS private
structures.

We only need to touch the pages again in the unlikely case that the
COMMIT fails because the server has rebooted. In this case we have to
resend the writes, and so the pages are marked as dirty, so we can go
through the whole writepages() rigmarole again...

So, no. I don't see sending pages through pageout() as being at all
helpful.

Trond

-

From: Peter Zijlstra
Date: Friday, October 5, 2007 - 11:32 am

Well, the thing is, we throttle pageout in throttle_vm_writeout(). As it
stand we can deadlock there because it just waits for the numbers to
drop, and unstable pages don't automagically dissapear. Only
write_inodes() - normally called from balance_dirty_pages() will call
COMMIT.

So my thought was that calling pageout() on an unstable page would do
the COMMIT - we're low on memory, otherwise we would not be paging, so
getting rid of unstable pages seems to make sense to me.



-

From: Trond Myklebust
Date: Friday, October 5, 2007 - 12:20 pm

Why not rather track which mappings have large numbers of outstanding
unstable writes at the VM level, and then add some form of callback to
allow it to notify the filesystem when it needs to flush them out?

Cheers
  Trond

-

From: Trond Myklebust
Date: Friday, October 5, 2007 - 12:23 pm

BTW: Please note that at least in the case of NFS, you will have to
allow for the fact that the filesystem may not be _able_ to cause the
numbers to drop. If the server is unavailable, then we're may be stuck
in unstable page limbo for quite some time.

Trond

-

From: Peter Zijlstra
Date: Friday, October 5, 2007 - 2:07 pm

That would be nice, its just that the pageout throttling is not quite

Agreed, it would be nice if that is handled is such a manner that it
does not take down all other paging.

The regular write path that only bothers with balance_dirty_pages()
already does this nicely.

-

From: Fengguang Wu
Date: Friday, October 5, 2007 - 5:40 pm

I wonder whether
        if (!bdi_nr_writeback)
                break;

I guess "many unstable pages" would be better if we are taking this way.

-

From: Peter Zijlstra
Date: Friday, October 5, 2007 - 12:32 am

I think just adding nr_cpus * ratelimit_pages to the dirth_thresh in
throttle_vm_writeout() will also solve the problem
From: Rik van Riel
Date: Friday, October 5, 2007 - 12:54 pm

On Fri, 05 Oct 2007 09:32:57 +0200

Agreed, that should fix the main latency issues.
 
-- 
All Rights Reversed
-

Previous thread: [PATCH 2/2] implement new notifier function to panic_notifier_list by Takenori Nagano on Thursday, October 4, 2007 - 4:38 am. (5 messages)

Next thread: video resume stuff by Pavel Machek on Thursday, October 4, 2007 - 6:05 am. (13 messages)