> On Fri, 22 Aug 2008 10:05:45 +0900 KOSAKI Motohiro <kosaki.motohiro@jp.fujitsu.com> wrote:
>
> > > > quicklist_total_size() is racy against cpu hotplug. That's OK for
> > > > /proc/meminfo purposes (occasional transient inaccuracy?), but will it
> > > > crash? Not in the current implementation of per_cpu() afaict, but it
> > > > might crash if we ever teach cpu hotunplug to free up the percpu
> > > > resources.
> > >
> > > First, Quicklist doesn't concern to cpu hotplug at all.
> > > it is another quicklist problem.
> > >
> > > Next, I think it doesn't cause crash. but I haven't any test.
> > > So, I'll test cpu hotplug/unplug testing today.
> > >
> > > I'll report result tommorow.
> >
> > OK.
> > I ran cpu hotplug/unplug coutinuous workload over 12H.
> > then, system crash doesn't happend.
> >
> > So, I believe my patch is cpu unplug safe.
>
> err, which patch?
>
> I presently have:
>
> mm-show-quicklist-memory-usage-in-proc-meminfo.patch
> mm-show-quicklist-memory-usage-in-proc-meminfo-fix.patch
> mm-quicklist-shouldnt-be-proportional-to-number-of-cpus.patch
> mm-quicklist-shouldnt-be-proportional-to-number-of-cpus-fix.patch
>
> Is that what you have?
>
> I'll consolidate them into two patches and will append them here. Please check.
>
>
> From: KOSAKI Motohiro <kosaki.motohiro@jp.fujitsu.com>
>
> At present the quicklists store some page for each CPU as a cache. (Each
> CPU has node_free_pages/16 pages)
>
> It is used for page table cache. Then, exit() increase cache, the other
> hand fork() spent it.
>
> So, if apache type (one parent and many child model) middleware run, One
> CPU process fork(), Other CPU process the middleware work and exit().
>
> At that time, One CPU don't have page table cache at all, Others have
> maximum caches.
>
> QList_max = (#ofCPUs - 1) x Free / 16
> => QList_max / (Free + QList_max) = (#ofCPUs - 1) / (16 + #ofCPUs - 1)
>
> So, How much quicklist spent memory at maximum case? That is #CPUs
> proposional because it is per CPU cache but cache amount calculation
> doesn't use #ofCPUs.
>
> Above calculation mean
>
> Number of CPUs per node 2 4 8 16
> ============================== ====================
> QList_max / (Free + QList_max) 5.8% 16% 30% 48%
>
>
> Wow! Quicklist can spent about 50% memory at worst case. More
> unfortunately, it doesn't have any cache shrinking mechanism. So it cause
> some wrong thing.
>
> 1. End user misunderstand to memory leak happend.
> => /proc/meminfo should display amount quicklist
>
> 2. It can cause OOM killer
> => Amount of quicklists shouldn't be proportional to number of CPUs.
>
>
>
> This patch:
>
> Quicklists can consume several GB memory. So, if end user can't see how
> much memory is used, he can fail to understand why a memory leak happend.
>
> after this patch applied, /proc/meminfo output following.
>
> % cat /proc/meminfo
>
> MemTotal: 7701504 kB
> MemFree: 5159040 kB
> Buffers: 112960 kB
> Cached: 337536 kB
> SwapCached: 0 kB
> Active: 218944 kB
> Inactive: 350848 kB
> Active(anon): 120832 kB
> Inactive(anon): 0 kB
> Active(file): 98112 kB
> Inactive(file): 350848 kB
> Unevictable: 0 kB
> Mlocked: 0 kB
> SwapTotal: 2031488 kB
> SwapFree: 2031488 kB
> Dirty: 320 kB
> Writeback: 0 kB
> AnonPages: 119488 kB
> Mapped: 38528 kB
> Slab: 1595712 kB
> SReclaimable: 23744 kB
> SUnreclaim: 1571968 kB
> PageTables: 14336 kB
> NFS_Unstable: 0 kB
> Bounce: 0 kB
> WritebackTmp: 0 kB
> CommitLimit: 5882240 kB
> Committed_AS: 356672 kB
> VmallocTotal: 17592177655808 kB
> VmallocUsed: 29056 kB
> VmallocChunk: 17592177626304 kB
> Quicklists: 283776 kB
> HugePages_Total: 0
> HugePages_Free: 0
> HugePages_Rsvd: 0
> HugePages_Surp: 0
> Hugepagesize: 262144 kB
>
> [akpm@linux-foundation.org: build fix]
> Signed-off-by: KOSAKI Motohiro <kosaki.motohiro@jp.fujitsu.com>
> Cc: Christoph Lameter <cl@linux-foundation.org>
> Cc: <stable@kernel.org> [2.6.25.x, 2.6.26.x]
> Signed-off-by: Andrew Morton <akpm@linux-foundation.org>
> ---
>
> fs/proc/proc_misc.c | 7 +++++--
> include/linux/quicklist.h | 7 +++++++
> 2 files changed, 12 insertions(+), 2 deletions(-)
>
> diff -puN fs/proc/proc_misc.c~mm-show-quicklist-memory-usage-in-proc-meminfo fs/proc/proc_misc.c
> --- a/fs/proc/proc_misc.c~mm-show-quicklist-memory-usage-in-proc-meminfo
> +++ a/fs/proc/proc_misc.c
> @@ -24,6 +24,7 @@
> #include <linux/tty.h>
> #include <linux/string.h>
> #include <linux/mman.h>
> +#include <linux/quicklist.h>
> #include <linux/proc_fs.h>
> #include <linux/ioport.h>
> #include <linux/mm.h>
> @@ -189,7 +190,8 @@ static int meminfo_read_proc(char *page,
> "Committed_AS: %8lu kB\n"
> "VmallocTotal: %8lu kB\n"
> "VmallocUsed: %8lu kB\n"
> - "VmallocChunk: %8lu kB\n",
> + "VmallocChunk: %8lu kB\n"
> + "Quicklists: %8lu kB\n",
> K(i.totalram),
> K(i.freeram),
> K(i.bufferram),
> @@ -221,7 +223,8 @@ static int meminfo_read_proc(char *page,
> K(committed),
> (unsigned long)VMALLOC_TOTAL >> 10,
> vmi.used >> 10,
> - vmi.largest_chunk >> 10
> + vmi.largest_chunk >> 10,
> + K(quicklist_total_size())
> );
>
> len += hugetlb_report_meminfo(page + len);
> diff -puN include/linux/quicklist.h~mm-show-quicklist-memory-usage-in-proc-meminfo include/linux/quicklist.h
> --- a/include/linux/quicklist.h~mm-show-quicklist-memory-usage-in-proc-meminfo
> +++ a/include/linux/quicklist.h
> @@ -80,6 +80,13 @@ void quicklist_trim(int nr, void (*dtor)
>
> unsigned long quicklist_total_size(void);
>
> +#else
> +
> +static inline unsigned long quicklist_total_size(void)
> +{
> + return 0;
> +}
> +
> #endif
>
> #endif /* LINUX_QUICKLIST_H */
> _
>
>
>
> From: KOSAKI Motohiro <kosaki.motohiro@jp.fujitsu.com>
>
> When a test program which does task migration runs, my 8GB box spends
> 800MB of memory for quicklist. This is not memory leak but doesn't seem
> good.
>
> % cat /proc/meminfo
>
> MemTotal: 7701568 kB
> MemFree: 4724672 kB
> (snip)
> Quicklists: 844800 kB
>
> because
>
> - My machine spec is
> number of numa node: 2
> number of cpus: 8 (4CPU x2 node)
> total mem: 8GB (4GB x2 node)
> free mem: about 5GB
>
> - Maximum quicklist usage is here
>
> Number of CPUs per node 2 4 8 16
> ============================== ====================
> QList_max / (Free + QList_max) 5.8% 16% 30% 48%
>
> - Then, 4.7GB x 16% ~= 880MB.
> So, Quicklist can use 800MB.
>
> So, if following spec machine run that program
>
> CPUs: 64 (8cpu x 8node)
> Mem: 1TB (128GB x8node)
>
> Then, quicklist can waste 300GB (= 1TB x 30%). It is too large.
>
> So, I don't like cache policies which is proportional to # of cpus.
>
> My patch changes the number of caches
> from:
> per-cpu-cache-amount = memory_on_node / 16
> to
> per-cpu-cache-amount = memory_on_node / 16 / number_of_cpus_on_node.
>
> I think this is reasonable. but even if this patch is applied, quicklist
> can cache tons of memory on big machine.
>
> (Although its patch applied, quicklist can waste 64GB on 1TB server (= 1TB
> / 16), it is still too much??)
>
> test program is below.
> --------------------------------------------------------------------------------
> #define _GNU_SOURCE
>
> #include <stdio.h>
> #include <errno.h>
> #include <stdlib.h>
> #include <string.h>
> #include <sched.h>
> #include <unistd.h>
> #include <sys/mman.h>
> #include <sys/wait.h>
>
> #define BUFFSIZE 512
>
> int max_cpu(void) /* get max number of logical cpus from /proc/cpuinfo */
> {
> FILE *fd;
> char *ret, buffer[BUFFSIZE];
> int cpu = 1;
>
> fd = fopen("/proc/cpuinfo", "r");
> if (fd == NULL) {
> perror("fopen(/proc/cpuinfo)");
> exit(EXIT_FAILURE);
> }
> while (1) {
> ret = fgets(buffer, BUFFSIZE, fd);
> if (ret == NULL)
> break;
> if (!strncmp(buffer, "processor", 9))
> cpu = atoi(strchr(buffer, ':') + 2);
> }
> fclose(fd);
> return cpu;
> }
>
> void cpu_bind(int cpu) /* bind current process to one cpu */
> {
> cpu_set_t mask;
> int ret;
>
> CPU_ZERO(&mask);
> CPU_SET(cpu, &mask);
> ret = sched_setaffinity(0, sizeof(mask), &mask);
> if (ret == -1) {
> perror("sched_setaffinity()");
> exit(EXIT_FAILURE);
> }
> sched_yield(); /* not necessary */
> }
>
> #define MMAP_SIZE (10 * 1024 * 1024) /* 10 MB */
> #define FORK_INTERVAL 1 /* 1 second */
>
> main(int argc, char *argv[])
> {
> int cpu_max, nextcpu;
> long pagesize;
> pid_t pid;
>
> /* set max number of logical cpu */
> if (argc > 1)
> cpu_max = atoi(argv[1]) - 1;
> else
> cpu_max = max_cpu();
>
> /* get the page size */
> pagesize = sysconf(_SC_PAGESIZE);
> if (pagesize == -1) {
> perror("sysconf(_SC_PAGESIZE)");
> exit(EXIT_FAILURE);
> }
>
> /* prepare parent process */
> cpu_bind(0);
> nextcpu = cpu_max;
>
> loop:
>
> /* select destination cpu for child process by round-robin rule */
> if (++nextcpu > cpu_max)
> nextcpu = 1;
>
> pid = fork();
>
> if (pid == 0) { /* child action */
>
> char *p;
> int i;
>
> /* consume page tables */
> p = mmap(0, MMAP_SIZE, PROT_WRITE, MAP_PRIVATE | MAP_ANONYMOUS, 0, 0);
> i = MMAP_SIZE / pagesize;
> while (i-- > 0) {
> *p = 1;
> p += pagesize;
> }
>
> /* move to other cpu */
> cpu_bind(nextcpu);
> /*
> printf("a child moved to cpu%d after mmap().\n", nextcpu);
> fflush(stdout);
> */
>
> /* back page tables to pgtable_quicklist */
> exit(0);
>
> } else if (pid > 0) { /* parent action */
>
> sleep(FORK_INTERVAL);
> waitpid(pid, NULL, WNOHANG);
>
> }
>
> goto loop;
> }
>
> [akpm@linux-foundation.org: fix build on sparc64]
> Signed-off-by: KOSAKI Motohiro <kosaki.motohiro@jp.fujitsu.com>
> Cc: Christoph Lameter <cl@linux-foundation.org>
> Cc: <stable@kernel.org> [2.6.25.x, 2.6.26.x]
> Signed-off-by: Andrew Morton <akpm@linux-foundation.org>
> ---
>
> mm/quicklist.c | 8 +++++++-
> 1 file changed, 7 insertions(+), 1 deletion(-)
>
> diff -puN mm/quicklist.c~mm-quicklist-shouldnt-be-proportional-to-number-of-cpus mm/quicklist.c
> --- a/mm/quicklist.c~mm-quicklist-shouldnt-be-proportional-to-number-of-cpus
> +++ a/mm/quicklist.c
> @@ -26,7 +26,9 @@ DEFINE_PER_CPU(struct quicklist, quickli
> static unsigned long max_pages(unsigned long min_pages)
> {
> unsigned long node_free_pages, max;
> - struct zone *zones = NODE_DATA(numa_node_id())->node_zones;
> + int node = numa_node_id();
> + struct zone *zones = NODE_DATA(node)->node_zones;
> + cpumask_t node_cpumask;
>
> node_free_pages =
> #ifdef CONFIG_ZONE_DMA
> @@ -38,6 +40,10 @@ static unsigned long max_pages(unsigned
> zone_page_state(&zones[ZONE_NORMAL], NR_FREE_PAGES);
>
> max = node_free_pages / FRACTION_OF_NODE_MEM;
> +
> + node_cpumask = node_to_cpumask(node);
> + max /= cpus_weight_nr(node_cpumask);
> +
> return max(max, min_pages);
> }
>
> _
>
> --
> To unsubscribe, send a message with 'unsubscribe linux-mm' in
> the body to
majordomo@kvack.org. For more info on Linux MM,
> see:
http://www.linux-mm.org/ .
> Don't email: <a href=mailto:"dont@kvack.org">
email@kvack.org </a>
