After several months of work, Neal Walfield [Interview] has finished the second stage of the Hurd/L4 virtual memory framework, building on his previous work for self-paging tasks [story]. This code drop includes support for logical copying (i.e. copy on write and shared memory) between so-called containers and hence processes. Further, he overhauled libhurd-mm, the high-level application memory-management library, to include an internal paging interface, which simplifies memory management firstly by providing abstractions to bind virtual memory to various predefined backing stores (e.g. file and anonymous memory) as well as to user defined backing stores and eventually by providing a default eviction policy.

This concludes work on the core of the virtual memory framework for now, the next step is to develop the I/O interfaces between clients and file systems. This work adds pressure to develop an IDE driver. As the Device Driver Framework is not in place, a simple IDE driver or a driver ported from Linux or NetBSD should suffice to allow for further development.

From: Neal H. Walfield
Subject: Logical Copying and Internal Paging Code Committed
Date: Wed, 06 Apr 2005 18:40:18 +0100

Friends,

On January 12th, I said[1] that

  the next step is to implement copying physical memory between
  containers.  With this mechanism available, work can seriously start
  on the device driver framework (deva and fabrica).  This is also a
  prerequisite to the porting of libpager, the Hurd's library for
  application level pagers, which is needed to flesh out the
  implementation of the root filesystem (initially using a ramdisk
  until the IDE driver is ready) and portability issues with the
  filesystem, file and io RPC interfaces.


I have just checked in some[2] code to implement logical copying (both
copy on write and normal shared memory).  Logical copies combined with
support for multiple page sizes makes deallocation quite hairy.  In
fact, 29.5% (or 928 out of 3143 lines of heavily commented code) of
physmem is dedicated to this procedure.  The good news is it seems to
work.  At least with my test cases.

I've also checked in an overhauled libhurd-mm.  There is a nice shiny
internal paging interface in place, cf. libhurd-mm/mm.h [2].  Using
this, an application can bind various parts of its address space to
stores.  A store can be a file, anonymous memory, computed data, etc.
When a thread faults on the area, the pager invokes the stores fault
handler with the fault address, the start of the region and the
corresponding location on backing store.  The fault handler needs to
get the data and install it in the cache.  All rather simple and all
in the application itself.


The next step is two fold: implement a real backing store (e.g. a
simple file system or block device) and implement the code for clients
to read data from and write data to it.  From there we can start
thinkink about porting ext2fs.  We'll then have to revisit physmem to
implement memory quotas so that the memory policy server can be worked
on.

A major problem right now is the lack of capability passing.  Getting
the initial next step working will be possible without this, however,
this is a hard requirement for dynamically open objects on servers.

A good project for someone is to think about an IDE driver.  I've
looked at Linux's IDE framework and it seems to be a bit of a tangled
mess.  Perhaps the BSDs IDE drivers will pry out easier.

There are a few small and medium projects related to physmem and
libhurd-mm.  For instance, one could write a slab allocator backing
store: cached but otherwise unused slabs don't need to be sent to
swap; they can be freed directly.  Another is improving the physical
memory allocator.  Right now, there is no way to request that memory
have particular properties, e.g. have a specific physical memory
address.  This is very important for DMA and page coloring.  zalloc
needs to be modified to support this behavior.  Applications should be
able to indicate what physical memory is acceptable by passing two
words: in the first one, any bit which is set indicates that the
corresponding bit in start of the physical memory must also be set and
a second one indicating that any bit set must not be set in the
starting address of the allocated phsyical memory.

I will add these and some more to the physmem/TODO in the coming days.

Thanks,
Neal

[1] http://lists.gnu.org/archive/html/l4-hurd/2005-01/msg00037.html
[2] diffstat says:

 deva/ChangeLog              |    6
 deva/mmap.c                 |   11
 libhurd-cap/cap-intern.h    |    6
 libhurd-cap/cap.c           |    2
 libhurd-cap/cap.h           |    9
 libhurd-mm/ChangeLog        |  150 +++++
 libhurd-mm/Makefile.am      |    5
 libhurd-mm/anonymous.c      |   80 +++
 libhurd-mm/anonymous.h      |   71 ++
 libhurd-mm/as.c             |   66 ++
 libhurd-mm/core.c           |   56 ++
 libhurd-mm/frame.c          |  222 --------
 libhurd-mm/headers.m4       |    1
 libhurd-mm/map.c            |  204 +++----
 libhurd-mm/memory.c         |  325 ++++++++++++
 libhurd-mm/mm-init.c        |   34 -
 libhurd-mm/mm.h             |  137 +++++
 libhurd-mm/pager.c          |   43 -
 libhurd-mm/physmem-user.c   |   54 --
 libhurd-mm/physmem-user.h   |   56 --
 libhurd-mm/priv.h           |  203 ++-----
 libhurd-mm/store.c          |  220 ++++++++
 libhurd-mm/vm.c             |  241 +++------
 libhurd-mm/vm.h             |   55 --
 physmem/ChangeLog           |   99 +++
 physmem/Makefile.am         |    2
 physmem/container.c         |  439 +++++++++-------
 physmem/frame-entry.c       | 1154 ++++++++++++++++++++++++++++++++++++++++++--
 physmem/frame.c             |   59 +-
 physmem/headers.m4          |   13
 physmem/ia32-cmain.c        |    2
 physmem/physmem.c           |    2
 physmem/physmem.h           |  299 ++---------
 physmem/priv.h              |  357 +++++++++++++
 physmem/tests/Makefile.am   |   27 +
 physmem/tests/environment.h |  138 +++++
 physmem/tests/t-frame.c     |   91 +++
 ruth/mmap.c                 |   11
 ruth/physmem-user.h         |  116 ++++
 ruth/ruth.c                 |  314 +++++++++++
 task/mmap.c                 |   11
 wortel/ChangeLog            |    6
 wortel/startup.c            |    5
 43 files changed, 4088 insertions(+), 1314 deletions(-)
 [3] http://sv.gnu.org/cgi-bin/viewcvs/*checkout*/hurd/hurd-l4/libhurd-mm/mm.h