As a major step towards the Hurd on the L4 microkernel, Neal Walfield [interview] has committed physmem (the physical memory manager) and libhurd-mm (the per-task physical and virtual memory manager) to the L4/Hurd repository. This means that tasks are self-paging now and mmap/munmap work, paving the way for further development.
Contrary to most other operating systems including the current Hurd on Mach implementation, the scheduling and virtualization of physical memory (i.e. the multiplexing of physical memory) of the Hurd on L4 takes place in user space on a per task basis ([1], [2], [3]). Applications virtualize physical memory themselves (which is typically done by using a library), allowing them to optimize around the actual availability rather than forcing the kernel to attempt to guess access patterns. This is unlike the UNIX-like paradigm where applications receive virtual memory and they do not know and cannot easily influence the scheduling attributes of the contents. This allows for applications whose memory usage patterns do not match LRU to use their own memory management routines, for example databases, multimedia applications, garbage collectors and data analysis programs.
With regards to future development, Neal says: "From here, the next step is to implement copying physical memory between containers. With this mechanism available, work can seriously start on the device driver framework (deva and fabrica). This is also a prerequisite to the porting of libpager, the Hurd's library for application level pagers, which is needed to flesh out the implementation of the root filesystem [...] and portability issues with the filesystem, file and io RPC interfaces". Discussions about this are already taking place on the l4-hurd mailing list.
From: Neal H. Walfield Subject: Physmem exists; Tasks self-paging Date: Wed, 12 Jan 2005 11:17:31 +0000 Good Morning, For those of you following commit-hurd@gnu.org, you will have noticed [1] that last night I checked in a real physmem server and added libhurd-mm. For those of you who don't follow address@bogus.example.com, last night, I checked in a real physmem server and added libhurd-mm. physmem is the physical memory manager. It doesn't do any virtual memory management at all: it schedules physical memory allocations; scheduling of the content (i.e. multiplexing of the frame) is the exclusive domain of the applications. In its current state, physmem is able to allocated physical memory into and deallocate it out of containers. Clients (i.e. applications) can also map physical memory into their address spaces. The current code supports sharing of physical memory between containers (a necessary prerequisite for COW and copying physical memory between containers), however, actual copying between containers has not yet been implemented. libhurd-mm is the default per-task physical and virtual memory manager: it multiplexes the physical memory available to the tasks, manages the virtual address space mappings, handles page faults and pages data in and out. Currently, it supports allocating and deallocating virtual memory mappings backed by anonymous storage. The task's pager thread is also brought up and it is able to handle page faults. In terms of POSIX, this means that mmap (for anonymous memory) and munmap work which is a prerequisite for pretty much everything. Details about the implementation can befound here [2,3]. For details about the design, please look at Chapter 5 of the design document. From here, the next step is to implement copying physical memory between containers. With this mechanism available, work can seriously start on the device driver framework (deva and fabrica). This is also a prerequisite to the porting of libpager, the Hurd's library for application level pagers, which is needed to flesh out the implementation of the root filesystem (initially using a ramdisk until the IDE driver is ready) and portability issues with the filesystem, file and io RPC interfaces. Thanks, Neal [1] http://lists.gnu.org/archive/html/commit-hurd/2005-01/msg00014.html [2] http://savannah.gnu.org/cgi-bin/viewcvs/*checkout*/hurd/hurd-l4/physmem/README?rev=1.2 [3] http://savannah.gnu.org/cgi-bin/viewcvs/*checkout*/hurd/hurd-l4/libhurd-mm/README?rev=1.1
What about the security?
What about the security?
Security in what regard?
The physical memory server only allocates physical memory to a given task if that has not exceed its physical memory quota. In this way, no single task can just claim all of the physical memory in the system for itself.
It is the policy server's responsibility to divvy up the available physical memory (and it must never overcommit). It will do this by considering, among other parameters, the user of the task, the task's niceness and policy rules which the system administrator configure in a file.
When a task runs out of physical memory, it must begin to multiplex what is available to it. This means that the task must select a page to evict and swap it out if necessary. The task does this on its own so there is no security problems here (it is just writing data to disk, for instance--nothing special). Once the data has been written to backing store, the task can reuse the physical memory. Voila, self-paging tasks.
Well
Hi Neal.
My question was just a general concern, nothing special, nothing technical since i dont know much. Thanks for your reply. :^)
What about memory sharing
what about sharing executable code between different processes?
wont it conflict with that?
Logical Copying and Copy on Write
Nope. Logical copying (as well as copy on write) is fully supported.
It order to read data from a file, a task sends a read request to the appropriate filesystem server. The filesystem checks its cache to see if the data is already in memory. If it is, it asks the physical memory server to make a logical copy of it and place it in a client supplied "container". If the data is not in core, the filesystem sends a request to the backing store driver (e.g. a hard disk) and requests that the data be read in. The filesystem, before returning the memory to the client, keeps a reference to the frame in its cache so that future read requests for the same data will be filled from the filesystem cache.
The design document (in the hurd cvs repository) goes into more detail about the actual mechanisms available to support logical copying and copy on write. But rest assured, this issued were considered from the start and the mechanisms fit nicely into the framework we have developped.
and physical ram preemption?
Hi Neal,
your comment make me confused and curious about this. You talk about out of physical memory, but there are situation when system require physical memory preemption to a process, how does it is managed? I guess system requests a physical ram preemption (mandatory) to a process and wait it satisfies that .. how much system waits? how many communication is involved?
Memory Allocation
We assume that the max amount of physical memory available to a task will be relatively stable over time. So, usually it will get negotiated at startup time, and then remain constant.
Eventually, the system may get under memory pressure and need to claim memory back. Then the tasks are notified about this and the amount of max memory is renegotiated and memory is claimed back. The exact mechanisms for this are not set in stone yet, there is a variety of options, ranging from simple notification systems to market-driven economic models (which may provide an incentive for tasks to stay below their maximum of guaranteed memory).
I don't think the mechanisms are too important here. What is important is that the consequences will be correct, and the right guarantees are made (this is not easy! If you contact a task to claim back memory, how long should you wait for a reply? We have no idea. Maybe we need support from the scheduler etc). So, we can not give a complete answer to your question at this time. But remember that we already are under memory pressure, and are probably trashing on swap and whatever. So, we already are very inefficient, and thus I don't think performance is the highest concern at this point.
how do the self-paging tasks
how do the self-paging tasks swap out?
Do they all have write access to disk (block) devices?
Do they have to use a filesystem server?
Or is there some sort of swap server they go through?
Swap
Nothing would prevent a task from doing its own swapping, say to a local, private, swap file, which could also be encrypted (!).
However, to make system-wide sharing of memory possible, even among non-cooperating tasks, the system needs to be involved when memory is swapped out. Otherwise, shared copies would become non-shared when swapping takes place (the task could still share the page with a group of cooperating other tasks, by implementing its own memory protocol!).
So, the task will have to indicate to the physical memory server that a memory page should be swapped out to the system's backing store. This can not be left to the user for all sorts of considerations, including security.
So, to answer your question: Yes and no. If you do it yourself, you will not be able to take advantage of some of the advantages, like transparent sharing. But if you have secret data that you don't want to send to system swap, then you don't want to share the page. In that case, you could use an encrypted private swap file. It's this type of flexibility and user freedom that was one of the main goals of our design.
I should also add that you don't need to swap some memory at all if you want (as long as you don't exceed your physical memory allocation). So, mlock() will actually be available to all tasks of all users, and not require special privileges. It will be implemented as a local call that informs the local pager that a certain page should not be swapped out, ever.
What about it?
What about it?
Interesting news
Thank you for news about Hurd. It's not so easy to find out what happens around its development without reading mailing lists.
Any Kernel Traffic equivalent for the Hurd ? Any planned ? On what mailing list(s) should/could/would it be based ?
Actually, is there any official / unofficial media about it ? I believe this would be the best way to answer to all the flamewares around...
There was a Hurd Traffic - bu
There was a Hurd Traffic - but actually it has been given up quite a while ago.
nifty
This is really neat. I like a lot of the design ideas of a microkernel-based OS. I'm really wanting to try out L4/Hurd as soon as it becomes at least somewhat practical.
Performance of L4 port
I have tried debian distro of hurd 3 or 4 years ago.
I was attracted by various document on hurd programming practice, but when I log in it was very slow compared to linux (in running perl script in particular). Since then a lot of work is done but I suspect that system was slow for Mach<->Hurd integration (and not pthread,ie), what about L4-Hurd performance?
Performance?
It is hard to say as we haven't gotten far enough with the port to L4 to actually run real applications. We are confident, however, that we should see better performance than Mach as the areas which we have identified as being problematic in Mach have been specifically addresssed in the new framework (e.g. paging decisions).
The new design does not just address Mach's problems. We examined, among others, UNIX, VMS, Nemesis, V++, Spring, Exokernels, QNX and Eros. We think that our resource management strategy has the potential to be better than a traditional monolithic kernel. But again, only time will tell.
You said in an interview that
You said in an interview that speed isnt the main concern. How slow will it be compared to Linux or how fast?
Speed may not be the prime mo
Speed may not be the prime mover, however, that doesn't mean that we design our algorithms to be slow. Far from it. Our first concern is security followed by flexibility and performance.
We believe that you can design unsecure systems that are very fast. But, frankly, we don't think those are very interesting as because they are insecure they have limited appeal.
What about number crunching?
What about number crunching? That's why scientists use linux!
the port of hurd to l4 is dea
the port of hurd to l4 is dead... see http://www.nongnu.org/l4hurd/
:(
It WAS dead :P
Just take a look at the l4-hurd mailing list.
wrong project
Actually, l4hurd the project is dead. It was never used by the core developers and never saw much light either.
All the development is taking place in the hurd savannah project, both the l4-hurd mailing list is there and the hurd-l4 CVS module, which is part of the official hurd CVS repository
This is not the official proj
This is not the official project, it's a dead attempt to port the Hurd to L4. The current efforts take place in a separate module in CVS of the Hurd project.
The official page with inform
The official page with information about L4 Hurd is:
http://www.gnu.org/software/hurd/hurd-l4.html
:D
Glad to see that this project is not dead :D
http://lists.gnu.org/archive/html/l4-hurd/2005-01/msg00135.html