login
Search
Hi,
I am running some memory intensive simulations on an Xeon machine with 8 GB of ram (kernel 2.6.8 w/ "Physical address extension" / High memory support). The OS sees all the physical memory but each process can’t allocate more that 3 GB. Is there a way to get around this limit?
yes
Yes, there is a way to get around this limit: Buy a new machine with 64 bit CPUs. AMD Opterons are said to perform well. If you want to stay with intel, the have new models (Nocona) supporting so called E64T. Or buy Mips, Alpha, Sparc64, PPC64... Use the old machine as file server or so. BTW you should have thought about it _before_ buying the hardware.
Just don't try to address more than 4G with 32 bit CPUs, because 4G is the highest address, you can encode in said 32 bits. To raise the limit from 3G to 4G you can try the so called 4:4-split kernel patch (which slows down system calls, but your code may not make syscalls very often), but this is a really small step, is a dead-end, and you seem to want to address the full 8G.
If you are desperate, you can try page-flipping techniques, to squeeze just the parts of the memory into your address space, which you really need at that time, e.g. multiple data segments (not supported by linux and gcc for reasons, every DOS programmer can tell you (there the segments were 64k, and there was EMS)), mmap()-games (map parts of a big scratch file into your address space), etc. But this would cause massive memory access slowdowns and forces you to rewrite large parts of your application, so don't.
oops...
forgot another alternative architecture: intel Itanium..
Anyway, PAE just helps the kernel to address physical 64G, but this does not change the in-application register width limits; the kernel operates with 4k-pages internally, and even there highmem support (i.e. support for not-easily-addressable PAE-pages) was a lot of work, and is somewhat half-assed, kernel internal data structures still live inside the kernel's own 4G space.
mmap()ing a large file goes into the full RAM, because you can just mmap() files in a suffiently large tmpfs. And unmapped pages stay in the page cache, if not replaced by others.
Thanks!
Then I have to get an Opteron machine. I am currently running debian testing/sid. Does anyone know if can get 64 bit support out of the box with this dist. just recompiling the 2.6 kernel setting cpu type to Opteron? Or do I have to go for another dist. like Enterprise Red Hat or the 64 bit Suse?
debian amd64 port is usable
Just recompiling the kernel is not enough, because of libraries and so on, but
http://www.debian.org/ports/amd64/
The port is still unofficial (i.e. what other distros would call ready :) and pure 64-bit, i.e. if you need to run (legacy, closed source) 32bit-programs, you have to install a second i386 debian inside a chroot environment. Pure 64 is no problem however, because everything in debian is compiled from source, this is like any other architecture, like alpha, ppc, etc, which can't run i386 executables either. You get a perfectly normal debian environment, with the added bonus, that you can add said chroot or dual-boot, if you need to (and i just found a »ia32-libs« packet i didn't know of before...). Only binary only kernel modules like graphics drivers (nvidia, ati) might be a problem, if you need them, last time i looked, the manufacturers weren't ready yet (i.e. every distro has this problem).
(Parallel lib directories for 32 and 64 are still worked on, they don't want to release something, which is incompatible to others and not generic enough.)
Another (horrible) option
If you are willing to play games and modify the application, you can actually access the whole range of memory, just not all of it at the same time. It's kind of like the old "overlay" scheme on PC's or a manual form of swapping. But in this case you are only using it to get around virtual memory size issues (32 bit pointers) instead of a lack of physical memory or address space limits on physical memory.
How to do it? Attach and detach shared memory segments. Depending on how your application is designed this will be moderately difficult to implement or such a huge pain that buying new hardware will be the only worthwile course of action.
You can get 4G for process with a patch
You can get as much as 4G of memory for each process by using the 4G/4G patch from Ingo Molnar - just search for it. It is included by default in Fedora - search for it if you have another distro. Disadvantage - the kernel becomes a bit slower. I can't say how much, because it depends from your workload, and even with the same test, it causes a bigger slowdown if it's Andrea Arcangeli to run the test :-)
See http://kerneltrap.org/node/view/2891.