login
Search
Hello everyone,
We have an embedded MPC8347 board with 1GB of RAM and a 1GB compact flash (CF) drive.
We recently had some RAM issues (applications allocated in total more than 1GB of memory due to overcommit) and thus OOM was triggered when some application tried to commit the last page.
We tried to handle it using 'overcommit_memory=2, overcommit_ratio=100' and it does actually work (all applications cannot allocate more than the total RAM available).
The problem is that when working a lot with the CF, and being on the edge of the free available RAM (i.e. 100MB free RAM), OOM is triggered, killing an innocent application that probably did nothing.
We tried narrowing it down to a simple recreatable command - 'cat < /dev/urandom > /lala' where / is mounted to the CF. The above command will eventually trigger OOM. It's worth mentioning that this command is being run as root, along with all other applications (this is a whole other discussion - should they be run as root or not, let's leave it as is :-)).
My assumption was that the pagecache is being filled up so bad that there are no free pages available at all, but shouldn't the Linux Memory Manager free up page caches (flush to CF?) and then have some more pages available?
Another idea we had in mind was using 'overcommit_ratio=95' so that 5% will always be free, but this did not work well too.
We dug in a bit into the Linux kernel and found out that the last 3% of RAM is reserved to root, thus if we ran the above command as root, we will be able to consume all memory... Perhaps this is the case?
It would be great if anyone could shed some light on the issue.
Thanks a lot in advance.
man ulimit
cat < /dev/urandom > /lalaThis will eventually fill you flash card, and at that point, cat will NOT get an ENOSPC error from libc. The kernel will allow you to add more and more data to /lala, but this will lock pages in ram until you truncate, move, close and delete /lala or run out of memory. When you run out of memory, I suspect the OOM killer does not spot that those locked pages are caused by cat, so it kills something else instead. Even if the OOM killer does kill the cat, /lala still exists and can be copied to another device (if killing cat released enough ram to run cp).
Relying on the OOM killer does not make a stable system. If your task can be completed with a finite amount of memory, add enough memory or swap space to complete the task or have the task predict the amount of memory required and fail if there is not enough. If the task requires an infinite amount of memory, fix the leak.
Thanks for your reply. Well,
Thanks for your reply.
Well, as for swap, I wouldn't want swap space on a CF since I'm afraid it will greatly affect its wear level.
What you're saying does make sense to me, but I used overcommit in the first place so that the processes won't succeed in allocating more memory than they can actually use. The task that is waking up the OOM killer is only writing big files to the CF, and not allocating any more RAM...
throttling
writing to disk first writes to RAM and sets the pages dirty to get them written out to disk some time in the future. if a process writes faster than the disk can handle, dirty pages accumulate in RAM and keep the disk busy even if the process finished long ago. to not make these dirty pages disturb the operation of the rest of the system, the writing process gets throttled if the amount of RAM allocated to this gets too big. if writing on CF is particularly slow, you may rely more on this mechanism. you could fiddle with parameters in /proc/sys/ that control this behaviour.
Well at least you can
Well at least you can protect some processes from being OOM-killed. echo -17 > /proc//oom_adj.
I run with overcommit most of my machines and set it up for init and sshd in case something very bad happens.
Mysteriously, the init process has a very bad oom score. Dunno why (cause I dunno the OOM score formula). My guess is because it forks lots of processes.
I would say to find out if
I would say to find out if this is a bug, try it on a real desktop type machine, and see if you can repro on a HD. If not, could be the CF driver losing memory.
The CF has an IDE controller
That's a good idea, but the CF has an IDE controller, so basically it's the IDE driver in Linux and not a custom CF driver...