From: Daniel Spång <daniel.spang@...>
Subject: Out of memory management in embedded systems
 [1]Date: Sep 28, 8:55 am 2007

Applications with dynamic input and dynamic memory usage have some
issues with the current overcommitting kernel. A high memory usage
situation eventually results in that a process is killed by the OOM
killer. This is especially evident in swapless embedded systems with
limited memory and no swap available.

Some kind of notification to the application that the available memory
is scarce and let the application free up some memory (e.g., by
flushing caches), could be used to improve the situation and avoid the
OOM killer. I am currently not aware of any general solution to this
problem, but I have found some approaches that might (or might not)
work:

o Turn off overcommit. Results in a waste of memory.

o Nokia uses a lowmem security module to signal on predetermined
thresholds. Currently available in the -omap tree. But this requires
manual tuning of the thresholds.
http://www.linuxjournal.com/article/8502 [2]

o Using madvise() with MADV_FREE to get the kernel to free mmaped
memory, typically application caches, when the kernel needs the
memory.

o A OOM handler that the application registers with the kernel, and
that the kernel executes before the OOM-killer steps in.

Does it exist any other solutions to this problem?

Daniel
-


From: linux-os (Dick Johnson) <linux-os@...>
Subject: Re: Out of memory management in embedded systems
 [2]Date: Sep 28, 9:09 am 2007

On Fri, 28 Sep 2007, [iso-8859-1] Daniel Spång wrote:

> Applications with dynamic input and dynamic memory usage have some
> issues with the current overcommitting kernel. A high memory usage
> situation eventually results in that a process is killed by the OOM
> killer. This is especially evident in swapless embedded systems with
> limited memory and no swap available.
>
> Some kind of notification to the application that the available memory
> is scarce and let the application free up some memory (e.g., by
> flushing caches), could be used to improve the situation and avoid the
> OOM killer. I am currently not aware of any general solution to this
> problem, but I have found some approaches that might (or might not)
> work:
>
> o Turn off overcommit. Results in a waste of memory.
>
> o Nokia uses a lowmem security module to signal on predetermined
> thresholds. Currently available in the -omap tree. But this requires
> manual tuning of the thresholds.
> http://www.linuxjournal.com/article/8502 [3]
>
> o Using madvise() with MADV_FREE to get the kernel to free mmaped
> memory, typically application caches, when the kernel needs the
> memory.
>
> o A OOM handler that the application registers with the kernel, and
> that the kernel executes before the OOM-killer steps in.
>
> Does it exist any other solutions to this problem?
>
> Daniel
> -

But an embedded system contains all the software that will
ever be executed on that system! If it is properly designed,
it can never run out of memory because everything it will
ever do is known at design time.

This should never be an issue with an embedded system.
If you have such a system issue, then you have
application(s) that have memory leaks because of
improper design or coding. For instance, there is
a common open-source web-server that is used in some
embedded systems. It has memory leaks. The solution,
if the server can't be fixed, is to execute a supervisor
process which periodically shuts it down and restarts
it --ugly, but effective if the developers refuse to
accept patches.

You shouldn't expect a kernel to be modified to "fix"
broken application code.

Cheers,
Dick Johnson
Penguin : Linux version 2.6.22.1 on an i686 machine (5588.29 BogoMips).
My book : http://www.AbominableFirebug.com/ [4]
_

****************************************************************
The information transmitted in this message is confidential and may be privileged.  Any review, retransmission, dissemination, or other use of this information by persons or entities other than the intended recipient is prohibited.  If you are not the intended recipient, please notify Analogic Corporation immediately - by replying to this message or by sending an email to DeliveryErrors@analogic.com [5] - and destroy all copies of this information, including any attachments, without reading or disclosing them.

Thank you.
-


From: Daniel Spång <daniel.spang@...>
Subject: Re: Out of memory management in embedded systems
 [5]Date: Sep 28, 9:30 am 2007

On 9/28/07, linux-os (Dick Johnson) <linux-os@analogic.com> wrote:
>
> On Fri, 28 Sep 2007, [iso-8859-1] Daniel Spång wrote:
>
> > Applications with dynamic input and dynamic memory usage have some
> > issues with the current overcommitting kernel. A high memory usage
> > situation eventually results in that a process is killed by the OOM
> > killer. This is especially evident in swapless embedded systems with
> > limited memory and no swap available.
> >
> > Some kind of notification to the application that the available memory
> > is scarce and let the application free up some memory (e.g., by
> > flushing caches), could be used to improve the situation and avoid the
> > OOM killer. I am currently not aware of any general solution to this
> > problem, but I have found some approaches that might (or might not)
> > work:
> >
> > o Turn off overcommit. Results in a waste of memory.
> >
> > o Nokia uses a lowmem security module to signal on predetermined
> > thresholds. Currently available in the -omap tree. But this requires
> > manual tuning of the thresholds.
> > http://www.linuxjournal.com/article/8502 [6]
> >
> > o Using madvise() with MADV_FREE to get the kernel to free mmaped
> > memory, typically application caches, when the kernel needs the
> > memory.
> >
> > o A OOM handler that the application registers with the kernel, and
> > that the kernel executes before the OOM-killer steps in.
> >
> > Does it exist any other solutions to this problem?
> >
> > Daniel
> > -
>
> But an embedded system contains all the software that will
> ever be executed on that system! If it is properly designed,
> it can never run out of memory because everything it will
> ever do is known at design time.

Not if its input is not known beforehand. Take a browser in a mobile
phone as an example, it does not know at design time how big the web
pages are. On the other hand we want to use as much memory as
possible, for cache etc., a method that involves the kernel would
simplify this and avoids setting manual limits.

Daniel
-


From: linux-os (Dick Johnson) <linux-os@...>
Subject: Re: Out of memory management in embedded systems
 [6]Date: Sep 28, 10:04 am 2007

On Fri, 28 Sep 2007, [iso-8859-1] Daniel Spång wrote:

> On 9/28/07, linux-os (Dick Johnson) <linux-os@analogic.com> wrote:
>>
>> On Fri, 28 Sep 2007, [iso-8859-1] Daniel Spång wrote:
>>
>>> Applications with dynamic input and dynamic memory usage have some
>>> issues with the current overcommitting kernel. A high memory usage
>>> situation eventually results in that a process is killed by the OOM
>>> killer. This is especially evident in swapless embedded systems with
>>> limited memory and no swap available.
>>>
>>> Some kind of notification to the application that the available memory
>>> is scarce and let the application free up some memory (e.g., by
>>> flushing caches), could be used to improve the situation and avoid the
>>> OOM killer. I am currently not aware of any general solution to this
>>> problem, but I have found some approaches that might (or might not)
>>> work:
>>>
>>> o Turn off overcommit. Results in a waste of memory.
>>>
>>> o Nokia uses a lowmem security module to signal on predetermined
>>> thresholds. Currently available in the -omap tree. But this requires
>>> manual tuning of the thresholds.
>>> http://www.linuxjournal.com/article/8502 [7]
>>>
>>> o Using madvise() with MADV_FREE to get the kernel to free mmaped
>>> memory, typically application caches, when the kernel needs the
>>> memory.
>>>
>>> o A OOM handler that the application registers with the kernel, and
>>> that the kernel executes before the OOM-killer steps in.
>>>
>>> Does it exist any other solutions to this problem?
>>>
>>> Daniel
>>> -
>>
>> But an embedded system contains all the software that will
>> ever be executed on that system! If it is properly designed,
>> it can never run out of memory because everything it will
>> ever do is known at design time.
>
> Not if its input is not known beforehand. Take a browser in a mobile
> phone as an example, it does not know at design time how big the web
> pages are. On the other hand we want to use as much memory as
> possible, for cache etc., a method that involves the kernel would
> simplify this and avoids setting manual limits.
>
> Daniel
>

Any networked appliance can (will) throw data away if there are
no resources available.

The length of a web-page is not relevent, nor is the length
of any external data. Your example will buffer whatever it
can and not read anything more from the external source until
it has resources available unless it is broken.


Cheers,
Dick Johnson
Penguin : Linux version 2.6.22.1 on an i686 machine (5588.29 BogoMips).
My book : http://www.AbominableFirebug.com/ [8]
_

****************************************************************
The information transmitted in this message is confidential and may be privileged.  Any review, retransmission, dissemination, or other use of this information by persons or entities other than the intended recipient is prohibited.  If you are not the intended recipient, please notify Analogic Corporation immediately - by replying to this message or by sending an email to DeliveryErrors@analogic.com [9] - and destroy all copies of this information, including any attachments, without reading or disclosing them.

Thank you.
-


From: Rik van Riel <riel@...>
Subject: Re: Out of memory management in embedded systems
 [9]Date: Sep 28, 10:17 am 2007

On Fri, 28 Sep 2007 10:04:23 -0400
"linux-os \(Dick Johnson\)" <linux-os@analogic.com> wrote:
> On Fri, 28 Sep 2007, [iso-8859-1] Daniel Spång wrote:
> 
> > On 9/28/07, linux-os (Dick Johnson) <linux-os@analogic.com> wrote:
> >>
> >> On Fri, 28 Sep 2007, [iso-8859-1] Daniel Spång wrote:

> >>> Some kind of notification to the application that the available memory
> >>> is scarce and let the application free up some memory (e.g., by
> >>> flushing caches), could be used to improve the situation 

> Any networked appliance can (will) throw data away if there are
> no resources available.

That is exactly what Daniel proposed in his first email.

I think his idea makes sense.

-- 
"Debugging is twice as hard as writing the code in the first place.
Therefore, if you write the code as cleverly as possible, you are,
by definition, not smart enough to debug it." - Brian W. Kernighan
-


From: Daniel Spång <daniel.spang@...>
Subject: Re: Out of memory management in embedded systems
 [9]Date: Sep 28, 10:14 am 2007

On 9/28/07, linux-os (Dick Johnson) <linux-os@analogic.com> wrote:
>
> On Fri, 28 Sep 2007, [iso-8859-1] Daniel Spång wrote:
>
> > On 9/28/07, linux-os (Dick Johnson) <linux-os@analogic.com> wrote:
> >>
> >> But an embedded system contains all the software that will
> >> ever be executed on that system! If it is properly designed,
> >> it can never run out of memory because everything it will
> >> ever do is known at design time.
> >
> > Not if its input is not known beforehand. Take a browser in a mobile
> > phone as an example, it does not know at design time how big the web
> > pages are. On the other hand we want to use as much memory as
> > possible, for cache etc., a method that involves the kernel would
> > simplify this and avoids setting manual limits.
> >
> > Daniel
> >
>
> Any networked appliance can (will) throw data away if there are
> no resources available.
>
> The length of a web-page is not relevent, nor is the length
> of any external data. Your example will buffer whatever it
> can and not read anything more from the external source until
> it has resources available unless it is broken.

And how do you determine when no resources are availabe? We are using
overcommit here so malloc() will always return non null.
-


From: linux-os (Dick Johnson) <linux-os@...>
Subject: Re: Out of memory management in embedded systems
 [9]Date: Sep 28, 11:16 am 2007

On Fri, 28 Sep 2007, [iso-8859-1] Daniel Spång wrote:

> On 9/28/07, linux-os (Dick Johnson) <linux-os@analogic.com> wrote:
>>
>> On Fri, 28 Sep 2007, [iso-8859-1] Daniel Spång wrote:
>>
>>> On 9/28/07, linux-os (Dick Johnson) <linux-os@analogic.com> wrote:
>>>>
>>>> But an embedded system contains all the software that will
>>>> ever be executed on that system! If it is properly designed,
>>>> it can never run out of memory because everything it will
>>>> ever do is known at design time.
>>>
>>> Not if its input is not known beforehand. Take a browser in a mobile
>>> phone as an example, it does not know at design time how big the web
>>> pages are. On the other hand we want to use as much memory as
>>> possible, for cache etc., a method that involves the kernel would
>>> simplify this and avoids setting manual limits.
>>>
>>> Daniel
>>>
>>
>> Any networked appliance can (will) throw data away if there are
>> no resources available.
>>
>> The length of a web-page is not relevent, nor is the length
>> of any external data. Your example will buffer whatever it
>> can and not read anything more from the external source until
>> it has resources available unless it is broken.
>
> And how do you determine when no resources are availabe? We are using
> overcommit here so malloc() will always return non null.
>

A networked appliance using embedded software is not your daddy's
Chevrolet. Any task that is permanent needs to allocate all its
resources when it starts. That's how it knows how much there are,
and incidentally, it doesn't do it blindly. The system designer
must know how much memory is available in the system and how much
is allocated to the kernel.

The fact that you can give a fictitious value to malloc() is not
relevant. If you don't provide resources for malloc(), like
(ultimately) a swap file, then you can't assume that it can do
any design work for you.

An embedded system is NOT an ordinary system that happens to
boot from flash. An embedded system requires intelligent design.

It is important to understand how a virtual memory system
operates. The basics are that the kernel only "knows" that
a new page needs to be allocated when it encounters a trap
called a "page fault." If you don't have any memory resources
to free up (read no swap file to write a seldom-used task's
working set), then you are screwed --pure and simple. So,
if you don't provide any resources to actually use virtual
memory, then you need to make certain that virtual memory
and physical memory are, for all practical purposes, the same.

With embedded servers, it's usually very easy to limit the
number of connections allowed, therefore the amount of
dynamic resources that must be provided. With clients
it should be equally easy, but generic software won't
work because, for instance, Mozilla doesn't keep track
of the number of "windows" you have up and the number
of connections you have. HOWEVER, remember that malloc()
is a library call. You can substitute your own using
LD_PRELOAD, they keeps track of everything if you must
use generic software.


Cheers,
Dick Johnson
Penguin : Linux version 2.6.22.1 on an i686 machine (5588.29 BogoMips).
My book : http://www.AbominableFirebug.com/ [10]
_

****************************************************************
The information transmitted in this message is confidential and may be privileged.  Any review, retransmission, dissemination, or other use of this information by persons or entities other than the intended recipient is prohibited.  If you are not the intended recipient, please notify Analogic Corporation immediately - by replying to this message or by sending an email to DeliveryErrors@analogic.com [11] - and destroy all copies of this information, including any attachments, without reading or disclosing them.

Thank you.
-