We do have limits to avoid memory getting too full.
The common solution is to have a reserve (min_free_kbytes). The problem
with the network stack seems to be that the amount of reserve needed
cannot be predicted accurately.
The solution may be as simple as configuring the reserves right and
avoid the unbounded memory allocations. That is possible if one
would make sure that the network layer triggers reclaim once in a
while.
-