On Thu, Dec 13, 2007 at 12:35:03PM +0100, Otto Moerbeek wrote:

:

quoted text
> > > or it might be a program "forgetting" to do a close.

> >

> > Does select() notify the application of FIN from the other side?

> >

> > If not, that would explain things, it wouldn't be reasonable for

> > httpd to manually try and receive from all sockets in keepalive

> > to see whether it needs to close the socket, since it will only

> > wait KeepAliveTimeout (default 15s) before it closes them anyway.

>

> Nice suggestion, but if you've marked the fd for read I would expect

> select to notify if the other side does a shutdown(SHUT_WR).

>

> Other scenarios are also thinkable: like the server socket being

> blocked because of outgoing data that cannot be written out. That

> might prevent the server from doing a close too. But in the end the

> close will happen, otherwise you would run out of fd's very soon.

>

> 	-Otto

The behaviour is starting to make sense now. Scenario:
* The client connects to the server, sends its request and

  then closes the socket, that is shutdown() aka half-close.

  It can still read the reply.

* The server accepts the connection, reads the request,

  and may or may not notice that the client has done

  a shutdown() - it is not important. Nevertheless the

  server can not close the socket since it has a

  reply to deliver. And the server host TCP stack

  has noticed the shutdown() so the socket already

  enters CLOSE_WAIT.

* The server starts sending the reply which may be large

  e.g a file download. In the middle of this transfer

  the client's ethernet cable gets plugged out, the

  client host gets powered off, a firewall in the

  path goes bananas or whatnot.

* The server is now stuck in a write() call since the

  server host TCP stack has to wait quite a while

  to be sure the connection is really dead.

  And the state is still CLOSE_WAIT.
If the client program would die, the client host TCP

stack would close the socket and tell the server host

TCP stack, that would fail the hanging write() call.

So there must be a harder error such as network

outage or power outage to induce this problem.
If this scenario is correct, there is nothing to do

about it, except decreasing the likelyhood of the

server socket being half-closed while sending

the reply, and having KeepAliveTimeout in

httpd.conf at its default (15) or slightly lower

seems to do the trick. But I do not know how.
If there is some quirk in httpd's implementation

of the KeepAliveTimeout that makes it not notice

the half-close and keeps the socket open the whole

KeepAliveTimeout, that would explain it.
-- 
/ Raimo Niskanen, Erlang/OTP, Ericsson AB