I am trying to improve my performance and fix my problem on httpd, but

look like I am hitting the roof regardless if I test in lab using an old

850MHz i386 or an new AMD64 at 1.6GHz. Both have > 2GB of ram, so that's

the issue both have. I can't pass more then ~300 to 325 simultaneous

httpd process and timeout goes jump high.
So, I guess may be the limit are in the connection process of the TCP

stack, more then the httpd itself. But I am at a lots as to where to

look. Tested both on 4.1 and 3.9 just to see.
Where are the OS bottleneck that I can may be improve here?
Please read for more details and more can be provided as well.
I need some help as I even went as far as order 4x X4100 with 2x dual

core processor 2.4GHz and 2x 10K SAS drives in them with 8GB of ram as

well, so 4GB per processors and I am afraid to hit the same limitations.

There isn't any reason that I shouldn't be able to pass these limits.
I don't have the new Sun yet, may be a week before I have them, but I am

trying to get ahead of the setup to fix my problem and test in lab. It

really is a capacity issue and look likes putting more powerful hardware

at it will not fix it.
I have:
# sysctl kern.maxproc

kern.maxproc=2048
Both also have noatime setup on the partition that the web files comes

from and I even send the logs of httpd to >/dev/null to be sure it's not

writing logs that would slow it down.
I use http_load to test my configuration and changes, but I am not

successful at improving it more. Look like connections are timing out

and I can't get more then ~ 300 process serving for httpd. Yes I have

also increase and recompile the httpd to allow more then the hard limit

of 250 and I can start 1500 httpd process if I want and they do run, but

they do not server traffic looks like and I am still getting timeout.
Even if I start "StartServers 2500" httpd process to be sure I don't run

out, or that the start of additional one is not the limit here, I can't

get more then about ~ 300 successful parallel at one with a decent timeout:
# http_load -parallel 500 -fetches 2500 -timeout 20 /tmp/www2

2500 fetches, 500 max parallel, 1.25616e+07 bytes, in 41.815 seconds

5024.62 mean bytes/connection

59.7872 fetches/sec, 300408 bytes/sec

msecs/connect: 1868.76 mean, 18014 max, 0.597 min

msecs/first-response: 2741.85 mean, 19968.2 max, 4.005 min

345 timeouts

345 bad byte counts

HTTP response codes:

   code 200 -- 2155
# http_load -parallel 500 -fetches 2500 -timeout 60 /tmp/www2

http://www2.netcampaign.com/: byte count wrong

http://www2.netcampaign.com/: byte count wrong

2500 fetches, 500 max parallel, 1.37498e+07 bytes, in 42.3446 seconds

5499.91 mean bytes/connection

59.0394 fetches/sec, 324711 bytes/sec

msecs/connect: 2064.88 mean, 42024.6 max, 0.621 min

msecs/first-response: 2408.3 mean, 21687.7 max, 4.136 min

2 bad byte counts

HTTP response codes:

   code 200 -- 2500
The response time goes pretty high with multiple parallel fetch, witch

is expected to be slower yes, but how can I improve that? See the jump

between 300 to 400 in the AMD64 version below. Even if I say to use 400

parallel connections, looking at the box top and all, looks like I can

pass ~325? Both in old server and new server. So, I guess it must be

something in the kernel setup that limit that?
Any clue would be appreciated and when can I possibly look for that?
Example:
OLD i386 850MHz

# http_load -parallel 100 -fetches 2500 -timeout 60 /tmp/www2

2500 fetches, 100 max parallel, 1.37438e+07 bytes, in 32.1498 seconds

5497.53 mean bytes/connection

77.7609 fetches/sec, 427493 bytes/sec

msecs/connect: 96.7252 mean, 6008.79 max, 0.49 min

msecs/first-response: 985.229 mean, 11051.5 max, 3.514 min

HTTP response codes:

   code 200 -- 2500
New AMD64 1,6GHz

# http_load -parallel 100 -fetches 2500 -timeout 60 /tmp/www1

2500 fetches, 100 max parallel, 1.38878e+07 bytes, in 12.8811 seconds

5555.11 mean bytes/connection

194.082 fetches/sec, 1.07815e+06 bytes/sec

msecs/connect: 84.7087 mean, 6003.59 max, 0.351 min

msecs/first-response: 236.256 mean, 1921.73 max, 2.066 min

HTTP response codes:

   code 200 -- 2500
# http_load -parallel 200 -fetches 2500 -timeout 60 /tmp/www1

2500 fetches, 200 max parallel, 1.36869e+07 bytes, in 11.8518 seconds

5474.78 mean bytes/connection

210.939 fetches/sec, 1.15484e+06 bytes/sec

msecs/connect: 178.411 mean, 6004.23 max, 0.353 min

msecs/first-response: 350.587 mean, 2427.51 max, 2.297 min

HTTP response codes:

   code 200 -- 2500
# http_load -parallel 300 -fetches 2500 -timeout 60 /tmp/www1

2500 fetches, 300 max parallel, 1.37912e+07 bytes, in 11.8928 seconds

5516.47 mean bytes/connection

210.211 fetches/sec, 1.15962e+06 bytes/sec

msecs/connect: 612.953 mean, 8995.56 max, 0.344 min

msecs/first-response: 266.107 mean, 2345.62 max, 2.069 min

HTTP response codes:

   code 200 -- 2500
# http_load -parallel 400 -fetches 2500 -timeout 60 /tmp/www1

2500 fetches, 400 max parallel, 1.35291e+07 bytes, in 18.209 seconds

5411.63 mean bytes/connection

137.295 fetches/sec, 742989 bytes/sec

msecs/connect: 1100.52 mean, 18014 max, 0.459 min

msecs/first-response: 262.247 mean, 7237.17 max, 2.157 min

HTTP response codes:

   code 200 -- 2500