Thanks for the hints. I've replied the questions below.

I was still wondering what could be considered "maximum" session 
concurrency that I could expect, with various hardware combinations? Is 
anyone that can tell me if it could be feasible with OpenBSD and better 
hardware? Even if we have to move to a different platform than i386, 
like maybe a Sun Fire T1000, as I don't see that as being a problem if 
it solves our issues. What we would like most if possible is to find 
something that could scale in the million concurrent sessions, but with 
a couple of thousands of new sessions per second. I know it's something 
very hardware demanding and even most enterprise class firewalls like 
Juniper and Fortinet don't scale much more than a million even on their 
higher end models, so that's why I'm curious as to what I could expect a 
PF setup to scale.

Thanks again,
Steve Johnson

Stuart Henderson wrote:
quoted text
> On 2008-05-08, Steve Johnson <maillist@sjohnson.info> wrote:
>   
>> Is the congestion issue that I'm getting considered "normal" under that
>> type of traffic and with the present hardware? Are there any other
>> settings that I should look into tweaking?
>>     
>
>   
>>> CPU states:  0.2% user,  0.0% nice,  1.9% system, 38.1% interrupt, 59.8% idle
>>>       
>
> cpu% in interrupt (which includes PF processing) will almost certainly 
> spike higher than this instantaneous reading at times, leading to congestion.
>
>   
>>> scrub all random-id fragment reassemble
>>>       
>
> do you need to scrub/random-id _all_ of the traffic, in+out, on all
> interfaces?
>   
Mostly on the outside, but it was a "nice to have". I'll test to see if 
it helps a lot by just scrubbing in on the outside interface.
quoted text
> you're natting on the network Henning suggested you 'set skip' on
> aren't you... if you can live with that breaking to test, try the 'set
> skip' anyway and see if it helps enough to be worth working out
> something else for the nat.
>   
Yes, I am at the moment. However, it's the default gateway interface and 
so I can't really skip filtering on that interface. I'll try skipping on 
the inside interface, since the traffic will have already been validated 
and see if that changes much. I'm also trying to just skip session based 
filtering on that service as well, since it's one of the first pass 
quick rule and see if it helps. I was still seeing a bit of congestion 
though.
quoted text
> btw personally I'd rather have all the information in the list post
> than have to fetch it by http, I expect it's probably the same for others..
>   
Good to know, thought most people didn't want to have too much stuff 
pasted in the emails.