Ian Boston
Over the standard Apache 2 setup on ssl we have seen speedups of about 10x (by using memory scorebords and removing modules) and have been able to run the Apache front end with about 1024 workers and a matching number of tomcat threads (BTW this is another tomcat app not Sakai). We have found using httpperf to test that if you dont allow tomcat to have enough worker threads to service the apache workers, you will get requests in cycle in the apache layer and the response times will go through the roof.
Also
If you dont have a large enough DB pool to service the worker threads in tomcat, requests will wait in cycle in tomcat..... which is even worse as then the request cycle objects in tomcat get out of the eden memory pool and start to cost.
Also you might consider serving /library and other areas known to be static direct through Apache, typically Tomcats static file server will run at 30-50ms per request, but we have seen apache do the same in the 5-10ms per request (comparitive, not absolute), thats 3-5 times the request throughput.
So Apache/OpenSSL/APR in front is faster but you have to make 100% certain that the setup behaves well under load. We have found that testing a single non authed URL with some DB access (very hard) will validate this (eg the gateway)
Incidentally, we found the ab and ab2 (Apache Benchmark) were not fast enough to do this property and it needed httpperf to really stress a well tuned stack on gateway.
Daniel can give more details on this as he did the work.
Daniel Parry
http://www.hpl.hp.com/research/linux/httperf/
is a useful tool for heavy benchmarking. You may want to up your
nofile limit if you really want to go flat out. Standard Apache
SSL conf OOTB is really not very optimal. If it would be useful, I
could put performance tweaks up on confluence. Where woud be a
good spot? Generally, it's a good idea to correlate your
MaxClients in apache with your MaxThreads in tomcat.
Tony Atkins
Although we use apache an awful lot in other areas, we use tomcat with APR for both collab and our local Sakai instance with no problems.
I think it might be more productive to keep following up on the netstat end. Are these connections that are staying open from end users, a load balancer, etc.?