How many threads?
1! The GIL isn't a very good context switcher, so just let the OS do what it does best. To add more concurrency then, just add more zopes.
The only time where you should use more threads is in the case that there is the potential for a web service deadlock. For example, plone contacts service X which needs more info from plone... and deadlock. You shouldn't code your services for this, but it happens to the best of us in unexpected ways. If you know there is the potential for this to happen, use 2 threads to be safe (this way you can handle 2 requests at any given time).
So what the real question is here, is how many zopes? Simply put, as many as possible but not a single one more. Start with by oversubscribing 2 zopes per cpu. So, if you have a 4 core cpu, start with 8 zopes. The goal is to utilize 50% of load on average so that if things get crazy the server can spike to 100%. If you aren't at 50% load on average, add more zopes. If you are higher than that, reduce the number.
An even better question is how many zopes do I need to handle X requests per second? Glad you asked! It depends. IMHO, a lot of the figures on plone performance are very misleading since they assume you are utilizing caching and/or serving silly things like user.gif. Requests per second should really be called "Rendered requests per second" and any long term plone administrator will tell you that is a much lower number. For example with Plone 2.5 and custom archetypes, we squeeze out 2.5 rendered requests per second on average. Of course some pages will be faster and some (especially writes) will be slower and you, mr or mrs sys admin, need to measure that. Be realistic and don't expect a lot. If you want to handle 10 requests per second with an average of 2.5 requests per second, then you are looking at 4 managing zopes.
The goal in any web app should be to rendered EVERY page in 200-300 milliseconds. If your page is rendered in 300 milliseconds, it will get to the user in about 1 second, which is the limit of most users attention span. If you have pages which take a significant more amount of time (i.e. reports), consider setting some zopes which specifically handle long running requests. You want fast to remain fast, and those long requests will affect your system.
How much zeo cache?
The zeo cache a disk buffer (usually in /tmp) of all the objects you are accessing from a zeo client. This is especially useful in the case that you are not running zope and zeo on the same box and you have to make a network request to get your db objects. If you run your zeo and zope on the same box, then the answer is none! No need to waste /tmp space. (I think that you can't technically set it to 0, but set it close to that and it shouldn't make a difference). If you are NOT on the same box, then set it as high as you can without blowing your tmp directory, keeping in mind that disk will utilize available ram to cache that disk movement.
How many zodb cache objects?
The simple answer is: as many as you can. This is different than the zeo cache in that its a cache of the objects, entirely in memory for performance on the SAME machine. If you have been a good boy or girl and separated your portal_catalog on a new mount, try to keep the whole catalog in memory. For example, set this to 100,000 and move up from there. These objects are so small that you won't see memory fill up too much.
As far as non-catalog mounts, the answer is not so clear. Start as high as you can i.e. 40,000, and monitor performance. Keep lowering until performance is impacted. If you immediately get worse performance, then turn around and start adding more. If you are worried about a RAM hungry process, setup monit to restart your zope once it reaches x% or RAM. It's better to restart and have performance then to be proud of a long running, under performing service.
Tuning the Operating System for Zeo
Do any lookup on tuning linux for hosting a databases, read, understand, and apply. I don't think you are going to get any huge performance gains from any of these (I didn't see any noticeable difference) but they didn't hurt so... here they are. In general, you will get lots of tips from googling "TUNE ORACLE LINUX". If an OS can handle oracle, it can handle zeo.
Changing Disk Algorithms
Of all the things you can tweak the OS itself, changing the scheduler algorithm on the zeo server box is a good place to start. Myroslav has some info to put here....
Messing with noatimeLinux marks every access to files in the system, which can hurt database performance. Changing Data.fs et al to not record this is simple and easy.
Optimizing with multiple disks/SANs
Of course, the best thing you can do is shard your dbs and put them on different disks, hopefully as part of a bigger/faster/awesomer disk array. At minimum, put your catalog on a different disk than the rest of your dbs and do NOT log on the same disk either. That's 3 disk partitions/mounts minimum. Plan for this from the beginning and your life will be much easier in the long term.
Do you RAID?
If you use RAID 5, none of this will help with the pain you are about to suffer. If you can afford it, go RAID 10 and if you can't, stick to RAID 1.
Setting Check Intervals