#93: Optimize Plone for speed

It's important that Plone becomes faster - although Plone is made to be fronted with Apache/Squid caching, it makes sense to optimize page load speeds for big deployments - and it also makes a difference in the day-to-day development usage.

This proposal mostly concerns itself with optimization of page views for anonymous users. While editing speed matters too, this is a more complex part to optimize, and is outside the scope of this PLIP, although should obviously be a focus area in the future.

After doing some initial, simple profiling, I identified some attack vectors that would give a lot of speed-up with little effort.

The areas where I think we should focus our attentions are:

1. The breadcrumbs code
2. The 'listMetaTags' method
3. The calendar portlet

Here is some output from 10 page loads as Anonymous User of the front page of a newly created Plone site on my 1.5GHz PowerBook G4 running in debug mode - Plone 2.1 branch (Revision: 6338). I use PTProfiler as my main analysis tool for this.

The table has been edited to show only the main offenders speed-wise. I have highlighted cases of very expensive single calls or with an excessive number of calls to methods.

As you can see, the main offender speed-wise is the listFilteredActionsFor method - although there's not much we can do about this, since it's a CMF construct. Initial testing with CMF 1.5 (which has lazy action evaluation) didn't show any improvement here - so we'll skip this as a target for our optimizations.

I have grouped some of the methods from the calendar portlet at the bottom of the table, and given them an alternate background color. As you see, there are a number of calls here that are done an excessive amount of times.

Our main targets and some comments about each one:

Breadcrumbs

This code is essentially trying to do the same as the nav tree - just in a flat, depth-only way. It has lots of exception handling, "clever" and unnecessary code, and things that are total overkill for a breadcrumb implementation. If people want to support all the special cases, that's fine - but the default implementation should not. The fact that breadcrumb.py is 166 (!) lines long with permission checks and multiple conditional branches should be a good indicator of this.
My suggestion: See if we can re-use code from the nav tree implementation and make it more efficient, and make it a bit stupider if necessary.

listMetaTags method

I have no idea why this thing is so expensive, but it is. Tiran recently moved it to a tool to see if running it in unrestricted code would make it faster, but it only made a marginal difference, easily attributable to testing variations.

Calendar portlet

This thing is a chapter in itself. It has an incredible amount of calls (I have only included the most exceptional ones, there are lots of others), and does multiple tal:defines inside tal:repeats, among other things.
My suggestion: This code should be rewritten. It's currently building the table for the calendar in a very inefficient way, and we should also remove the pop-up divs and let it use the HTML title attribute instead, like the rest of Plone. This is also better for accessibility, and will remove half (well, almost ;) of the excessive white space in the Plone HTML output.

Some interesting numbers from Plone 2.0.5, mainly showing that listMetaTags is less expensive here, and also that listFilteredActions takes less time here (what is adding a lot of actions in 2.1, and how can we minimize the impact?):

On a positive note, we see that getAllowedTypes takes up a lot of time in 2.0.5, and that it has been totally eliminated from the 2.1 anonymous view. Also eliminated is the nav tree cost, which is negligible in 2.1. All in all, we've eliminated about 1 second on the 10 page loads with the new stuff in 2.1, but something else is bogging us down.

Unfortunately, something is sucking up the CPU time we won with the optimizations. The AT-based types?

Investigated the listMetaTags part after Tiran moved it to unrestricted code - it is a bit faster, but not a lot. There is a lot of crazy checks and conditionals going on to produce DC.* tags that none of the web crawlers use, and most interpret as line noise.

My proposal is to introduce a switch in site_properties called exposeDCMetaTags that is off by default, since no search engines or crawlers use it, and let the 3 people in the world (of which 2 are librarians and the last one flunked librarian school ;) turn it on with a performance penalty if they need it.

We only want meta name="description" in Plone by default, as this is the only one used by search engines - even keywords are of questionable usefulness.

listMetaTags - before: 0.45s - after: 0.09s

Yay for Alec! His implementation of my suggestion slices the time to 1/5 of the previous usage and introduces the DC metadata switch. Next target is breadcrumbs. Tesdal has added breadcrumb support to ExtendedPathIndex, so it should be possible to make it significantly cheaper than it is now.

breadcrumbs - before: 0.20s - after: 0.06s

Nice. Using Helge Tesdal's new ExtendedPathIndex (that also powers the new nav tree and the site map), Alec Mitchell implemented a version of the breadcrumbs that cuts rendering time to a third of the original.

listFilteredActionsFor - before: 0.78s - after: 0.56s

A simple, but effective speedup was to remove an unnecessary loop in listFilteredActionsFor, and it's by almost a third, and is more effective the more actions you have, so it should really make a difference if you are logged in too. This is the final change we're doing on the branch, merge time! May 25th, 2005, limi:

We had an interesting use case at a client site where they had 113(!) content types. This lead us to a quite interesting discovery that listTypeInfo in CMF is extremely expensive when you get a lot of types.

Alec stepped up (as usual ;) and helped out. There is now a new method override in Plone's type tool that gets rid of the madness and uses a much more light-weight method to do the exact same job.

The result? listTypeInfo - before: 1.20s - after: 0.40s

Alexander Limi
Alec Mitchell