This reference manual is intended to give an overview of the processes and conventions of the development of Plone, both as a point of reference for existing developers, and a source of explanation for new developers.

It is not intended as a general guide for how to develop applications on top of Plone, but some of the "best practice" documentation here may be useful to third party developers as well.

This document is a work in progress, and will probably continue to be so for a good while. We will endeavour to publish pages and sections when they are complete, but don't be surprised if you find that there are major pieces of the puzzle missing at present.

Like all open source products, Plone only moves forward on the contributions of volunteers. If you would like to contribute, we will be incredibly happy!

However, there are many people and companies who rely on Plone day-to-day, so we have to introduce some degree of quality control over the code base. Plone's source code is hosted in an open Subversion respository on http://svn.plone.org, but only members of the developer team have commit-rights. Before you can get commit rights, three things need to happen:

1. You must familiarise yourself a little with the community. Get on the chatroom, start asking and (especially) answering questions on the mailing lists, and get to know the more active developers a bit. If you arrive in the #plone chatroom and ask how you can help, chances are you will be met with gratitude and openness, so don't be afraid to introduce yourself. If no-one answers, it's probably just a bad time of the day: try the developers' list instead.
2. You must prove that your code does not suck. There are two types of people in the world - those who can write code, and those who cannot. Actually, we take that back - there are also UI people. And accessibility experts. And translators. And documentation writers. And testers. However you wish to contribute, though, we'd like to see the calibre of what you produce. See below for ways of contributing without commit-access to the source code repository.
3. You must sign the contributor agreement. This offers some intellectual copyright protection, and ensures that the Plone Foundation is able to exercise some control over the codebase to ensure it is not appropriated for someone's unethical purposes. Think IBM vs. SCO.

Now, there are a number of other ways you can help out, which do not require contributor agreements or anything else than the right attitude.

Testing

We always need more testers. If you believe you have found a bug in Plone, particularly if you are using a less common operating system, have a very large site, or operate in non-English locales, please report it in the bug collector. Please search the tracker thoroughly before submitting a bug, so that we avoid duplicates.

Before submitting a bug, please log in with your plone.org username. If you don't have an account, just click the "join" link in the blue bar at the top on plone.org. You can submit bugs without being logged in, but when you are logged in, you will be given email notification when your bug is responded to. This is important if we need clarification, for example - you can see a request for clarification, and then add a follow-up with more details.

Please think carefully before submitting a bug. If the bug is really in a third party product, do not use the Plone tracker, as we won't be able to help you - contact the authors directly, instead, or use their tracker in the products area. If it is a Plone bug, search for similar issues and add a follow-up to an existing bug if necessary - we will be notified of this just as clearly as with new bugs.

If you have to submit a new bug, try to write as clearly and concisely as you can to ensure that we understand your problem. Include relevant product versions, and always enter your Plone and Zope versions. Be honest about the severity - submitting a bug as "critical" if it is not is unlikely to win you any sympathy.

Patches and verification

Very often, issues end up in the collector for a long time because we don't know whether there was actually a problem, or if the submitter was experiencing a local configuration issue. If a bug looks sketchy, it's very helpful if you can try to replicate the problem on your own setup and add a follow-up about your experiences. If you have more insight to offer than original submitter, this is of course incredibly useful!

If you think you know how to fix a bug, this is your chance to shine. Once logged in, you can submit patches to the tracker. Even if you can't provide a patch, but can do some research and come up with a way in which the issue may be resolved easily, this will help the developers immensely. Adding patches to the collector is one of the best ways of showing off your skills and proving that you should be given commit-access to the Plone repository.

Documentation

Any member of plone.org (that includes you, if you want to be) can submit documentation to the Plone documentation area. If you have discovered something that you feel was poorly documented, you will gain much praise by writing a short How-to or a Tutorial on the subject. Once you are finished, submit your document for review, and we will publish it if it is not giving bad advice or too difficult to understand. You will probably find that writing documentation also helps you structure your own understanding of Plone, and forces you to consider issues you may have otherwise glossed over.

If you need help with writing documentation, or you're not sure where to start, send a mail to the plone-docs list/newsgroup

Helping others

The best way to gain respect in the community is by helping others. If you spend some time in the #plone chatroom or on the Plone users' mailing list answering people's questions when you know the answer, people will take notice. Helping others will help your own understanding, not to mention the warm fuzzy feeling you'll get afterwards.

General

Plone uses a time-based release process, which helps to produce releases of predictable size and complexity. In order to make this work there are some requirements on how new features get accepted and implemented.

Special roles

There are some special roles in the Plone community of which the release manager and the framework team are the two important ones for the release process. For every major release series a release manager is elected by the Plone foundation board who has the authority to make certain final decisions for a release. The framework team is a group of people who assist the release manager in the process of reviewing any suggested features.

Release phases

The normal development cycle of a Plone release consists of various phases. See the Plone roadmap for concrete dates for the next upcoming releases.

Discussion and Planning

People suggest new features and discuss them. We use a lightweight formal process which helps to foster and document this. See the below paragraph on enhancing Plone for the details.

Proposal freeze date

For each release there is a certain date by which all new feature proposals must have been submitted for review. After this date no features will be accepted anymore for this particular release.

Alpha releases

We will release various snapshots of the current release bundle that we package up to give out for testing. Migration to/from specific alphas is not supported.

Feature freeze date / Beta releases

By this date all features must have been completed and all code has to be integrated. Only non-invasive changes, user interface work and bug fixes are done now. Migrations supported to/from specific betas.

Release Candidate

Ideally, the last beta that was bug free. No changes to the code. Should not require any migration steps apart from the ones required in the betas. If any problems are found and fixed, a new release candidate is issued.

Final release / Expected release date

Normally the last release candidate that was issued without any show-stopper bugs.

Bug fix releases

No software is perfect. Once a sufficient large or critical number of bugs have been found for a certain release, the release manager releases a new bug fix release a.k.a. third-dot release (for example 2.1.2).

Enhancing Plone

Work out the idea

If you have any idea on how to make Plone even better as it is today, you should first talk with other people about your ideas. Both the IRC channel and the developers mailing list are good starting points for this. Talking to people about it will give you some pointers about the feasibility of the feature, and give you a good starting point for further work.

If your change or improvement is a big one, then you'll be asked to write a PLIP - Plone Improvement Proposal. In the PLIP you'll explain exactly what it is you want to do, how to do it and how it will integrate. To add a PLIP, you need to be on the development team, so you can add it as an Improvement Proposal in the Plone Roadmap area.

When writing a PLIP, be as specific and to-the-point as you can. Remember your audience - to get support for your proposal, people will have to be able to read it! A good PLIP is sufficiently clear for a knowlegable Plone user to be able to understand the proposed changes, and sufficiently detailed for the release manager and other developers to understand the full impact the proposal would have on the codebase. You don't have to list every line of code that needs to be changed, but you should also give an indication that you have some idea that how the change can be feasibly implemented.

If your change is minor then a ticket in the tracker will be sufficient, added as an enhancement. The key point here is that each change needs documentation so other users can see what it is. This can be in the form of an issue tracker entry, or a PLIP in the case of a bigger change. A bug or minor change does normally not need to go through a review process - a PLIP does.

When you've done your PLIP and you're happy with it, announce it on the framework team mailing list . People will come and read the proposal and give feedback (hopefully positive).

At this point there is one key decision made by the developers:

• your PLIP will be adopted into Plone
• your PLIP will not be adopted

The latter isn't bad, it's just that this is not appropriate to be in the Plone core and have all the Plone team support. Each PLIP that gets accepted places a burden on the team, so people are selective in this process. There are many great products Plone uses that aren't directly part of Plone.

Start working on it

You can start the development at any time - but if you are going to modify Plone itself, you might want to wait to see if your idea is approved first to save yourself some work if it isn't. Your code has to be in an open accessible code versioning repository and you have to make a release bundle which allows the framework team to easily review your work. Please signify the release bundle on your PLIP. For a technical explanation see the "Using bundles for code review" paragraph of the "Version Control" page of this reference manual.

Once you've done developing it let the framework team know. At this point we'll do some things:

• review it and check you've done what you said
• see which release it will be accepted into

These are the criterias by which the framework team will review your review bundle:

1. Does the PLIP itself need work?
   • i.e. is this idea well baked and expressed clearly
2. Does the work proposed belong in Plone now, in the future?
   • Does it do the needful for Plone?
   • Is it more appropriate as a qualified add-on?
3. What condition is the review bundle in:
   • tests
   • sane implementation
   • clearly coded
   • uses current idioms of development
   • documentation / doc-tests

Note that if you don't finish your implementation in time for the feature freeze, it will not be included in the next Plone release. You can always submit your PLIP for the next release.

Finishing your work

Once you have finished your work and it was accepted by the framework team and the release manager, you will be asked to merge your work into the main development line. Merging the PLIP in is not the hardest part, but you must think about it when you develop. You'll have to interact with a large number of people to get it all set up. The merge may cause problems with other PLIP's coming in. During the merge phase you must be prepared to help out with all the features and bugs that arise.

If all went as planned the next Plone release will carry on with your PLIP in it. You'll be expected to help out with that feature after it's been released within reason, and we look forward to the next feature.

Overview

Part of our development process are some special events that are organized from time to time. The main idea behind these is to get people together to work on Plone at the same time, both as it is more efficient as you can get immediate feedback and as it is certainly more fun to work with others.

What is a Bug Day?

From time to time, especially in the weeks leading up to a release, the Plone Community arranges so-called "Bug Days". These days focus on identifying and fixing bugs and other issues with the Plone core, and is an excellent chance to get to know both Plone and its developers better. We therefor get together on IRC and collectively fix a selected set of bugs.

The Bug-Wrangler is the person who determines what bugs needs to be fixed and prioritizes as well as allocates bugs to each individual developer. Bug Day's happen in many other open source projects. Anyone can participate in a bug day; general users who are comfortable with using Subversion can test that a developer has fixed a bug or that the user can not provoke the bug in a different manner. And no matter what your current skill level is - from totally new to Plone to experienced Plone developer - you can make a difference!

Bug days are a critical development and social event that brings the developers closer through producively exorcising software of evil bugs. Bug days are usually announced on the developer mailing list and on the frontpage of plone.org.

What is a Sprint?

Tres Seaver originally came up with this idea. While a bug day lasts only one day or a weekend and people usually meet online, a Sprint typically lasts for several days to a full week and people meet in person. The idea of a Sprint is for a group of people to enhance, create, or fix one or more pieces of infrastructure. Usually they are focused on a specific set of topics instead of the entire product line.

Sprints are either funded by organizations or individuals who need specific features or are interested in working on some. Sprints happen yearly in the Plone community from sunny San Francisco to the top of mountains in the Alps where the only electricity is produced by a generator and a satellite uplink is used for internet connectivity. Sprints are extremly productive cultural events in the world of Plone. You can have a look at the list of past and upcoming Sprints.

What is a Symposium/Conference?

This is not a special kind of event for Plone or the open source world, but exactly what happens in other businesses as well. Usually there is one official Plone conference per year and at least one regional Symosium. Most of the time there is a business and a development orientated track each consisting of a series of talks and tutorials. See the events section for past and upcoming events.

Zope 3 is the new, cooler, faster, sexier version of Zope. Unfortunately, there is no direct migration path from Zope 2 to Zope 3. However, we are beginning to use Zope 3 technologies in Plone, alongside the Zope 2 core.

Zope 3 is a fairly different way of programming. It is based on the principle of components, that expose their functionality through well-defined interfaces. Whereas in Zope 2, you would need to mix classes into the inheritance hierarchy to gain certain functionality, for example by mixing in CopySupport to get cut/copy/paste support, Zope 3 uses the principle of adapters.

An adapter is simply a piece of glue code that can adapt an object to a given interface, so that the original object itself doesn't need to know anything about that interface or what it does. This makes it easier to re-use other Python objects that are not Zope aware, and makes components smaller, more precisely defined and thus easier to re-use.

Other types of Zope 3 components include utilities - stateless, global objects that can be looked up through a generic registry, and views, a special type of adapter that is typically used to contain the logic of page templates in a way that is more performant and easier to test.

Porting Plone wholesale to Zope 3 would essentially involve a re-write, and would break every installation, third party product and customisation out there. It is simply not an option. However, because Zope 3 is built out of small, re-usable components, we can use certain Zope 3 technologies today. Starting with Zope 2.8, the whole of Zope 3 ships as an add-on library and is available to use for Zope 2 applications like Plone. Over time, more and more parts of Zope 2 will get re-written to use Zope 3 technologies. This is accomplished via a product called Five (what's Zope 2 + 3?) that performs some magic to make the fundamental Zope 3 building blocks usable in Zope 2.

Zope 3 integration begun with the Plone 2.5 release cycle, and will continue gradually over time. That means that developers will have to start learning Zope 3 concepts. At the moment, there is still some overlap between what can be done with "old" techniques and what can be done in Zope 3. There are also dependencies on what happens in CMF and Zope 2 in general before certain things become available.

However, the basic mantra starting with Plone 2.5 is:

• New components should use Zope 3 technologies whenever possible and appropriate
• Any re-factoring of existing code should use Zope 3 technologies where possible and appropriate

This means that page templates with complex logic should use views. Any new component should make proper use of interfaces and adapters, and existing components should get interfaces and use adapters if at all possible.

It is important to realise that this is a gradual process, and there are many pragmatic decisions to be made. In general, you should always consult the developer list, the release manager or other members of the development team on non-trivial design decisions.

You can learn more about Zope 3 from one of the published books, from Philipp von Weitershausen's WorldCookery web site, from this tutorial or from the rest of documentation area.

This guide may not (and probably will not) give you all the details you need in order to resolve any confusion about Plone's internals. Luckily, there are several other places you can go.

• The rest of the documentation should be your first point of call.
• The Plone Developer list is a prime source of help. Please only post to this list with topics regarding the development of Plone itself, not third-party product development or customisation.
• Similarly, the #plone IRC chatroom is where most real-time discussion about Plone takes place. People with operator status are typically Plone developers.
• If you are contributing to Plone, you should make a point to both read the developer list regularly, and be online in #plone whenever you can, especially while you are working on Plone.

Going forward, there is consensus that we should use Zope 3 style programming practices whenever possible. One part of that is that things that don't need to be Zope 2 products shouldn't live in in $INSTANCE_HOME/Products/ - instead, they should be simple Python packages, living in $INSTANCE_HOME/lib/python or somewhere else on the PYTHONPATH that Zope is given when it starts up.

In doing so, code that is part of Plone core should adhere to the following conventions

Generic components without specific Plone dependencies live directly under the top-level plone namespace

A good example is the the Zope3-style plone.i18n.

It is desirable to factor out generic interfaces and code into such packages whenever possible, to foster re-use.

Packages should have as few dependencies as possible. Thus, if plain-Python will do, that is better than plain-Zope 3, which is better than code that depends on specific facets of Zope 2.

Extensions of such general components (or separate components) that provide a tighter integration with Plone-the-application should live in the secondary namespace plone.app

See for example plone.app.i18n.

Components that need to be Zope 2 products will most likely continue to live in $INSTANCE_HOME/Products still, at least for now. If possible (which it may not be), however, you should try to not depend on components being Zope 2 products.

Packages in the svn repository should follow modern Python guidelines and provide the necessary information to be packagable as eggs

The easiest way of achieving this is to use the ZopeSkel paste deploy script, as follows.

1. Download ez_setup.py from PEAK
2. Run ez_setup.py in Python:

      $ python ez_setup.py
   

This will install the easy_install program, normally to the Place where the Python binary is found. Look in the log messages of the installation script to see where they land. For more information, see the documentation

1. Install ZopeSkel:

      $ easy_install ZopeSkel==dev
   

This will install Paste Deploy and the paster script in the same location as easy_install (again, watch the terminal output) and some Zope skeletons to use with this tool. To see them, run:

   $ paster create --list-templates

1. To create a basic Plone package, run:

      $ paster create -t plone
   

It will ask you a number of questions and then generate the basic package layout. When asked for a project name, use the dotted name of the package, e.g. plone.i18n.

To create a plone.app package instead, run:

   $ paster create -t plone_app

When ready, import the package to the Plone svn repository.

To learn more about setuptools, see its documentation. In particular, see the documentation on development-mode

Python, like any programming language, can be written in a number of styles. We're the first to admit that Zope and Plone are not the finest examples of stylistic integrity, but that doesn't stop us from trying!

First of all, you need to read PEP 8 - the python style guide. You might also have a look at the Zope codying style guidelines but these are not a mandatory reading.

Naming conventions

• Above all else, be consistent with any code your are modifying!

• Write method names and variable names in mixedCase
• Write class names in CapitalisedCase

File conventions

• If you need to create a new file, chances are you're doing something fairly fundamental. In that case, you should probably discuss it with the release manager or someone else more experienced first.

• In Zope 2, file names used to be MixedCase. In Zope 3, and thus in Plone going forward, we prefer all-lowercase filenames. This has the advantage that you can instantly see if you refer to a module / file or a class:

      from zope.pagetemplate.pagetemplate import PageTemplate
  

compare that to:

  from Products.PageTemplates.PageTemplate import PageTemplate

• Filenames should be short and descriptive. Think about how an import would read:

       from Products.CMFPlone.utils import safe_hasattr
  

compare that to:

     from Products.CMFPlone.PloneUtilities import safe_hasattr

the former is obviously much easier to read, less redundant and generally more aesthetically pleasing.

Some concrete rules

Certain things should never be seen in code. Some of these are covered in more detail in the patterns section, but briefly:

• Do not use tabs in Python code! Use spaces as indenting, 4 spaces for each level. Maybe you didn't hear that - do not use tabs and keep indentation to 4, that's four, spaces.
• Indent properly, even in HTML. (limi's note: especially in HTML ;)
• Never use a bare except. Anything like except: pass will get you in real trouble
• Avoid tal:on-error, since this swallows exceptions
• Don't use hasattr() - this swallows exceptions, use CMFPlone.utils.base_hasattr or CMFPlone.utils.safe_hasattr instead!

  The problem with swallowed exceptions is not just poor error reporting. This can also mask ConflictErrors, which indicate that something has gone wront at the ZODB level!

• Never, ever put any HTML in Python code and return it as a string
• Do not raise string exceptions - raise proper exception objects instead
• Do not acquire tools e.g. using context.plone_utils. Instead, use:

     from Products.CMFCore.utils import getToolByName
     plone_utils = getToolByName(context, 'plone_utils')
  

(it's OK to acquire tools in unit tests)

• Do not put too much logic in ZPT (use Views instead!)
• Remember to add i18n tags in ZPTs and Python code

It is important that you familiarise yourself with subversion before committing to the Plone source repository, to reduce the risk of accidental screw-ups. The Subversion book is a comprehensive, but surprisingly accessible resource.

Setting up your environment

There is a list of non-command line clients which you might find easier to use than the standard command line client. If you are a developer from the Windows platform, we highly recommend the excellent TortoiseSVN product instead of the "standard" command line client.

Note that we are following the same policy as svn.zope.org regarding setup for line endings, please read SVN Configuration for line endings

Getting the source code

Development versions of Plone are managed in Subversion bundles. A bundle is simply a set of top-level modules that must be checked out. In general, you can check out a bundle into the Products/ directory of your Zope instance, run any external scripts needed to get resources not available in svn (such as getExternalEditor.sh), and start Zope. The bundles are found in https://svn.plone.org/svn/plone/bundles/.

At the time of writing, the 2.5 bundle contains the current development version of Plone. You can check this out by doing:

  svn co https://svn.plone.org/svn/plone/bundles/2.5
  ./getExternalEditor.sh

Commits

You should make one change per commit. If you are fixing three bugs, make three commits. That way, it is easier to see what was done when, and easier to roll back any changes if necessary. Also, large, monolithic commits are very difficult to read. Yes - people do read the checkin list to see what happens to their favourite CMS! If you want to make large changes cleaning up whitespace or renaming variables, it is especially important to do so in a separate commit.

It is a cardinal sin to commit any change without properly testing it with unit tests, as well as in the browser (see next page). Please review your code for silly mistakes, lingering pdb statements, debugging print statements or any other junk that shouldn't be there.

ALWAYS run all the Plone unit tests before commiting (see next page). There is NO exception to this rule. If you submit code that breaks a test, your change will almost certainly be reverted with a terse note to plone-dev. If you are at all in doubt, please also run the ATContentTypes and Archetypes unit tests, at the very least.

Before commiting any bugfix or feature enhancement, you must also add a note in Plone's HISTORY.txt. Follow the convention in that file, and note your name in brackets. This file is used to clarify what new features have been added when a new release is made, and should be an accurate account of the changes you have made.

Checkin messages

It is vital that you write proper checkin messages. A checkin message should inform the reader what was done, and why. If a bug was closed, the bug number should be included, e.g.:

  Added the plone_deprecated skin layer back into migration. Fixes #1234.

The "Fixes #1234" will actually cause bug #1234 to be marked as closed in the bug tracker automatically.

Using branches for non-trivial changes

If you are doing anything non-trivial, changes should go on a branch! See also the page on the PLIP process. A svn branch is simply a copy of the existing development version that you can work on undisturbed. When your work is finished, tested and approved, the branch can be merged back.

It is important that you name your branch so that people can easily see who is working on it, and what it is for. For example, to branch CMFPlone trunk to work on refactoring foobar, I could write:

  svn up   # update to the latest revision beforehand
  svn info # find out which revision you are branching from
  svn cp https://svn.plone.org/svn/plone/CMFPlone/trunk \
         https://svn.plone.org/svn/plone/CMFPlone/branches/optilude-2.5-foobar-refactoring

The checkin message for this operation must be informative, and must include the revision number of the branch you started from:

  Branched Plone trunk r1234 to begin work on foobar refactoring.

Switching to an existing branch, for example if you want to test someone else's code, is also very simple:

 cd CMFPlone./
 svn switch https://svn.plone.org/svn/plone/CMFPlone/branches/optilude-2.5-foobar-refactoring

Never create a branch directly from your working directory. Copy it from the repostitory instead: "svn cp original_url branch_url". If you have changes in your working copy that sould be included in the new branch switch to it and commit (after checking for conflicts).

When a branch is ready to be merged, notify the release manager. If you are merging yourself, it is important that you include which revisions you merged in your checkin message:

  Merged optilude-2.5-foobar-refactoring branch r1234:1245 into trunk.

Note the format of the range of revisions (r1234:1245). Please follow this format exactly, as it will make the job of the release manager much easier should there be a problem.

Please do not rename or move branches unless absolutely necessary - Subversion is not very good at handling this. Again, include the revision numbers of what you renamed or moved in the checkin message if you must do this. The reason for this is that the only way to find out where to branch from is to use svn log --stop-on-copy, which will reveal which revision the branch was originally copied from. When you rename or move a branch, this command will stop on the revision where the rename or move happened.

Using bundles for code review

When you are sponsoring a PLIP or wish to demonstrate some new feature, you should make life easier for the framework team, the release manager and the general public by using review bundles. A review bundle should work as the main development bundles do: Another developer should be able to check it out into the Products/ folder of a fresh Zope instance and see exactly what changes have been implemented, immediately.

To create a bundle, first create the necessary branches on the modules you are changing, and set up a sandbox that contains only the relevant checkouts. If the code is a moving target, packages are bundled at the revision that the developer wants reviewed.

All bundles must include:

• A README.txt in the root including any caveats or explanation necessary.
• A COMMENTS.txt for developers to record comments
• TODO.txt for recording things that need to be done
• The bundle must compile and run upon checkout!

A bundle is managed using a file in the root of the bundle, conventionally called EXTERNALS.txt. You should use the EXTERNALS.txt of your development bundle checkout as a starting point. Then, to create a new bundle, you do:

  svn mkdir https://svn.plone.org/svn/plone/review/plip1234-my-crazy-feature-bundle
  svn co https://svn.plone.org/svn/plone/review/plip1234-my-crazy-feature-bundle my-bundle
  cd my-bundle
  svn propget svn:externals > EXTERNALS.txt
  # edit the file, changing any svn paths to point to the relevant branches
  svn propset svn:externals -F EXTERNALS.txt .
  svn commit

You can now test this by doing:

  mkdir bundle-test
  cd bundle-test
  svn co https://svn.plone.org/svn/plone/review/plip1234-my-crazy-feature-bundle

This should include all the necessary products to start a Zope site demonstrating your new feature.

Plone is a large and complex system. Were you to blindly change some feature, you would probably break another one in ways you could never have imagined. Therefore, good unit testing is vital to the survival of Plone. Every feature needs a set of unit tests for base cases and edge cases.

Don't look at unit testing as some necessary evil: Unit testing actually makes testing fun! Ideally, you should write tests before you implement your code or make changes. Your mission is then to make the tests pass. This is generally much easier and more enjoyable than fiddling with things in UI. Unit tests also save you time in the long run, because you are less likely to let obscure bugs go unnoticed, or break code which you thought worked with seemingly "harmless" changes.

Please refer to the unit testing tutorial for the basics of unit testing. This is required reading!

Plone unit tests live in CMFPlone/tests. You should most definitely read through some of the existing test files and familiarise yourself with how unit tests are written.

Fixing bugs

When you fix a bug in Plone, you must write at least one unit test to prove that it's there! That test should fail the first time you run it. Then fix the bug and run the test again. This time, the test must pass. Be sure to test Plone in the browser too, and run all the unit tests before commiting your bug fix.

Adding features

When you add a new feature, it is especially important that you write unit tests up front. If you need to write migrations (more on that later) you will need to test both the migration methods themselves, and the state the migration would bring the portal to.

If you are touching code that appears to have poor test coverage, it is your duty to expand the test coverage. Just because the guy before you was negligent doesn't mean you have a license to be. Plone's test coverage is improving all the time, but could be better. If anything, writing tests for code you are interacting with will reduce the chance that you'll get blamed for someone else's mistakes.

When to change a test

Sometimes you may be tempted to change a test. Be prepared to defend yourself! If you change a test, you are invalidating the unit test runs of every single person who's run the test suite with that test before you. Occasionally, tests are wrong - sometimes they test the wrong thing, sometimes they never can fail (which means the person who wrote the test to begin with was bad and wrote the test after the code - remember, all tests start with a failure!). If you do feel you should change a test, make sure your checkin message is clear on why you changed it, and discuss it with the release manager if you are at all in doubt.

Here's a little fact we should admit up front: Plone is slow. Not as slow as it used to be, not as slow that it's useless, but neither Zope nor Plone have ever been sold on speed. In general, Plone is a content production system, not a content delivery system, and has been written for features and ease-of-use — not for blazing speed. (We use caching to improve speed, or separate delivery platforms altogether.)

This has less to do with the design of the software (or the capabilities of the people involved) than with the trade-offs that have been made for usability and flexibility. However, it is important to think about performance when writing code for Plone core.

Profiling

To find performance bottlenecks, you need to use a profiler. The PTProfiler is very useful for finding out which expressions in the page templates are using up the most CPU time. An alternative profiler is the Call profiler. See also Dieter Mauer's Zope Profiler.

Waking up objects

If there is one axiomatic law of performance in Zope and Plone it is this: Do not wake up objects unless you need to. "Waking up" objects refers to pulling them out of the ZODB - traversing to them, loading them into memory. This is an expensive operation. The old navigation tree in Plone 2.0 used to wake up each and every object it displayed. Sometimes it could wake up each object in a BTreeFolder (aka "Large Folder") with 1,000 children, even if it only rendered a single one of them. This was bad.

In general, we prefer to use catalog queries to find objects. This has been possible since the introduction of ExtendedPathIndex in Plone 2.1. The getFolderContents script uses a catalog query. The navigation tree uses a catalog query. The portlets all use catalog queries.

Essentially, if you find yourself using objectValues() or contentValues() you need to think long and hard about whether you should be using a catalog query instead. The following pattern is the most common use of a path search:

  portal_catalog = getToolByName(self, 'portal_catalog')
  results = portal_catalog.searchResults(path = {'query' : '/'.join(self.getPhysicalPath()),
                                                 'depth' : 1})
  for brain in results:
    ...

Catalog queries themselves are not the fastest to set up, so if you are expecting only one or two items, traversal may be better. For the breadcrumbs in Plone 2.5, for example, we use traversal up the acquisition parent chain (by using aq_parent, which returns full objects) instead of a catalog query, because the parent objects are quite likely to be in the ZODB cache already, and because setting up a catalog query for what is most likely going to be 1-5 results is sub-optimal.

Actions

CMF actions are used to drive things like the tabs in the editable border, the site actions (the sitemap, etc.), the personal bar and many other parts of the Plone UI. Unfortunately, actions are also slow, because each may be associated with a TALES expression that is used to determine whether the action should be displayed or not. Each expression requires an expression context to be set up, the expression to be executed, and the result to be evaluated. This is not particularly efficient.

Actions are still used for many of the dynamic parts of the Plone UI, but if you are using them, you should be aware of the performance implications. At the very least, avoid complex TALES expressions if possible.

Avoiding 404s

A 404, or a NotFoundException in Zope, is an expensive operation. 404s can occur unnoticed when a page template references and image or stylesheet that does not exist. Be careful about referencing items that may or may not exist. You can code defensively by doing something like:

  <tal:block condition="exists:some-name">
    ...
  </tal:block>

If you are in doubt, go to the error_log and remove NotFoundException from the list of ignored execptions. Re-load your page and watch out for new entries in the error log.

Making pages cacheable

One of the ways in which real-world installations make Plone faster is by using cacheing, for example via CacheFu. It is important that we make pages and elements of pages as easy to cache as possible. Most of the time, this simply boils down to avoiding unnecessarily dynamic elements.

Take the sitemap, for example. Constructing this is an expensive operation, since the sitemap view needs to execute query for all objects matching the current navigation filter settings. It then needs to render these using a recursive page template. Notice that the action pointing to the sitemap is ${portal_url}/sitemap.

At first sight, this is sub-optimal, because the sitemap can't know the context the user was in before clicking the link, and thus can't put a "you are here" marker in the sitemap. But if the path were ${object_url}/sitemap, then the sitemap on /foo would be different from the sitemap at /bar. Any cache for /foo/sitemap would be completely separate from /bar/sitemap even though 98% of the information on those two pages would be the same. Worse, when Google comes to index the site it will sprider a /sitemap for each and every page, which may bring your site to a grinding halt from executing too many queries.

This principle applies to other things too, such as the author page (which is linked to from all the by-lines), and most commonly image src attributes for common icons.

Cache with v attributes

Sometimes you can't avoid executing an expensive operation more than once during a request. In this case, you may want to cache the result. Zope provides a mechanism for simple caching called "volatile" attributes, or _v_ attributes.

Any non-transient object (that is, any object that lives in the ZODB - note that view classes are not included in this group) can store a value in a v attribute. A v attribute may or may not exist at any given point in time! It will only be valid whilst the object is cached in memory, and there is no guarantee it will stay in memory even during a single request cycle. However, it most likely will be, so it pays off to use it.

You need to code defensively whilst using v attributes:

  from Products.CMFPlone.utils import base_hasattr
  ...

  if not base_hasattr(self, '_v_cachedValue'):
    self._v_cachedValue = self.calculateExpensiveValue()
  return self._v_cachedValue

Download size

Nobody likes having an 800k download to get a few lines of text. Plone's UI is by necessity heavy, and there are many stylesheets, images and HTML snippets included when you download even the lightest of Plone pages. The least you can do is to not make this problem any worse. Think carefully about what you are including, especially if it is going to be included on every page. Keep whitespace and superfluous comments to a minimum, and reduce the size of images as much as you can.

Plone differentiates itself on usability. The intuitiveness of the user interface is what attracts people to Plone the most.

There are many subtle principles of usability that won't be discussed here, but there is one golden rule: be consistent. Do not introduce new UI metaphors without a damned good reason. Re-use existing HTML and CSS patterns. Make sure things look consistent.

Also, you must think about the purpose of a UI element and avoid overloading it. The classic example is that people use portlets for static UI elements. Remember that portlets can be removed and re-ordered by users, so depending on them for functionality is not a very good idea.

Similarly, the personal bar (the blue one with my folder, preferences etc.) is used for "personal" items, related to the user, not for site-wide actions. The site actions are used for ... site-wide actions like the sitemap and the site setup, and so on.

Usability often comes down to taste and finesse. If in doubt, ask on the developer list, do your best, and don't be offended if limi changes your carefully crafted HTML.

As explained many times before, every feature, bug fix and re-factoring needs to be backed up with appropriate tests. Sometimes, it is necessary to write code in such a way that it can be tested. Normally, however, making code testable also forces you to think about how that code is modularised and what interfaces other code should be using to talk to your code. If your methods are predictable in a test, they will be predictable to calling code as well. Display logic is most easily tested by moving it from page templates to views.

And remember: the easiest way of making sure that code is testable is to write the tests first!

Plone prides itself on being multi-lingual, multi-national and multi-cultural. The whole user interface is translatable, and supports right-to-left languages such as Hebrew or Farsi.

The most common interaction you will have with internationalisation (i18n for short) is through page templates. Please read the developers guide to internationalisation, but in general, be mindful that any user-visible string will need a translation key, and that this key will need to be unique within its domain.

Internationalisation of user-visible strings may also be an issue if you are returning a string from a script or method. For example, you may set a status message in a form controller script. In that case, you will need to invoke the translation machinery directly:

  from Products.CMFPlone import PloneMessageFactory as _
  def myMethod(self):
      value = self.getValue()
      return _(u'My name is ${fullname}', mapping=${u'fullname' : value})

Note the use of unicode strings (the u prefix) - all strings that form part of the UI should use unicode. Also note that this assumes the message will be processed by a page template (e.g. via a tal:replace or tal:content statement).

If you need to translate something that is not going to be processed by a page template, you will need to invoke the translation service directly:

  from Products.CMFCore.utils import getToolByName
  ...
  translation_service = getToolByName(self, 'translation_service')
  value = u'John Doo'
  return translation_service.utranslate('plone',
                                         u'My name is ${fullname}',
                                         mapping={u'fullname' : value})

In versions of Plone prior to 2.5, there used to be a translate python script. Please do not use this anymore.

Page templates promise to separate logic from presentation. Unfortunately, on their own they only deliver half of that promise. How many times have you made a page template that has a very complex python: expression and thought "I should move this out of the page template"? The options until Zope 3 have been:

• Make ever more complex python: expressions in the page template and look the other way when people ask who the hell put that in there. Unfortunately, such expressions are difficult to maintain and debug, nearly impossible to test, and expensive to execute.
• Make a new Script (Python) in a skins folder such as plone_scripts. Unfortunately, skin scripts are hard to debug, hard to test, quite slow to execute and litter a global namespace.
• Put the code in a content type class that is the context of the script. This is quite common in Archetypes, for example. The problem is that this bloats the interface to that class with quite display-specific code, making the type and (particularly) the page template more difficult to re-use.
• Put the code in the plone_utils tool (PloneTool.py). This has made PloneTool an incredibly bloated set of losely related functions that may or may not be useful to more than one or two templates.

In Zope 3, this is solved with a view. A view is basically just a multi-adapter that is looked up on the context object (usually a content object) and the request (so that the view can be different for HTTP requests that for FTP requests, say). In pure Zope 3, views are used to represent objects to XML-RPC or FTP and typically comprise a Python class implementing a particular interface and a page template. Views are registered in ZCML under a given name and referred to by @@name (the @@ looks like a pair of eyes peering out, if it helps you remember the symbol).

In Plone 2.5, we used to use a custom base class in Products.CMFPlone.utils.BrowserView and a related utility method in Products.CMFPlone.utils called context() to deal with acquisition problems in views. However, this is now frowned-upon, because it introduced a hard dependency on CMFPlone and slightly deviates from the agreed IBrowserView interface.

Instead, views tend to be used like this in Plone 3:

 from Acquisition import aq_inner
 from Products.Five.browser import BrowserView

 class MyView(BrowserView):
     """My new view
     """

     def some_function(self):
         context = aq_inner(self.context)
         ...

Notice the use aq_inner(). This ensures that the acqusition chain of the context does not include the view as a parent. There are numerous examples of browser views in the plone.app.* packages.

View tips and tricks

• View can (and should) be unit tested! They are just simple python classes, nothing else.
• Views are transient - they are never stored in the ZODB, and never acquisition-wrapped.
• Views are created fresh for each request cycle. Therefore, they cannot cache values directly between requests, nor can they rely on any form of persistence.
• Sometimes it's necessary to call a view method more than once. If that method does some expensive calculation, you should cache the results of the operation. To make this easier, a cache decorator exists in plone.memoize. Using it is pretty simple:

    from plone.memoize.instance import memoize
  
    class MyView(BrowserView):
      ...
  
      @memoize
      def someExpensiveOperation(self):
          x = 10
          for x in range(10000):
              x = x ** x
          return x
  

Consider a Python library that is not aware of Zope. To use this in Zope 2, one would have to mix in a large number of classes just to get the basic Zope object support, making this a very difficult task. In Zope 3, you use adapters to provide the glue. You start by defining an interface that explains how you want other Zope components to talk to the library. Then you provide an adapter that can adapt an object from the non-Zope library to that interface. The Component Architecture provides the registry that makes it easy to look up the appropriate adapter for any given context to any given interface (providing one exists).

As it turns out, adapters are also very useful in creating loosely-coupled components. An adapter is registered from a given interface, to a given interface. Because interfaces can subclass each other, you can have a general adapter for a very general interface, and a more specific adapter for a more specific interface. Similarly, you can provide a very different adapter for one interface than another. Thus, you could make one adapter that makes any code expecting the IMyFunctionality interface able to talk to any object implementing IMyContentType, and a very different adapter that works on ISomeOtherContentType.

By factoring functionality out into separate interfaces implemented with separate adapters, you create slimmer, more re-usable components that are easier to understand and test. Adapters do not necessarily have to "extend" the functionality of an object in a conventional sense. Consider the following example:

Navigation tree needs to consider a large number of options, set in portal_properties. The code that builds the navigation tree performs one complex, but well-defined task: taking a catalog query and turning it into a tree of nodes that can be displayed in the navtree. The old navigation tree code used to mix these tasks together, which meant that the code was impossible to re-use. If you wanted a navtree-like structure (such as the table of contents in this reference manual), you had to write it all again!

For Plone 2.1.3, this was rewritten to use a more generic method that externalised the navtree preferences into a more generic "strategy" object. This let other objects re-use the basic algorithm using different strategies. However, the navtree strategy for the actual navigation tree was still hard-coded into Plone, so if you wanted slightly different strategies in a different context, say, you couldn't do that.

In Plone 2.5, the strategy objects were defined with the following interface in 'CMFPlone.browser.interfaces':

    class INavtreeStrategy(Interface):

        rootPath = Attribute("The path to the root of the navtree (None means use portal root)")

        showAllParents = Attribute("Whether or not to show all parents of the current context always")

        def nodeFilter(node):
            """Return True or False to determine whether to include the given node
            in the tree. Nodes are dicts with at least one key - 'item', the
            catalog brain of the object the node represents.
            """

        def subtreeFilter(node):
            """Return True or False to determine whether to expand the given
            (folderish) node
            """

        def decoratorFactory(node):
            """Inject any additional keys in the node that are needed and return
            the new node.
            """

The basic implementations for these are defined in CMFPlone.browser.navtree, e.g.:

    class SitemapNavtreeStrategy(NavtreeStrategyBase):
        """The navtree building strategy used by the sitemap, based on
        navtree_properties
        """
        implements(INavtreeStrategy)
        #adapts(*, ISiteMap)

        def __init__(self, context, view=None):
            ...

        def nodeFilter(self, node):
            ...

        def subtreeFilter(self, node):
            ...

        def decoratorFactory(self, node):
            ...

ISiteMap is the interface that is used on the view that builds the sitemap, in CMFPlone.browser.navigation. The adapts line is commented out - in later versions of Zope it will be the easiest way of specifying which object this adapter class adapts from (the implements line tells you what it adapts to), but for now we need to wire this up in ZCML. In CMFPlone/browser/configure.zcml we have:

  <adapter for="*
                .interfaces.ISiteMap"
            factory=".navtree.SitemapNavtreeStrategy"
            provides=".interfaces.INavtreeStrategy" />

Note the newline - the adapter is registered for * (any interface) and ISiteMap. Since it's registered for more than one interface, it's a multi-adapter. The factory attribute specifies the class that is used to create the adapter, and the provides attribute tells the component architecture that this provider should be loaded when someone is looking for an INavtreeStrategy.

Also note the class paths starting with a .. This means they are relative to the directory that configure.zcml is in, CMFPlone.browser in this case. You could also use a full path, e.g. Products.CMFPlone.browswer.interfaces.INavtreeStrategy, but this makes it more difficult to refactor the package layout and is harder to read.

The sitemap creation code in CMFPlone.browser.navigation adapts the context (i.e. the content object being viewed) and the view class itself to an INavtreeStrategy in order to find out which strategy to use:

  from zope.component import getMultiAdapter
  ...
  strategy = getMultiAdapter((context, self), INavtreeStrategy)

In fact, if you were using a single-context adapter, you could just instantiate it like so:

  from interfaces import IFoobar
  ...
  foobar = IFoobar(context)

Here you are instantiating the interface as if it were an actual class (it kind of is, but that's beside the point). The component architecture knows to find the appropriate adapter factory by figuring out what interfaces context implements and finding the most specific adapter from one of these interfaces to IFoobar.

Note that the factory (e.g. the __init__ method) for the IFoobar adapter must take exactly one argument - the context (i.e. the object being adapted from), for a single-item adapter. For multi-adapters, it must take one argument for each interface specified in the for attribute in the adapter directive.

Now, let's say you wanted to use a different strategy when the sitemap was being viewed on IMyContentType. All you would need to do is to register a more specific adapter:

  <adapter for=".interfaces.IMyContentType
                .interfaces.ISiteMap"
            factory=".mycontent.MyContentTypeNavtreeStrategy"
            provides=".interfaces.INavtreeStrategy" />

The various Zope 3 resources cover adapters in much greater detail, but it is important to be familiar with adapters and understand the pattern they are trying to solve. Using adapters means talking to interfaces. If you talk to interfaces and not to classes directly, you not only ensure that your interface adequately covers the uses of your code, you also make it possible to substitute different implementations in different scenarios that you never dreamt of. Similarly, by using adapters to glue components together, you can avoid making "fat" interfaces that make components harder to re-use and understand.

As Plone progresses towards Zope 3, there are some general principles you should keep in mind to make sure your code is "future proof" and won't need to be refactored in the near future. This list may grow, but briefly:

• Look around you! A lot of problems have been solved in CMF, in Zope 3, or by other Python or JavaScript libraries. Before you start inventing something new that the community will have to maintain, consider carefully whether you can re-use something else.
• Declare interfaces for new (or existing) code. In general, any new component should use a well thought-out interface. A class should be registered as a factory for an adapter, and any code using that class should use the interface directly, instantiating the class through the component architecture rather than use the class constructor directly. This will make code much easier to refactor and much easier to override.
• Similarly, where existing tools and objects have interfaces (the CMF tools all have interfaces starting with CMF 1.6, for example), instantiate the objects through the component architecture, not directly from the classes.
• Don't add new Script (Python)'s. In general, any view logic should be in a Zope 3 view. Generic functionality should use adapters or utilities.
• We only ever use DTML for email handling (because it's better preserving whitespace than page templates), and even there we keep it to a minimum. DTML is hard to debug and maintain, and is most definitely frowned upon.
• Use Zope 3 naming conventions for filenames and package layout.

Whilst working on Plone, there will surely be times when you need to debug your own code, or someone else's code. Debugging Plone core is no different from debugging a third party producy, but there are some general tips you should bear in mind:

• pdb is your friend. Put import pdb; pdb.set_trace() in your code, restart Zope, reload the page and use the debugger in your terminal. Learn how the debugger works and how to make the most of it.
• Putting pdb sessions inside unit tests is often the most predictable way of entering a debugging session
• If it helps, put print statements in your code and watch the output in the terminal, or put debug elements in a page template with tal:content.
• If you need to debug a TTW Python Script, use context.plone_log("message") and watch your event.log
• If it makes your life easier, put some debug code deep inside CMF or Archetypes or even Zope itself. You can always remove the file, run an svn up and get back to the previous state.
• To debug visual problems, use the Mozilla DOM Inspector and the Web Developer and Firebug extensions to Firefox. These will save you hours of trial-and-error.
• To debug JavaScript, install the Venkman Debugger for Mozilla.
• Always remove any debug statements and run all tests before checkin! Checking in pdb statements is truly embarrassing.