To use Dexterity, you simply need to depend on the plone.app.dexterity package. If you are on Zope 2.10 (which is likely if you're using Plone 3.x), you will also need to ensure that certain packages that come with Zope are upgraded to newer versions. The easiest way to achieve this is to use a buildout that pins certain versions.

For a minimal buildout, see the installation how-to. In this section, will expand upon this to add some development tools.

To create the buildout, you can start with a standard Plone 3 buildout and modify buildout.cfg to looks something like this. You should update the Dexterity and Plone versions as appropriate:

[buildout]
extensions = mr.developer buildout.dumppickedversions
unzip = true
parts = instance omelette zopepy test roadrunner
extends = 
    http://dist.plone.org/release/3.3.5/versions.cfg
    http://good-py.appspot.com/release/dexterity/1.0b1?plone=3.3.5
versions = versions
develop = 
# If you're not using mr.developer to manage develop eggs, list eggs here. Globs OK.
#    src/*

sources = sources
auto-checkout =
    example.conference

[instance]
recipe = plone.recipe.zope2instance
zope2-location = ${zope2:location}
user = admin:admin
http-address = 8080
debug-mode = on
verbose-security = on
eggs = 
    Plone
    example.conference
# development tools
    plone.reload
    Products.PDBDebugMode
zcml =

[zope2]
recipe = plone.recipe.zope2install
url = ${versions:zope2-url}
fake-zope-eggs = true
additional-fake-eggs = zdaemon

[omelette]
recipe = collective.recipe.omelette
eggs = ${instance:eggs}
packages = ${zope2:location}/lib/python ./

[zopepy]
recipe = zc.recipe.egg
eggs = ${instance:eggs}
interpreter = zopepy
extra-paths = ${zope2:location}/lib/python
scripts = zopepy

[roadrunner]
recipe = roadrunner:plone
packages-under-test =
    example.conference

[test]
recipe = zc.recipe.testrunner
eggs = 
    example.conference
extra-paths = ${zope2:location}/lib/python
defaults = ['--exit-with-status', '--auto-color', '--auto-progress']

[sources]
example.conference = svn https://svn.plone.org/svn/collective/example.conference/trunk

You will see references to a package called example.conference. We'll create that shortly. Let's first go through and explain the buildout, however.

• We define two buildout extensions, mr.developer, which helps us manage our code, and buildout.dumppickedversions, which helps us keep track of which versions buildout has picked for our dependencies. If you are not familiar with mr.developer, you should read its documentation, in particular the documentation about the ./bin/develop command.
• We extend two version configurations: that for the desired version of Plone, and that for release 1.0b1 of Dexterity, targeted at Plone 3.3.5. You should adjust your Plone version accordingly. This allows Dexterity to update certain packages in Plone. You should check the Dexterity project page to discover which is the latest version. The known good set URL will give us the latest known good set of Dexterity packages, making it safe to depend on plone.app.dexterity in our example.conference product. The versions = versions line will make buildout use the known good set defined by the extended URLs.
• We tell mr.developer which section contain the sources to our packages with sources = sources (the [sources] section is at the end of the file). We also tell it to automatically check out and configure our example.conference package. This will check out the package form the given version repository URL, put it in src/ and ensure that it is configured as a develop egg.
• If you don't have a version repository yet, you can just put the egg in the src/ directory. The develop = src/* line will pick the egg up from there. Note: With mr.developer installed, we comment this out so that we don't get the same egg loaded twice.
• We pin zc.buildout to make sure we use the latest version.
• We configure a standard Zope instance and add two development tools to the eggs line:
• Products.PdbDebugMode will drop to a pdb shell when an exception occurs.
• plone.reload lets you go to localhost:8080/@@reload to reload code. Look at the plone.reload documentation for details
• We also add the Plone egg and our new package.
• We configure a standard Zope 2 server, which is used by our instance.
• We configure collective.recipe.omelette so that we get a set of links in parts/omelette giving access to all the code that is currently used by the instance. If you are on Windows, you will need to install junction.exe for this to work. See the omelette documentation for details.
• We install a testrunner. This will give us a bin/test command which can use to run our tests. Note: Only those eggs listed directly here will be available to the test runner. If you want to run tests for a dependency, you need to list it explicitly under the eggs option in the [test] part.
• We configure roadrunner to speed up integration tests. See the roadrunner documentation.

With this buildout, and a standard bootstrap.py file, you can run the usual python bootstrap.py; ./bin/buildout sequence to configure Plone and Dexterity. Before we do that, though, we need to create the package.

Typically, our content types will live in a separate package to our theme and other customisations. In the previous section, we showed how our buildout refers to a package in the src/ directory, either placed there manually or checked out by mr.developer, called example.conference. You can find the latest version of this package in the Collective repository.

To create a new package, we can start with ZopeSkel and the plone template. See this how-to for more information on how to install ZopeSkel.

We run the following from the src/ directory:

$ paster create -t plone example.conference

If you are using this template, make sure that you specify a namespace (example) and package name (conference) that matches the egg name (example.conference) on the command line. Answer False when asked to create a Zope 2 product, and False again when asked if the product is zip-safe.

Next, we will normalise the code created by paster, mainly by removing things we don't need.

First, we edit setup.py to add plone.app.dexterity as a dependency and specify the package as a z3c.autoinclude plug-in. This ensures that we do not need to load its ZCML separately once the package is configured in buildout.cfg (this feature is enabled in Plone 3.3 and later). We will also add a dependency on collective.autopermission, which will help us define custom permissions later. 

We can remove the paster plugin entry point and paster_plugins line. We will not need these.

from setuptools import setup, find_packages
import os

version = '1.0a1'

setup(name='example.conference',
      version=version,
      description="Example accompanying http://plone.org/products/dexterity/documentation/manual/developers-manual/",
      long_description=open("README.txt").read() + "\n" +
                       open(os.path.join("docs", "HISTORY.txt")).read(),
      # Get more strings from http://www.python.org/pypi?%3Aaction=list_classifiers
      classifiers=[
        "Framework :: Plone",
        "Programming Language :: Python",
        "Topic :: Software Development :: Libraries :: Python Modules",
        ],
      keywords='plone dexterity example',
      author='Martin Aspeli',
      author_email='optilude@gmail.com',
      url='http://plone.org/products/dexterity',
      license='GPL',
      packages=find_packages(exclude=['ez_setup']),
      namespace_packages=['example'],
      include_package_data=True,
      zip_safe=False,
      install_requires=[
          'setuptools',
          'Plone',
          'plone.app.dexterity',
          'collective.autopermission',
      ],
      entry_points="""
      [z3c.autoinclude.plugin]
      target = plone
      """,
      )

Next, we edit configure.zcml and add the following:

<configure
    xmlns="http://namespaces.zope.org/zope"
    xmlns:grok="http://namespaces.zope.org/grok"
    xmlns:genericsetup="http://namespaces.zope.org/genericsetup"
    i18n_domain="example.conference">

    <!-- Include configuration for dependencies listed in setup.py -->
    <includeDependencies package="." />

    <!-- Grok the package to initialise schema interfaces and content classes -->
    <grok:grok package="." />

    <!-- Register an extension profile to make the product installable -->
    <genericsetup:registerProfile
        name="default"
        title="Conference management"
        description="A Dexterity demo"
        directory="profiles/default"
        provides="Products.GenericSetup.interfaces.EXTENSION"
        />
        
</configure>

Here, we first automatically include the ZCML configuration for all packages listed under install_requires in setup.py. This feature is part of z3c.autoinclude, which is included with Plone 3.3 and later. The alternative would be to manually add a line like <include package="plone.app.dexterity" /> for each dependency.

Next, we grok the package to construct and register schemata, views, forms and so on based on conventions used in the various files we will add throughout this tutorial.

Finally, we register a GenericSetup profile to make the type installable, which we will build up over the next several sections.

The profile requires a directory profiles/default. You should create the profiles directory in the same folder as configure.zcml, and default under that. In default, add a file called metadata.xml with the following contents:

<metadata>
    <version>1</version>
    <dependencies>
        <dependency>profile-plone.app.dexterity:default</dependency>
    </dependencies>
</metadata>

This gives the profile a version number (which is different to the package version set in setup.py) in case we need to define upgrade steps in the future, and declares that plone.app.dexterity should be installed when this package is installed. We can add other profiles to depend on in the same way if you need to.

With this in place, we should be able to go up to the buildout root and run:

$ python2.4 bootstrap.py
$ ./bin/buildout

The buildout should now configure Plone, Dexterity and the example.conference package.

We are now ready to start adding types.

A simple Dexterity types consists of a schema and an FTI (Factory Type Information, the object configured in portal_types in the ZMI). We'll create the schemata here, and the FTI on the next page.

Each schema is typically in a separate module. Thus, we will add three files to our product: presenter.py, program.py, and session.py. Each will start off with a schema interface.

First, we will define a message factory to aid future internationalisation of the package. Every string that is presented to the user should be wrapped in _() as shown with the titles and descriptions below.

The message factory lives in the package root __init__.py file:

from zope.i18nmessageid import MessageFactory

_ = MessageFactory("example.conference")

Notice how we use the package name as the translation domain.

We can now define the schemata for our three types.

For the Presenter type, presenter.py looks like this:

from five import grok
from zope import schema

from plone.directives import form, dexterity

from plone.app.textfield import RichText
from plone.namedfile.field import NamedImage

from example.conference import _

class IPresenter(form.Schema):
    """A conference presenter. Presenters can be added anywhere.
    """
    
    title = schema.TextLine(
            title=_(u"Name"),
        )
    
    description = schema.Text(
            title=_(u"A short summary"),
        )
    
    bio = RichText(
            title=_(u"Bio"),
            required=False
        )
    
    picture = NamedImage(
            title=_(u"Picture"),
            description=_(u"Please upload an image"),
            required=False,
        )

Notice how we use the field names title and description for the name and summary. We do this to provide values for the default title and description metadata used in Plone's folder listings and searches, which defaults to these fields. In general, every type should have a title field, although it could be provided by behaviors (more on those later).

For the Program type, program.py looks like this:

from five import grok
from zope import schema

from plone.directives import form, dexterity
from plone.app.textfield import RichText

from example.conference import _

class IProgram(form.Schema):
    """A conference program. Programs can contain Sessions.
    """
    
    title = schema.TextLine(
            title=_(u"Program name"),
        )
    
    description = schema.Text(
            title=_(u"Program summary"),
        )
    
    start = schema.Datetime(
            title=_(u"Start date"),
            required=False,
        )

    end = schema.Datetime(
            title=_(u"End date"),
            required=False,
        )
    
    details = RichText(
            title=_(u"Details"),
            description=_(u"Details about the program"),
            required=False,
        )

Finally, session.py for the Session type looks like this:

from five import grok
from zope import schema

from plone.directives import form, dexterity
from plone.app.textfield import RichText

class ISession(form.Schema):
    """A conference session. Sessions are managed inside Programs.
    """
    
    title = schema.TextLine(
            title=_(u"Title"),
            description=_(u"Session title"),
        )
    
    description = schema.Text(
            title=_(u"Session summary"),
        )
    
    details = RichText(
            title=_(u"Session details"),
            required=False
        )

Note that we haven't added information about speakers or tracks yet. We'll do that when we cover vocabularies and references later.

Schema interfaces vs. other interfaces

As you may have noticed, each schema is basically just an interface (zope.interface.Interface) with fields. The standard fields are found in the zope.schema package. You should look at its interfaces (parts/omelette/zope/schema/interfaces.py) to learn about the various schema fields available, and review the online documentation for the package. You may also want to look up plone.namedfile, which you can use if you require a file field, z3c.relationfield, which can be used for references, and plone.app.textfield, which supports rich text with a WYSIWYG editor. We will cover these field types later in this manual. They can also be found in the reference at the end.

Unlike a standard interface, however, we are deriving from form.Schema (actually, plone.directives.form.Schema). This is just a marker interface that allows us to add some form hints to the interface, which are then used by Dexterity (actually, the plone.autoform package) to construct forms. Take a look at the plone.directives.form documentation to learn more about the various hints that are possible. The most common ones are form.fieldset(), to define groups of fields, form.widget(), to set a widget for a particular field, and form.omit() to hide one or more fields from the form. We will see examples of these later in the manual.

With the schema in place, we just need to make our types installable. We do this with GenericSetup.

First, we add a types.xml file to profiles/default:

<object name="portal_types">
 <object name="example.conference.presenter" meta_type="Dexterity FTI" />
 <object name="example.conference.program" meta_type="Dexterity FTI" />
 <object name="example.conference.session" meta_type="Dexterity FTI" />
</object>

We use the package name as a prefix and the type name in lowercase to create a unique name. It is important that the meta_type is Dexterity FTI.

We then need to add an XML file for each of the types, where the file name matches the type name. First, we add a directory profiles/default/types, and then add the following:

For the Presenter type, we have example.conference.presenter.xml:

<?xml version="1.0"?>
<object name="example.conference.presenter" meta_type="Dexterity FTI"
   i18n:domain="example.conference" xmlns:i18n="http://xml.zope.org/namespaces/i18n">
 
 <!-- Basic metadata -->
 <property name="title" i18n:translate="">Presenter</property>
 <property name="description" i18n:translate="">A person presenting sessions</property>
 <property name="content_icon">user.gif</property>
 <property name="allow_discussion">True</property>
 <property name="global_allow">True</property>
 <property name="filter_content_types">True</property>
 <property name="allowed_content_types" />
 
 <!-- schema interface -->
 <property name="schema">example.conference.presenter.IPresenter</property> 
 
 <!-- class used for content items -->
 <property name="klass">plone.dexterity.content.Item</property>
 
 <!-- add permission -->
 <property name="add_permission">cmf.AddPortalContent</property>
 
 <!-- enabled behaviors -->
 <property name="behaviors">
     <element value="plone.app.content.interfaces.INameFromTitle" />
 </property>
 
 <!-- View information -->
 <property name="default_view">view</property>
 <property name="default_view_fallback">False</property>
 <property name="view_methods">
  <element value="view"/>
 </property>
 
 <!-- Method aliases -->
 <alias from="(Default)" to="(selected layout)"/>
 <alias from="edit" to="@@edit"/>
 <alias from="sharing" to="@@sharing"/>
 <alias from="view" to="@@view"/>
 
 <!-- Actions -->
 <action title="View" action_id="view" category="object" condition_expr=""
    url_expr="string:${object_url}" visible="True">
  <permission value="View"/>
 </action>
 <action title="Edit" action_id="edit" category="object" condition_expr=""
    url_expr="string:${object_url}/edit" visible="True">
  <permission value="Modify portal content"/>
 </action>
</object>

There is a fair amount of boilerplate here which could actually be omitted, because the Dexterity FTI defaults will take care of most of this. However, it is useful to see the options available so that you know what you can change.

The important lines here are:

• The name attribute on the root element must match the name in types.xml and the filename
• We use the package name as the translation domain again, via i18n:domain
• We set a title and description for the type
• We also specify an icon. Here, we use a standard icon from Plone's plone_images skin layer. You'll learn more about static resources later.
• We set global_allow to True. This means that the type will be addable in standard folders.
• The schema interface is referenced by the schema property.
• We set the klass property to the standard plone.dexterity.content.Item. There is also plone.dexterity.content.Container. 
• We specify the name of an add permission. The default cmf.AddPortalContent should be used unless you configure a custom permission. Custom permissions are convered later in this manual.
• We add a behavior. Behaviors are re-usable aspects providing semantics and/or schema fields. Here, we add the INameFromTitle behavior, which will give our content object a readable id based on the title property. We'll cover other behaviors later.

The Session type, in example.conference.session.xml, is very similar:

<?xml version="1.0"?>
<object name="example.conference.session" meta_type="Dexterity FTI"
   i18n:domain="example.conference" xmlns:i18n="http://xml.zope.org/namespaces/i18n">
 
 <!-- Basic metadata -->
 <property name="title" i18n:translate="">Session</property>
 <property name="description" i18n:translate="">A session on the program</property>
 <property name="content_icon">document_icon.gif</property>
 <property name="allow_discussion">True</property>
 <property name="global_allow">False</property>
 <property name="filter_content_types">True</property>
 <property name="allowed_content_types" />
 
 <!-- schema interface -->
 <property name="schema">example.conference.session.ISession</property> 
 
 <!-- class used for content items -->
 <property name="klass">plone.dexterity.content.Item</property>
 
 <!-- add permission -->
 <property name="add_permission">cmf.AddPortalContent</property>
 
 <!-- enabled behaviors -->
 <property name="behaviors">
     <element value="plone.app.content.interfaces.INameFromTitle" />
 </property>
 
 <!-- View information -->
 <property name="default_view">view</property>
 <property name="default_view_fallback">False</property>
 <property name="view_methods">
  <element value="view"/>
 </property>
 
 <!-- Method aliases -->
 <alias from="(Default)" to="(selected layout)"/>
 <alias from="edit" to="@@edit"/>
 <alias from="sharing" to="@@sharing"/>
 <alias from="view" to="@@view"/>
 
 <!-- Actions -->
 <action title="View" action_id="view" category="object" condition_expr=""
    url_expr="string:${object_url}" visible="True">
  <permission value="View"/>
 </action>
 <action title="Edit" action_id="edit" category="object" condition_expr=""
    url_expr="string:${object_url}/edit" visible="True">
  <permission value="Modify portal content"/>
 </action>
</object>

Again, this is an Item. Here, we have set global_allow to False, since these objects should only be addable inside a Program.

The Program, in example.conference.program.xml, looks like this:

<?xml version="1.0"?>
<object name="example.conference.program" meta_type="Dexterity FTI"
   i18n:domain="example.conference" xmlns:i18n="http://xml.zope.org/namespaces/i18n">
 
 <!-- Basic metadata -->
 <property name="title" i18n:translate="">Program</property>
 <property name="description" i18n:translate="">A conference program</property>
 <property name="content_icon">folder_icon.gif</property>
 <property name="allow_discussion">True</property>
 <property name="global_allow">True</property>
 <property name="filter_content_types">True</property>
 <property name="allowed_content_types">
     <element value="example.conference.session" />
 </property>
 
 <!-- schema interface -->
 <property name="schema">example.conference.program.IProgram</property> 
 
 <!-- class used for content items -->
 <property name="klass">plone.dexterity.content.Container</property>
 
 <!-- add permission -->
 <property name="add_permission">cmf.AddPortalContent</property>
 
 <!-- enabled behaviors -->
 <property name="behaviors">
     <element value="plone.app.content.interfaces.INameFromTitle" />
 </property>
 
 <!-- View information -->
 <property name="default_view">view</property>
 <property name="default_view_fallback">False</property>
 <property name="view_methods">
  <element value="view"/>
 </property>
 
 <!-- Method aliases -->
 <alias from="(Default)" to="(selected layout)"/>
 <alias from="edit" to="@@edit"/>
 <alias from="sharing" to="@@sharing"/>
 <alias from="view" to="@@view"/>
 
 <!-- Actions -->
 <action title="View" action_id="view" category="object" condition_expr=""
    url_expr="string:${object_url}" visible="True">
  <permission value="View"/>
 </action>
 <action title="Edit" action_id="edit" category="object" condition_expr=""
    url_expr="string:${object_url}/edit" visible="True">
  <permission value="Modify portal content"/>
 </action>
</object>

The difference here is that we use the Container class, and we filter the containable types (filter_content_types and allowed_content_types) to allow only Sessions to be added inside this folder.

With a schema and FTI for each type, and our GenericSetup profile registered in configure.zcml, we should be able to test our type. Make sure that you have run a buildout, and then start ./bin/instance fg as normal. Add a Plone site, and go to the portal_quickinstaller in the ZMI. You should see your package there and be able to install it.

Once installed, you should be able to add objects of the new content types.

If Zope doesn't start up:

• Look for error messages on the console, and make sure you start in the foreground with ./bin/instance fg. You could have a syntax error or a ZCML error.

If you don't see your package in portal_quickinstaller:

• Ensure that the package is either checked out by mr.developer or that you have a develop line in buildout.cfg to load it as a develop egg. develop = src/* should suffice, but you can also add the package explicitly, e.g. with develop = src/example.conference.
• Ensure that the package is actually loaded as an egg. It should be referenced in the eggs section under [instance].
• You can check that the package is correctly configured in the buildout by looking at the generated bin/instance script (bin\instance-script.py on Windows). There should be a line for your package in the list of eggs at the top of the file.
• Make sure that the package's ZCML is loaded. You can do this by installing a ZCML slug (via the zcml option in the [instance] section of buildout.cfg) or by adding an <include /> line in another package's configure.zcml. However, the easiest way with Plone 3.3 and later is to add the z3c.autoinclude.plugin entry point to setup.py.
• Ensure that you have added a <genericsetup:registerProfile /> stanza to configure.zcml.

If the package fails to install in portal_quickinstaller:

• Look for errors in the error_log at the root of the Plone site, in your console, or in your log files.
• Check the syntax and placement of the profile files. Remember that you need a types.xml listing your types, and corresponding files in types/*.xml.

If your forms do not look right (e.g. you are missing custom widgets):

• Make sure your schema derives from form.Schema.
• Remember that the directives require you to specify the correct  field name, even if they are placed before or after the relevant field.
• Check that you have a <grok:grok package="." /> line in configure.zcml.

So far, our types have used the default views, which use the display widgets from z3c.form, much like the add and edit forms use the edit widgets. This is functional, but not very attractive. Most types will need one or more custom view templates.

Dexterity types are no different to any other content type in Plone. You can register a view for your schema interface, and it will be available on your type. If the view is named view, it will be the default view, at least if you use the standard FTI configuration. This is because the FTI's default_view property is set to view, and view is in the list of view_methods.

When working with Dexterity, we will typically configure our views using the five.grok configuration system, eschewing ZCML configuration. Below, we will show how to add simple views for the Program and Speaker types. Next, we will show how to use display forms to take advantage of the standard widgets if required.

The five.grok view approach uses a class in the content type's module, which is automatically associated with a template in an accompanying directory. These directories should be created next to the module files, so we will have program_templates, presenter_templates and session_templates.

In program.py, the view is registered as follows:

class View(grok.View):
    grok.context(IProgram)
    grok.require('zope2.View')
    
    def sessions(self):
        """Return a catalog search result of sessions to show
        """
        
        context = aq_inner(self.context)
        catalog = getToolByName(context, 'portal_catalog')
        
        return catalog(object_provides=ISession.__identifier__,
                       path='/'.join(context.getPhysicalPath()),
                       sort_on='sortable_title')

This creates a view registration similar to what you may do with a <browser:page /> ZCML directive. We have also added a helper method which will be used in the view. Note that this requires some imports at the top of the file:

from Acquisition import aq_inner
from Products.CMFCore.utils import getToolByName

from example.conference.session import ISession

The view registration works as follows:

• The view name will be @@view, taken from the class name in lowercase. You can specify an alternative name with grok.name('some-name') if required.
• The grok.context() directive specifies that this view is used for objects providing IProgram.
• You can add a grok.layer() directive if you want to specify a browser layer.
• The grok.require() directive specifies the required permission for this view. It uses the Zope 3 permission name. zope2.View and zope.Public are the most commonly used permissions (in fact, zope.Public is not actually a permission, it just means "no permission required"). For a list of other standard permissions, see parts/omelette/Products/Five/permissions.zcml. We will cover creating custom permissions later in this manual.
• Any methods added to the view will be available to the template via the view variable. The content object is available via context, as usual.

This is associated with a file in program_templates/view.pt. The file name matches the class name (even if a different view name was specified). This contains:

<html xmlns="http://www.w3.org/1999/xhtml" xml:lang="en"
      xmlns:tal="http://xml.zope.org/namespaces/tal"
      xmlns:metal="http://xml.zope.org/namespaces/metal"
      xmlns:i18n="http://xml.zope.org/namespaces/i18n"
      lang="en"
      metal:use-macro="context/main_template/macros/master"
      i18n:domain="example.conference">
<body>

<metal:main fill-slot="main">
    <tal:main-macro metal:define-macro="main"
        tal:define="toLocalizedTime nocall:context/@@plone/toLocalizedTime">

        <div tal:replace="structure provider:plone.abovecontenttitle" />

        <h1 class="documentFirstHeading" tal:content="context/title" />
        
        <div class="discreet">
            <tal:block condition="context/start">
                <span i18n:translate="label_from">From:</span>
                <span tal:content="python:context.start.strftime('%x %X')" />
            </tal:block>
            <tal:block condition="context/end">
                <span i18n:translate="label_to">To:</span>
                <span tal:content="python:context.end.strftime('%x %X')" />
            </tal:block>
        </div>

        <div tal:replace="structure provider:plone.belowcontenttitle" />

        <p class="documentDescription" tal:content="context/description" />

        <div tal:replace="structure provider:plone.abovecontentbody" />

        <div tal:content="structure context/details/output" />
        
        <h2 i18n:translate="heading_sessions">Sessions</h2>
        <dl>
            <tal:block repeat="session view/sessions">
                <dt>
                    <a tal:attributes="href session/getURL"
                       tal:content="session/Title" />
                </dt>
                <dd tal:content="session/Description" />
            </tal:block>
        </dl>

        <div tal:replace="structure provider:plone.belowcontentbody" />

    </tal:main-macro>
</metal:main>

</body>
</html>

For the most part, this template outputs the values of the various fields, using the sessions() method on the view to obtain the sessions contained within the program.

Notice how the details RichText field is output as tal:content="structure context/details/output". The structure keyword ensure the rendered HTML is not escaped. The extra traversal to details/output is necessary because the RichText field actually stores a RichTextValue object that contains not only the raw text as entered by the user, but also a MIME type (e.g. text/html) and the rendered output text. RichText fields are covered in more detail later in this manual.

The view for Presenter, in presenter.py, is even simpler:

class View(grok.View):
    grok.context(IPresenter)
    grok.require('zope2.View')

Its template, in presenter_templates/view.pt, is similar to the previous template:

<html xmlns="http://www.w3.org/1999/xhtml" xml:lang="en"
      xmlns:tal="http://xml.zope.org/namespaces/tal"
      xmlns:metal="http://xml.zope.org/namespaces/metal"
      xmlns:i18n="http://xml.zope.org/namespaces/i18n"
      lang="en"
      metal:use-macro="context/main_template/macros/master"
      i18n:domain="example.conference">
<body>

<metal:main fill-slot="main">
    <tal:main-macro metal:define-macro="main">

        <div tal:replace="structure provider:plone.abovecontenttitle" />

        <h1 class="documentFirstHeading" tal:content="context/title" />
        
        <div tal:replace="structure provider:plone.belowcontenttitle" />

        <p class="documentDescription" tal:content="context/description" />

        <div tal:replace="structure provider:plone.abovecontentbody" />

        <div tal:content="structure context/bio/output" />

        <div tal:replace="structure provider:plone.belowcontentbody" />

    </tal:main-macro>
</metal:main>

</body>
</html>

Obviously, these views are very basic. Much more interesting views could be created by putting a little more work into the templates.

You should also realise that you can create any type of view using this technique. Your view does not have to be related to a particular content type, even. You could set the context to Interface, for example, to make a view that's available on all types.

In the previous section, we created a view extending grok.View. This kind of view is the most common, but sometimes we want to make use of the widgets and information in the type's schema more directly, for example to invoke transforms or re-use more complex HTML.

To do this, you can use a display form. This is really just a view base class that knows about the schema of a type. We will use an example in session.py, with a template insession_templates/view.pt.

Note: Display forms involve the same type of overhead as add- and edit-forms. If you have complex forms with many behaviors, fieldsets and widget hints, you may notice a slow-down compared to standard views, at least on high volume sites.

The new view class is pretty much the same as before, except that we derive from dexterity.DisplayForm (plone.directives.dexterity.DisplayForm):

class View(dexterity.DisplayForm):
    grok.context(ISession)
    grok.require('zope2.View')

This gives our view a few extra properties that we can use in the template:

• view.w is a dictionary of all the display widgets, keyed by field names. For fields provided by behaviors, that is usually prefixed with the behavior interface name (IBehaviorInterface.field_name). For the default schema, unqualified names apply.
• view.widgets contains a list of widgets in schema order for the default fieldset
• view.groups contains a list of fieldsets in fieldset order.
• view.fieldsets contains a dict mapping fieldset name to fieldset
• On a fieldset (group), you can access a widgets list to get widgets in that fieldset

The w dict is the mostly commonly used.

The session_templates/view.pt template contains the following:

<html xmlns="http://www.w3.org/1999/xhtml" xml:lang="en"
      xmlns:tal="http://xml.zope.org/namespaces/tal"
      xmlns:metal="http://xml.zope.org/namespaces/metal"
      xmlns:i18n="http://xml.zope.org/namespaces/i18n"
      lang="en"
      metal:use-macro="context/main_template/macros/master"
      i18n:domain="example.conference">
<body>

<metal:main fill-slot="main">
    <tal:main-macro metal:define-macro="main">

        <div tal:replace="structure provider:plone.abovecontenttitle" />

        <h1 class="documentFirstHeading" tal:content="context/title" />
        
        <div tal:replace="structure provider:plone.belowcontenttitle" />

        <p class="documentDescription" tal:content="context/description" />

        <div tal:replace="structure provider:plone.abovecontentbody" />

        <div tal:content="structure view/w/details/render" />

        <div tal:replace="structure provider:plone.belowcontentbody" />

    </tal:main-macro>
</metal:main>

</body>
</html>

Notice how we use expressions like view/w/details/render (where details is the field name) to get the rendering of a widget. Other properties include __name__, the field name, and label, the field title.

It is often useful to calculate a default value for a field. This value will be used on the add form, before the field is set.

To continue with our conference example, let's set the default values for the start and end dates to one week in the future and ten days in the future, respectively. We can do this by adding the following to program.py:

@form.default_value(field=IProgram['start'])
def startDefaultValue(data):
    # To get hold of the folder, do: context = data.context
    return datetime.datetime.today() + datetime.timedelta(7)


@form.default_value(field=IProgram['end'])
def endDefaultValue(data):
    # To get hold of the folder, do: context = data.context
    return datetime.datetime.today() + datetime.timedelta(10)

We also need to import datetime at the top of the file, of course.

Notice how the functions specify a particular schema field that they provide the default value for. The decorator will actually register these as "value adapters" for z3c.form, but you probably don't need to worry about that.

The data argument is an object that contains an attribute for each field in the schema. On the add form, most of these are likely to be None, but on a different form, the values may be populated from the context. The data object also has a context attribute that you can use to get the form's context. For add forms, that's the containing folder; for other forms, it is normally a content object being edited or displayed. If you need to look up tools (getToolByName) or acquire a value from a parent object, use data.context as the starting point, e.g.:

from Products.CMFCore.utils import getToolByName
...
catalog = getToolByName(data.context, 'portal_catalog')

The value returned by the method should be a value that's allowable for the field. In the case of Datetime fields, that's a Python datetime object.

It is possible to provide different default values depending on the type of context, a request layer, the type of form, or the type of widget used. See the plone.directives.form documentation for more details.

For example, if you wanted to have a differently calculated default for a particular form, you could use a decorator like:

@form.default_value(field=IProgram['start'], form=FormClass)

We'll cover creating custom forms later in this manual.

Many applications require some form of data entry validation. The simplest form of validation you get for free – the z3c.form library ensures that all data entered on Dexterity add and edit forms is valid for the field type.

It is also possible to set certain properties on the fields to add further validation (or even create your own fields with custom validation logic, although that is a lot less common). These properties are set as parameters to the field constructor when the schema interface is created. You should see the zope.schema package for details, but the most common constraints are:

• required=True/False, to make a field required or optional
• min and max, used for Int, Float, Datetime, Date, and Timedelta fields, specify the minimum and maximum (inclusive) allowed values of the given type
• min_length and max_length, used for collection fields (Tuple, List, Set, Frozenset, Dict) and text fields (Bytes, BytesLine, ASCII, ASCIILine, Text, TextLine), set the minimum and maximum (inclusive) length of a field

Constraints

If this does not suffice, you can pass your own constraint function to a field. The constraint function should take a single argument: the value that is to be validated. This will be of the field's type. The function should return a boolean True or False.

def checkForMagic(value):
    return 'magic' in value

Hint: The constraint function does not have access to the context, but if you need to acquire a tool, you can use the zope.app.component.hooks.getSite() method to obtain the site root

To use the constraint, pass the function as the constraint argument to the field constructor, e.g.:

    my_field = schema.TextLine(title=_(u"My field"), constraint=checkForMagic)

Constraints are easy to write, but do not necessarily produce very friendly error messages. It is however possible to customise these error messages using z3c.form error view snippets. See the z3c.form documentation for more details.

Invariants

You'll also notice that constraints only check a single field value. If you need to write a validator that compares multiple values, you can use an invariant. Invariants use exceptions to signal errors, which are displayed at the top of the form rather than next to a particular field.

To illustrate an invariant, let's make sure that the start date of a Program is before the end date. In program.py, we add the following. Code not relevant to this example is snipped with an ellipsis (...):

...

from zope.interface import invariant, Invalid

class StartBeforeEnd(Invalid):
    __doc__ = _(u"The start or end date is invalid")

class IProgram(form.Schema):
    
    ...
    
    start = schema.Datetime(
            title=_(u"Start date"),
            required=False,
        )

    end = schema.Datetime(
            title=_(u"End date"),
            required=False,
        )
    
    ...
    
    @invariant
    def validateStartEnd(data):
        if data.start is not None and data.end is not None:
            if data.start > data.end:
                raise StartBeforeEnd(_(u"The start date must be before the end date."))

...

Form validators

Finally, you can write more powerful validators by using the z3c.form widget validators. See the z3c.form documentation for details.

Vocabularies are normally used in conjunction with selection fields, and are supported by the zope.schema package, with widgets provided by z3c.form.

Selection fields use the Choice field type. To allow the user to select a single value, use a Choice field directly:

class IMySchema(form.Schema):
    myChoice = schema.Choice(...)

For a multi-select field, use a List, Tuple, Set or Frozenset with a Choice as the value_type:

class IMySchema(form.Schema):
    myList = schema.List(..., value_type=schema.Choice(...))

The choice field must be passed one of the following arguments:

• values can be used to give a list of static values
• source can be used to refer to an IContextSourceBinder or ISource instance
• vocabulary can be used to refer to an IVocabulary instance or (more commonly) a string giving the name of an IVocabularyFactory named utility.

In the remainder of this section, we will show the various techniques for defining vocabularies through several iterations of a new field added to the Program type allowing the user to pick the organiser responsible for the program.

Static vocabularies

Our first attempt uses a static list of organisers. We use the message factory to allow the labels to be translated. The values stored in the organizer field will be a unicode string representing the chosen label, or None if no value is selected.

    organizer = schema.Choice(
            title=_(u"Organiser"),
            values=[_(u"Bill"), _(u"Bob"), _(u"Jim"),],
            required=False,
        )

Since required is False, there will be a "no value" option in the drop-down list.

Dynamic sources

The static vocabulary is obviously a bit limited. Not only is it hard-coded in Python, it also does not allow separation of the stored values and the labels shown in the selection widget.

We can make a one-off dynamic vocabulary using a context source binder. This is simply a callable (usually a function or an object with a __call__ method) that provides the IContextSourceBinder interface and takes a context parameter. The context argument is the context of the form (i.e. the folder on an add form, and the content object on an edit form). The callable should return a vocabulary, which is most easily achieved by using the SimpleVocabulary class from zope.schema. 

Here is an example using a function to return all users in a particular group.

from zope.schema.interfaces import IContextSourceBinder
from zope.schema.vocabulary import SimpleVocabulary
from Products.CMFCore.utils import getToolByName

@grok.provider(IContextSourceBinder)
def possibleOrganizers(context):
    acl_users = getToolByName(context, 'acl_users')
    group = acl_users.getGroupById('organizers')
    terms = []
    
    if group is not None:
        for member_id in group.getMemberIds():
            user = acl_users.getUserById(member_id)
            if user is not None:
                member_name = user.getProperty('fullname') or member_id
                terms.append(SimpleVocabulary.createTerm(member_id, str(member_id), member_name))
            
    return SimpleVocabulary(terms)

We use the PAS API to get the group and its members, building a list, which we then turn into a vocabulary.

When working with vocabularies, you'll come across some terminology that is worth explaining:

• A term is an entry in the vocabulary. The term has a value. Most terms are tokenised terms which also have a token, and some terms are titled, meaning they have a title that is different to the token.
• The token must be an ASCII string. It is the value passed with the request when the form is submitted. A token must uniquely identify a term.
• The value is the actual value stored on the object. This is not passed to the browser or used in the form. The value is often a unicode string, but can be any type of object.
• The title is a unicode string or translatable message. It is used in the form.

The SimpleVocabulary class contains two class methods that can be used to create vocabularies from lists:

• fromValues() takes a simple list of values and returns a tokenised vocabulary where the values are the items in the list, and the tokens are created by calling str() on the values.
• fromItems() takes a list of (token, value) tuples and creates a tokenised vocabulary with the token and value specified.

You can also instantiate a SimpleVocabulary yourself and pass a list of terms in the initialiser. The createTerm() class method can be used to create a term from a value, token and title. Only the value is required.

In the example above, we have chosen to create a SimpleVocabulary from terms with the user id used as value and token, and the user's full name as a title.

To use this context source binder, we use the source argument to the Choice constructor:

    organizer = schema.Choice(
            title=_(u"Organiser"),
            source=possibleOrganizers,
            required=False,
        )

Parameterised sources

We can improve this example by moving the group name out of the function, allowing it to be set on a per-field basis. To do so, we turn our IContextSourceBinder into a class that is initialised with the group name.

class GroupMembers(object):
    """Context source binder to provide a vocabulary of users in a given
    group.
    """
    
    grok.implements(IContextSourceBinder)
    
    def __init__(self, group_name):
        self.group_name = group_name
    
    def __call__(self, context):
        acl_users = getToolByName(context, 'acl_users')
        group = acl_users.getGroupById(self.group_name)
        terms = []
    
        if group is not None:
            for member_id in group.getMemberIds():
                user = acl_users.getUserById(member_id)
                if user is not None:
                    member_name = user.getProperty('fullname') or member_id
                    terms.append(SimpleVocabulary.createTerm(member_id, str(member_id), member_name))
            
        return SimpleVocabulary(terms)

Again, the source is set using the source argument to the Choice constructor:

    organizer = schema.Choice(
            title=_(u"Organiser"),
            source=GroupMembers('organizers'),
            required=False,
        )

When the schema is initialised on startup, the a GroupMembers object is instantiated, storing the desired group name. Each time the vocabulary is needed, this object will be called (i.e. the __call__() method is invoked) with the context as an argument, expected to return an appropriate vocabulary.

Named vocabularies

Context source binders are great for simple dynamic vocabularies. They are also re-usable, since you can import the source from a single location and use it in multiple instances.

Sometimes, however, we want to provide an additional level of decoupling, by using named vocabularies. These are similar to context source binders, but are components registered as named utilities, referenced in the schema by name only. This allows local overrides of the vocabulary via the Component Architecture, and makes it easier to distribute vocabularies in third party packages.

Note: Named vocabularies cannot be parameterised in the way as we did with the GroupMembers context source binder, since they are looked up by name only.

We can turn our first "members in the organizers group" vocabulary into a named vocabulary by creating a named utility providing IVocabularyFactory, like so:

from zope.schema.interfaces import IVocabularyFactory

...

class OrganizersVocabulary(object):
    grok.implements(IVocabularyFactory)
    
    def __call__(self, context):
        acl_users = getToolByName(context, 'acl_users')
        group = acl_users.getGroupById('organizers')
        terms = []
    
        if group is not None:
            for member_id in group.getMemberIds():
                user = acl_users.getUserById(member_id)
                if user is not None:
                    member_name = user.getProperty('fullname') or member_id
                    terms.append(SimpleVocabulary.createTerm(member_id, str(member_id), member_name))
            
        return SimpleVocabulary(terms)

grok.global_utility(OrganizersVocabulary, name=u"example.conference.Organizers")

By convention, the vocabulary name is prefixed with the package name, to ensure uniqueness.

We can make use of this vocabulary in any schema by passing its name to the vocabulary argument of the Choice field constructor:

    organizer = schema.Choice(
            title=_(u"Organiser"),
            vocabulary=u"example.conference.Organizers",
            required=False,
        )

Some common vocabularies

As you might expect, there are a number of standard vocabularies that come with Plone. These are found in the plone.app.vocabularies package. Some of the more useful ones include

• plone.app.vocabularies.AvailableContentLanguages, a list of all available content languages
• plone.app.vocabularies.SupportedContentLanguages, a list of currently supported content languages
• plone.app.vocabularies.Roles, the user roles available in the site
• plone.app.vocabularies.PortalTypes, a list of types installed in portal_types
• plone.app.vocabularies.ReallyUserFriendlyTypes, a list of those types that are likely to mean something to users
• plone.app.vocabularies.Workflows, a list of workflows
• plone.app.vocabularies.WorkflowStates, a list of all states from all workflows
• plone.app.vocabularies.WorkflowTransitions, a list of all transitions from all workflows

In addition, the package plone.principalsource provides several vocabularies that are useful for selecting users and groups in a Dexterity context:

• plone.principalsource.Users provides users
• plone.principalsource.Groups provides groups
• plone.principalsource.Principals provides security principals (users or groups)

Importantly, these sources are not iterable, which means that you cannot use them to provide a list of all users in the site. This is intentional: calculating this list can be extremely expensive if you have a large site with many users, especially if you are connecting to LDAP or Active Directory. Instead, you should use a search-based source such as one of these.

We will use one of these together with an auto-complete widget to finalise our organizer field. To do so, we need to add plone.principalsource as a dependency of example.conference. In setup.py, we add:

    install_requires=[
          ...
          'plone.principalsource',
      ],

Since we use an <includeDependencies /> line in configure.zcml, we do not need a separate <include /> line in configure.zcml for this new dependency.

The organizer field now looks like:

    organizer = schema.Choice(
            title=_(u"Organiser"),
            vocabulary=u"plone.principalsource.Users",
            required=False,
        )

The autocomplete selection widget

The organizer field now has a query-based source. The standard selection widget (a drop-down list) is not capable of rendering such a source. Instead, we need to use a more powerful widget. For a basic widget, see z3c.formwidget.query, but in a Plone context, you will more likely want to use plone.formwidget.autocomplete, which extends z3c.formwidget.query to provide friendlier user interface.

The widget is provided with plone.app.dexterity, so we do not need to configure it ourselves. We only need to tell Dexterity to use this widget instead of the default, using a form widget hint as shown earlier. At the top of program.py, we add the following import:

from plone.formwidget.autocomplete import AutocompleteFieldWidget

If we were using a multi-valued field, such as a List with a Choice value_type, we would use the AutocompleteMultiFieldWidget instead

In the IProgram schema (which, recall, derives from form.Schema and is therefore processed for form hints at startup), we then add the following:

    form.widget(organizer=AutocompleteFieldWidget)
    organizer = schema.Choice(
            title=_(u"Organiser"),
            vocabulary=u"plone.principalsource.Users",
            required=False,
        )

You should now see a dynamic auto-complete widget on the form, so long as you have JavaScript enabled. Start typing a user name and see what happens. The widget also has fall-back for non-JavaScript capable browsers.

References are a way to maintain links between content that remain valid even if one or both content items are moved or renamed.

Under the hood, Dexterity's reference system uses five.intid, a Zope 2 integration layer for zope.intid, to give each content item a unique integer id. These are the basis for relationships maintained with the zc.relationship package, which in turn is accessed via an API provided by z3c.relationfield, integrated into Zope 2 with plone.app.relationfield. For most purposes, you need only to worry about the z3c.relationfield API, which provides methods for finding source and target objects for references and searching the relationship catalog.

References are most commonly used in form fields with a selection or content browser widget. Dexterity comes with a standard widget in plone.formwidget.contenttree configured for the RelationList and RelationChoice fields from z3c.relationfield.

To illustrate the use of references, we will allow the user to create a link between a Session and its Presenter. Since Dexterity already ships with and installs plone.formwidget.contenttree and z3c.relationfield, we do not need to add any further setup code, and we can use the field directly in session.py:

...

from z3c.relationfield.schema import RelationChoice
from plone.formwidget.contenttree import ObjPathSourceBinder

...

from example.conference.presenter import IPresenter

class ISession(form.Schema):
    """A conference session. Sessions are managed inside Programs.
    """
    
    ...
        
    presenter = RelationChoice(
            title=_(u"Presenter"),
            source=ObjPathSourceBinder(object_provides=IPresenter.__identifier__),
            required=False,
        )

To allow multiple items to be selected, we could have used a RelationList like:

    relatedItems = RelationList(
            title=u"Related Items",
            default=[],
            value_type=RelationChoice(title=_(u"Related"),
                                      source=ObjPathSourceBinder()),
            required=False,
        )

The ObjPathSourceBinder class is an IContextSourceBinder that returns a vocabulary with content objects as values, object titles as term titles and object paths as tokens.

You can pass keyword arguments to the constructor for ObjPathSourceBinder() to restrict the selectable objects. Here, we demand that the object must provide the IPresenter interface. The syntax is the same as that used in a catalog search, except that only simple values and lists are allowed (e.g. you can't use a dict to specify a range or values for a field index).

If you want to restrict the folders and other content shown in the content browser, you can pass a dictionary with catalog search parameters (and here, any valid catalog query will do) as the first non-keyword argument (navigation_tree_query) to the ObjPathSourceBinder() constructor.

If you want to use a different widget, you can use the same source (or a custom source that has content objects as values) with something like the autocomplete widget. The following line added to the interface will make the presenter selection similar to the organizer selection widget we showed in the previous section:

    form.widget(presenter=AutocompleteFieldWidget)

Once the user has created some relationships, the value stored in the relation field is a RelationValue object. This provides various attributes, including:

• from_object, the object from which the relationship is made
• to_object, the object to which the relationship is made
• from_id and to_id, the integer ids of the source and target
• from_path and to_path, the path of the source and target

The isBroken() method can be used to determine if the relationship is broken. This normally happens if the target object is deleted.

To display the relationship on our form, we can either use a display widget on a DisplayForm, or use this API to find the object and display it. We'll do the latter in session_templates/view.pt:

        <div tal:condition="context/presenter">
            <label i18n:translate="presenter">Presenter:</label>
            <span tal:content="context/presenter/to_object/Title | nothing" />
        </div>

Many content items need to allow users to provide rich text in some kind of markup, be that HTML (perhaps entered using a WYSIWYG editor), reStructuredText, Markdown or some other format. This markup typically needs to be transformed into HTML for the view template, but we also want to keep track of the original "raw" markup so that it can be edited again. Even when the input format is HTML, there is often a need for a transformation to tidy up the HTML and strip out tags that are not permitted.

It is possible to store HTML in a standard Text field. You can even get a WYSIWYG widget, by using a schema such as this:

from plone.directives import form
from zope import schema
from plone.app.z3cform.wysiwyg import WysiwygFieldWidget

class ITestSchema(form.Schema):

    form.widget(body=WysiwygFieldWidget)
    body = schema.Text(title=u"Body text")

However, this approach does not allow for alternative markups or any form of content filtering. For that, we need to use a more powerful field: RichText from the plone.app.textfield package.

from plone.directives import form
from plone.app.textfield import RichText

class ITestSchema(form.Schema):

    body = RichText(title=u"Body text")

The RichText field constructor can take the following arguments in addition to the usual arguments for a Text field:

• default_mime_type, a string representing the default MIME type of the input markup. This defaults to text/html.
  
• output_mime_type, a string representing the default output MIME type. This defaults to text/x-html-safe, which is a Plone-specific MIME type that disallows certain tags.Use the HTML Filtering control panel in Plone to control the tags.
• allowed_mime_types, a tuple of strings giving a vocabulary of allowed input MIME types. If this is None (the default), the allowable types will be restricted to those set in Plone's Markup control panel.

Also note: The default field can be set to either a unicode string (in which case it will be assumed to be a string of the default MIME type) or a RichTextValue object (see below).

Below is an example of a field allow StructuredText and reStructuredText, transformed to HTML by default.

from plone.directives import form
from plone.app.textfield import RichText

defaultBody = """\
Background
==========

Please fill this in

Details
=======

And this
"""

class ITestSchema(form.Schema):

    body = RichText(
            title=u"Body text",
            default_mime_type='text/x-rst',
            output_mime_type='text/x-html',
            allowed_mime_types=('text/x-rst', 'text/structured',),
            default=defaultBody,
        )

The RichTextValue

The RichText field does not store a string. Instead, it stores a RichTextValue object. This is an immutable object that has the following properties:

• raw, a unicode string with the original input markup
• mimeType, the MIME type of the original markup, e.g. text/html or text/structured.
  
• encoding, the default character encoding used when transforming the input markup. Most likely, this will be utf-8
• raw_encoded, the raw input encoded in the given encoding
• outputMimeType, the MIME type of the default output, taken from the field at the time of instantiation
• output, a unicode string representing the transformed output. If possible, this is cached persistently until the RichTextValue is replaced with a new one (as happens when an edit form is saved, for example).
  

The storage of the RichTextValue object is optimised for the case where the transformed output will be read frequently (i.e. on the view screen of the content object) and the raw value will be read infrequently (i.e. on the edit screen). Because the output value is cached indefinitely, you will need to replace the RichTextValue object with a new one if any of the transformation parameters change. However, as we will see below, it is possible to apply a different transformation on demand should you need to.

The code snippet belows shows how a RichTextValue object can be constructed in code. In this case, we have a raw input string of type text/plain that will be transformed to a default output of text/html. (Note that we would normally look up the default output type from the field instance.)

from plone.app.textfield.value import RichTextValue

...

context.body = RichTextValue(u"Some input text", 'text/plain', 'text/html')

Of course, the standard widget used for a RichText field will correctly store this type of object for you, so it is rarely necessary to create one yourself.

Using rich text fields in templates

What about using the text field in a template? If you are using a DisplayForm, the display widget for the RichText field will render the transformed output markup automatically. If you are writing TAL manually, you may try something like this:

<div tal:content="structure context/body" />

This, however, will render a string like:

RichTextValue object. (Did you mean <attribute>.raw or <attribute>.output?)

The correct syntax is:

<div tal:content="structure context/body/output" />

This will rendred the cached, transformed output. This operation is approximately as efficient as rendering a simple Text field, since the transformation is only applied once, when the value is first saved.

Alternative transformations

Sometimes, you may want to invoke alternative transformations. Under the hood, the default implementation uses the portal_transforms tool to calculate a transform chain from the raw value's input MIME type to the desired output MIME type. (Should you need to write your own transforms, take a look at this tutorial.) This is abstracted behind an ITransformer adapter to allow alternative implementations.

To invoke a transformation in code, you can use the following syntax:

from plone.app.textfield.interfaces import ITransformer

transformer = ITransformer(context)
transformedValue = transformer(context.body, 'text/plain')

The __call__() method of the ITransformer adapter takes a RichTextValue object and an output MIME type as parameters.

If you are writing a page template, there is an even more convenient syntax:

<div tal:content="structure context/@@text-transform/body/text/plain" />

The first traversal name gives the name of the field on the context (body in this case). The second and third give the output MIME type. If the MIME type is omitted, the default output MIME type will be used.

Note: Unlike the output property, the value is not cached, and so will be calculated each time the page is rendered.

Plone has dedicated File and Image types, and it is often preferable to use these for managing files and images. However, it is sometimes useful to treat binary data as fields on an object. When working with Dexterity, you can accomplish this by using plone.namedfile and plone.formwidget.namedfile.

The plone.namedfile package includes four field types, all found in the plone.namedfile.field module:

• NamedFile stores non-BLOB files. This is useful for small files when you don't want to configure BLOB storage.
• NamedImage stores non-BLOB images.
• NamedBlobFile stores BLOB files (see note below). It is otherwise identical to NamedFile.
• NamedBlobImage stores BLOB images (see note below). It is otherwise identical to NamedImage.

Note that NamedBlobFile and NamedBlobImage depends on z3c.blobfile. This dependency is specified via the [blobs] extra.

If you do not have z3c.blobfile in your buildout (most likely because you did not depend on the [blobs] extra for plone.namedfile), the NamedBlobFile and NamedBlobImage field and value types will not be importable. They are only defined if BLOB support is detected.

In use, the four field types are all pretty similar. They actually store persistent objects of type plone.namedfile.NamedFile, plone.namedfile.NamedImage, plone.namedfile.NamedBlobFile (if available) and plone.namedfile.NamedBlobImage (if available), respectively. Note the different module! These objects have attributes like data, to access the raw binary data, contentType, to get a MIME type, and filename, to get the original filename. The image values also support _height and _width to get image dimensions.

To use the non-BLOB image and file fields, it is sufficient to depend on plone.formwidget.namedfile, since this includes plone.namefile as a dependency. We prefer to be explicit in setup.py, however, since we will actually import directly from plone.namedfile:

      install_requires=[
          ...
          'plone.namedfile',
          'plone.formwidget.namedfile',
      ],

To use the [blobs] extra, we would need the following line instead of the plone.namedfile line instead:

          'plone.namedfile[blobs]',

Again, we do not need separate <include /> lines in configure.zcml for these new dependencies, because we use <includeDependencies />.

For the sake of illustration, we will add a (non-BLOB) image of the speaker to the Presenter type. In presenter.py, we add:

from plone.namedfile.field import NamedImage

class IPresenter(form.Schema):
    ...
    
    picture = NamedImage(
            title=_(u"Please upload an image"),
            required=False,
        )

To use this in a view, we can either use a display widget via a DisplayForm, or construct a download URL manually. Since we don't have a DisplayForm for the Presenter type, we'll do the latter (of course, we could easily turn the view into a display form as well).

In presenter_templates/view.pt, we add this block of TAL:

        <div tal:define="picture nocall:context/picture"
             tal:condition="nocall:picture">
            <img tal:attributes="src string:${context/absolute_url}/@@download/picture/${picture/filename};
                                 height picture/_height | nothing;
                                 width picture/_width | nothing;"
                />
        </div>

This constructs an image URL using the @@download view from plone.namedfile. This view takes the name of the field containing the file or image on the traversal subpath (/picture), and optionally a filename on a further sub-path. The filename is used mainly so that the URL ends in the correct extension, which can help ensure web browsers display the picture correctly. We also define the height and width of the image based on the values set on the object.

For file fields, you can construct a download URL in a similar way, using an <a /> tag, e.g.:

<a tal:attributes="href string:${context/absolute_url}/@@download/some_field/${context/some_field/filename}" />

Earlier in this manual, we have seen how to create views, and how to use file and image fields. These are all dynamic, however, and often we just want to ship with a static image/icon, CSS or JavaScript file. For this, we need to register static resources.

Registering a static resource directory

The easiest way to manage static resources is to make use of the static resource directory feature in five.grok. Simply add a directory called static in the package and make sure that the <grok:grok package="." /> line appears in configure.zcml.

If a static resource directory in the example.conference package contains a file called conference.css, it will be accessible on a URL like http://<server>/site/++resource++example.conference/conference.css. The resource name is the same as the package name wherein the static directory appears.

If you need to register additional directories, you can do so using the <browser:resourceDirectory /> ZCML directive. This requires two attributes: name is the name that appears after the ++resource++ namespace; directory is a relative path to the directory containing resources.

Importing CSS and JavaScript files in templates

One common use of static resources is to add a static CSS or JavaScript file to a specific template. We can do this by filling the style_slot or javascript_slot in Plone's main_template in our own view template and using an appropriate resource link.

For example, we could add the following near the top of presenter_templates/view.pt:

<head>
    <metal:block fill-slot="style_slot">
        <link rel="stylesheet" type="text/css" 
            tal:define="navroot context/@@plone_portal_state/navigation_root_url"
            tal:attributes="href string:${navroot}/++resource++example.conference/conference.css"
            />
    </metal:block>
</head>

Always create the resource URL relative to the navigation root as shown here, so that the URL is the same for all content objects using this view. This allows for efficient resource caching.

Registering resources with Plone's resource registries

Sometimes it is more appropriate to register a stylesheet with Plone's portal_css registry (or a JavaScript file with portal_javascripts), rather than add the registration on a per-template basis. This ensures that the resource is available site-wide.

It may seem wasteful to include a resource that is not be used on all pages in the global registry. Remember, however, that portal_css and portal_javascripts will merge and compress resources, and set caching headers such that browsers and caching proxies can cache resources well. It is often more effective to have one slightly larger file that caches well, than to have a variable number of files that may need to be loaded at different times.

To add a static resource file, you can use the GenericSetup cssregistry.xml or jsregistry.xml import steps in the profiles/default directory. For example, an import step to add the conference.css file site-wide may involve a cssregistry.xml file that looks like this:

<?xml version="1.0"?>
<object name="portal_css">
 <stylesheet id="++resource++example.conference/conference.css"
    title="" cacheable="True" compression="safe" cookable="True"
    enabled="1" expression="" media="screen" rel="stylesheet" rendering="import"
    />
</object>

Similarly, a JavaScript resource could be imported with a jsregistry.xml like:

<?xml version="1.0"?>
<object name="portal_javascripts">
 <javascript cacheable="True" compression="none" cookable="True"
    enabled="False" expression=""
    id="++resource++example.conference/conference.js" inline="False"/>
</object>

Image resources

Images can be added to resource directories just like any other type of resource. To use the image in a view, you can construct an <img /> tag like this:

        <img style="float: left; margin-right: 2px; margin-top: 2px"
             tal:define="navroot context/@@plone_portal_state/navigation_root_url"
             tal:attributes="src string:${navroot}/++resource++example.conference/program.gif"
             />

Content type icons

Finally, to use an image resource as the icon for a content type, simply list it in the FTI under the content_icon property. For example, in profiles/default/types/example.conference.presenter.xml, we can use the following line, presuming we have a presenter.gif in the static directory:

 <property name="content_icon">++resource++example.conference/presenter.gif</property>

Dexterity introduces the concept of behaviors – re-usable bundles of functionality and/or form fields which can be turned on or off on a per-type basis.

Each behavior has a unique interface. When a behavior is enabled on a type, you will be able to adapt that type to the behavior's interface. If the behavior is disabled, the adaptation will fail. The behavior interface can also be marked as an IFormFieldsProvider, in which case it will add fields to the standard add and edit forms. Finally, a behavior may imply a sub-type: a marker interface which will be dynamically provided by instances of the type for which the behavior is enabled.

We will not cover writing new behaviors in this manual, but we will show how to enable behaviors on a type. In fact, we've already seen one standard behavior applied to our example types, registered in the FTI and imported using GenericSetup:

 <property name="behaviors">
     <element value="plone.app.content.interfaces.INameFromTitle" />
 </property>

Other behaviors are added in the same way, by listing additional behavior interfaces as elements of the behaviors property.

Behaviors are normally registered with the <plone:behavior /> ZCML directive. When registered, a behavior will create a global utility providing IBehavior, which is used to provide some metadata, such as a title and description for the behavior. 

You can find and apply behaviors via the Dexterity Content Types control panel that is installed with plone.app.dexterity. For a list of standard behaviors that ship with Dexterity, see the reference at the end of this manual.

So far, we have mainly been concerned with content types' schemata and forms created from these. However, we often want to add more dynamic functionality, reacting when something happens to objects of our type. In Zope, that usually means writing event subscribers.

Zope's event model is synchronous. When an event is broadcast (via the notify() function from the zope.event package), for example from the save action of an add form, all registered event handlers will be called. There is no guarantee of which order the event handlers will be called in, however.

Each event is described by an interface, and will typically carry some information about the event. Some events are known as object events, and provide zope.component.interfaces.IObjectEvent. These have an object attribute giving access to the (content) object that the event relates to. Object events allow event handlers to be registered for a specific type of object as well as a specific type of event.

Some of the most commonly used event types in Plone are shown below. They are all object events.

• zope.lifecycleevent.interfaces.IObjectCreatedEvent, fired by the standard add form just after an object has been created, but before it has been added on the container. Note that it is often easier to write a handler for IObjectAddedEvent (see below), because at this point the object has a proper acquisition context.
• zope.lifecycleevent.interfaces.IObjectModifiedEvent, fired by the standard edit form when an object has been modified
• zope.app.container.interfaces.IObjectAddedEvent, fired when an object has been added to its container. The container is available as the newParent attribute, and the name the new item holds in the container is available as newName.
• zope.app.container.interfaces.IObjectRemovedEvent, fired when an object has been removed from its container. The container is available as the oldParent attribute, and the name the item held in the container is available as oldName.
• zope.app.container.interfaces.IObjectMovedEvent, fired when an object is added to, removed from, renamed in, or moved between containers. This event is a super-type of IObjectAddedEvent and IObjectRemovedEvent, shown above, so an event handler registered for this interface will be invoked for the 'added' and 'removed' cases as well. When an object is moved or renamed, all of oldParent, newParent, oldName and newName will be set.
• Products.CMFCore.interfaces.IActionSucceededEvent, fired when a workflow event has completed. The workflow attribute holds the workflow instance involved, and the action attribute holds the action (transition) invoked.

Event handlers can be registered using ZCML with the <subscriber /> directive, but when working with Dexterity types, we'll more commonly use the grok.subscriber() in Python code.

As an example, let's add an event handler to the Presenter type that tries to find users with matching names matching the presenter id, and send these users an email.

First, we require a few additional imports at the top of presenter.py:

from zope.app.container.interfaces import IObjectAddedEvent
from Products.CMFCore.utils import getToolByName

Then, we'll add the following event subscriber after the schema definition:

@grok.subscribe(IPresenter, IObjectAddedEvent)
def notifyUser(presenter, event):
    acl_users = getToolByName(presenter, 'acl_users')
    mail_host = getToolByName(presenter, 'MailHost')
    portal_url = getToolByName(presenter, 'portal_url')
    
    portal = portal_url.getPortalObject()
    sender = portal.getProperty('email_from_address')
    
    if not sender:
        return
    
    subject = "Is this you?"
    message = "A presenter called %s was added here %s" % (presenter.title, presenter.absolute_url(),)
    
    matching_users = acl_users.searchUsers(fullname=presenter.title)
    for user_info in matching_users:
        email = user_info.get('email', None)
        if email is not None:
            mail_host.secureSend(message, email, sender, subject)

There are many ways to improve this rather simplistic event handler, but it illustrates how events can be used. The first argument to grok.subscribe() is an interface describing the object type. For non-object events, this is omitted. The second arugment is the event type. The arguments to the function reflects these two, so the first argument is the IPresenter instance and the second is an IObjectAddedEvent instance.

Plone's security system is based the concept of permissions protecting operations (like accessing a view, viewing a field, modifying a field, or adding a type of content) that are granted to roles, which in turn are granted to users and/or groups. In the context of developing content types, permissions are typically used in three different ways:

• A content type or group of related content types often has a custom add permission which controls who can add this type of content.
• Views (including forms) are sometimes protected by custom permissions.
• Individual fields are sometimes protected by permissions, so that some users can view and edit fields that others can't see.

It is easy to create new permissions. However, be aware that it is considered good practice to use the standard permissions wherever possible and use workflow to control which roles are granted these permissions on a per-instance basis. We'll cover workflow later in this manual.

Standard permissions

The standard permissions can be found in Product.Five's permissions.zcml (parts/omelette/Products/Five/permissions.zcml). Here, you will find a short id (also known as the Zope 3 permission id) and a longer title (also known as the Zope 2 permission title). For historical reasons, some areas in Plone use the id, whilst others use the title. As a rule of thumb:

• Browser views defined in ZCML or protected via a grok.require() directive use the Zope 3 permission id
• Security checks using zope.security.checkPermission() use the Zope 3 permission id
• Dexterity's add_permission FTI variable uses the Zope 3 permission id.
• The rolemap.xml GenericSetup handler and workflows use the Zope 2 permission title.
• Security checks using AccessControl's getSecurityManager().checkPermission(), including the methods on the portal_membership tool, use the Zope 2 permission title.

The most commonly used permission are shown below. The Zope 2 permission title is shown in parentheses.

• zope2.View (View) is used to control access to the standard view of a content item
• zope2.DeleteObjects (Delete objects) is used to control the ability to delete child objects in a container
• cmf.ModifyPortalContent (Modify portal content) is used to control write access to content items
• cmf.ManagePortal (Manage portal) is used to control access to management screens
• cmf.AddPortalContent (Add portal content) is the standard add permission required to add content to a folder
• cmf.SetOwnProperties (Set own properties) is used to allow users to set their own member properties
• cmf.RequestReview (Request Review) is typically used as a workflow transition guard to allow users to submit content for review
• cmf.ReviewPortalContent (Review portal content) is usually granted to the Reviewer role, controlling the ability to publish or reject content

Standard roles

As with permissions, it is easy to create custom roles (use the rolemap.xml GenericSetup import step - see CMFPlone's version of this file for an example), although you should use the standard roles where possible.

The standard roles in Plone are:

• Anonymous, a pseudo-role that represents non-logged in users.

Note that if a permission is granted to Anonymous, it is effectively granted to everyone. It is not possible to grant permissions to non-logged in users without also granting them to logged in ones.

• Authenticated, a pseudo-role that represents logged-in users.
• Owner, which is automatically granted to the creator of an object.
• Manager, which represents super-users/administrators. Almost all permissions that are not granted to Anonymous are granted to Manager.
• Reviewer, which represents content reviewers separately from site administrators. It is possible to grant the Reviewer role locally on the Sharing tab, where it is shown as Can review.
• Member, representing "standard" Plone users

In addition, there are three roles that are intended to be used as local roles only. These are granted to specific users or groups via the Sharing tab, where they appear under more user friendly pseudonyms.

• Reader, aka Can view, confers the right to view content. As a role of thumb, the Reader role should have the View and Access contents information permissions if the Owner roles does.
• Editor, aka Can edit, confers the right to edit content. As a role of thumb, the Editor role should have the Modify portal content permission if the Owner roles does.
• Contributor, aka Can add, confers the right to add new content. As a role of thumb, the Contributor role should have the Add portal content permission and any type-specific add permissions globally (i.e. granted in rolemap.xml), although these permissions are sometimes managed in workflow as well.

Performing permission checks in code

It is sometimes necessary to check permissions explicitly in code, for example in a view. A permission check always checks a permission on a context object, since permissions can change with workflow.

Never make security dependent on users' roles directly. Always check for a permission, and assign the permission to the appropriate role or roles.

As an example, let's display a message on the view of a Session type if the user has the cmf.RequestReview permission. In session.py, we update the View class with the following:

from zope.security import checkPermission

class View(dexterity.DisplayForm):
    grok.context(ISession)
    grok.require('zope2.View')
    
    def canRequestReview(self):
        return checkPermission('cmf.RequestReview', self.context)

And in the session_templates/view.pt template, we add:

        <div class="discreet" tal:condition="view/canRequestReview" i18n:translate="suggest_review">
            Please submit this for review.
        </div>

Creating custom permissions

Although the standard permissions should be used to control basic operations (view, modify, delete, review), it is sometimes useful to create new permissions. Combined with custom workflows, custom permissions can be used to create highly tailored content review cycles and data entry applications. They are also an important way to control who can add what content.

The easiest way to create a custom permission is with the help of the collective.autopermission package, which allows permissions to be defined using the <permission /> ZCML statement.

collective.autopermission is obsolete in Zope 2.12, where its functionality has been merged into Zope itself

As an example, let's create some custom permissions for use with the Session type. We'll create a new add permission, so that we can let any member submit a session to a program, and a permission which we will later use to let reviewers edit some specific fields on the Session type.

First, we need to depend on collective.autopermission. In setup.py:

      install_requires=[
          ...
          'collective.autopermission',
      ],

Make sure collective.autopermission's configuration is included before any custom permissions are defined.  In our case, the <includeDependencies /> line takes care of this.

Next, we'll create a file called permissions.zcml to hold the permissions (we could also place them directly into configure.zcml). We need to include this in configure.zcml, just after the <includeDependencies /> line.

    <include file="permissions.zcml" />

Note: All permissions need to be defined before the <grok:grok package="." /> line in configure.zcml. Otherwise, you may get errors trying to use the permission with a grok.require() directive.

The permissions.zcml file looks like this:

<configure
    xmlns="http://namespaces.zope.org/zope"
    i18n_domain="example.conference">

    <permission
        id="example.conference.AddSession"
        title="example.conference: Add session"
        />

    <permission
        id="example.conference.ModifyTrack"
        title="example.conference: Modify track"
        />
        
</configure>

New permissions are granted to the Manager role only by default. To set a different default, we can use the rolemap.xml GenericSetup import step, which maps permissions to roles at the site root.

In profiles/default/rolemap.xml, we have the following:

<?xml version="1.0"?>
<rolemap>
  <permissions>
    <permission name="example.conference: Add session" acquire="True">
      <role name="Owner"/>
      <role name="Manager"/>
      <role name="Member"/>
      <role name="Contributor"/>
    </permission>
    <permission name="example.conference: Modify track" acquire="True">
      <role name="Manager"/>
      <role name="Reviewer"/>
    </permission>
  </permissions>
</rolemap>

This file uses the Zope 2 permission title instead of the shorter Zope 3 permission id

Content type add permissions 

Dexterity content types' add permissions are set in the FTI, using the add_permission property. This can be changed through the web or in the GenericSetup import step for the content type.

To make the Session type use our new permission, we modify the add_permission line in profiles/default/example.conference.session.xml:

 <property name="add_permission">example.conference.AddSession</property>

Protecting views and forms

Access to views and other browser resources (like viewlets or portlets) can be protected by permissions, either using the permission attribute on ZCML statements like <browser:page /> or using the grok.require() directive.

We have already seen this directive on our views:

class View(grok.View):
    grok.context(IPresenter)
    grok.require('zope2.View')

We could use a custom permission name as the argument to grok.require(). We could also use the special zope.Public permission name to make the view accessible to anyone.

Protecting form fields

Individual fields in a schema may be associated with a read permission and a write permission. The read permission is used to control access to the field's value via protected code (e.g. scripts or templates created through the web) and URL traversal, and can be used to control the appearance of fields when using display forms (if you use custom views that access the attribute directly, you'll need to perform your own checks). Write permissions can be used to control whether or not a given field appears on a type's add and edit forms.

In both cases, read and write permissions are annotated onto the schema using directives similar to those we've already seen for form widget hints. The read_permission() and write_permission() directives are found in the plone.directives.dexterity package.

As an example, let's add a field for Session reviewers to record the track for a session. We'll store the vocabulary of available tracks on the parent Program object in a text field, so that the creator of the Program can choose the available tracks.

First, we add this to the IProgram schema in program.py:

    form.widget(tracks=TextLinesFieldWidget)
    tracks = schema.List(
            title=_(u"Tracks"),
            required=True,
            default=[],
            value_type=schema.TextLine(),
        )

The TextLinesFieldWidget is used to edit a list of text lines in a text area. It is imported as:

from plone.z3cform.textlines.textlines import TextLinesFieldWidget

Next, we'll add a vocabulary for this to session.py:

from Acquisition import aq_inner, aq_parent
from zope.schema.interfaces import IContextSourceBinder
from zope.schema.vocabulary import SimpleVocabulary

...


@grok.provider(IContextSourceBinder)
def possibleTracks(context):
    
    # we put the import here to avoid a circular import
    from example.conference.program import IProgram
    while context is not None and not IProgram.providedBy(context):
        context = aq_parent(aq_inner(context))
    
    values = []
    if context is not None and context.tracks:
        values = context.tracks
    
    return SimpleVocabulary.fromValues(values)

This vocabulary finds the closest IProgram (in the add form, the context will be the Program, but on the edit form, it will be the Session, so we need to check the parent) and uses its tracks variable as the vocabulary.

Next, we add a field to the ISession interface in the same file and protect it with the relevant write permission:

    dexterity.write_permission(track='example.conference.ModifyTrack')
    track = schema.Choice(
            title=_(u"Track"),
            source=possibleTracks,
            required=False,
        )

The dexterity module is the root of the plone.directives.dexterity package, imported as:

from plone.directives import dexterity

With this in place, users with the example.conference: Modify track permission should be able to edit tracks for a session. For everyone else, the field will be hidden in the edit form.

Workflow is used in Plone for three distinct, but overlapping purposes:

• To keep track of metadata, chiefly an object's state
• To create content review cycles and model other types of processes
• To manage object security

When writing content types, we will often create custom workflows to go with them. In this section, we will explain at a high level how Plone's workflow system works (pardon the pun), and then show an example of a simple workflow to go with our example types. An exhaustive manual on using workflows is beyond the scope of this manual, but hopefully this will cover the basics.

There is nothing Dexterity-specific about this section. Everything here applies equally well to content objects created with Archetypes or using CMF directly.

A DCWorkflow refresher

What follows is a fairly detailed description of DCWorkflow, originally posted here. You may find some of this a little detailed on first reading, so feel free to skip to the specifics later on. However, is useful to be familiar with the high level concepts. You're unlikely to need multi-workflow chains in your first few attempts at workflow, for instance, but it's useful to know what it is if you come across the term.

Plone's workflow system is known as DCWorkflow. It is a states-and-transitions system, which means that your workflow starts in a particular state (the initial state) and then moves to other states via transitions (also called actions in CMF).

When an object enters a particular state (including the initial state), the workflow is given a chance to update permissions on the object. A workflow manages a number of permissions - typically the "core" CMF permissions like View, Modify portal content and so on - and will set those on the object at each state change. Note that this is event-driven, rather than a real-time security check: only by changing the state is the security information updated. This is why you need to click Update security settings at the bottom of the portal_workflow screen in the ZMI when you change your workflows' security settings and want to update existing objects.

A state can also assign local roles to groups. This is akin to assigning roles to groups on Plone's Sharing tab, but the mapping of roles to groups happens on each state change, much like the mapping of roles to permissions. Thus, you can say that in the pending_secondary state, members of the Secondary reviewers group has the Reviewer local role. This is powerful stuff when combined with the more usual role-to-permission mapping, although it is not very commonly used.

State changes result in a number of variables being recorded, such as the actor (the user that invoked the transition), the action (the name of the transition), the date and time and so on. The list of variables is dynamic, so each workflow can define any number of variables linked to TALES expressions that are invoked to calculate the current value at the point of transition. The workflow also keeps track of the current state of each object. The state is exposed as a special type of workflow variable called the state variable. Most workflows in Plone uses the name review_state as the state variable.

Workflow variables are recorded for each state change in the workflow history. This allows you to see when a transition occurred, who effected it, and what state the object was in before or after. In fact, the "current state" of the workflow is internally looked up as the most recent entry in the workflow history.

Workflow variables are also the basis for worklists. These are basically pre-defined catalog queries run against the current set of workflow variables. Plone's review portlet shows all current worklists from all installed workflows. This can be a bit slow, but it does meant that you can use a single portlet to display an amalgamated list of all items on all worklists that apply to the current user. Most Plone workflows have a single worklist that matches on the review_state variable, e.g. showing all items in the pending state.

If states are the static entities in the workflow system, transitions (actions) are provide the dynamic parts. Each state defines zero or more possible exit transitions, and each transition defines exactly one target state, though it is possible to mark a transition as "stay in current state". This can be useful if you want to do something in reaction to a transition and record that the transition happened in the workflow history, but not change the state (or security) of the object.

Transitions are controlled by one or more guards. These can be permissions (the preferred approach), roles (mostly useful for the Owner role - in other cases it is normally better to use permissions) or TALES expressions. A transition is available if all its guard conditions are true. A transition with no guard conditions is available to everyone (including anonymous!).

Transitions are user-triggered by default, but may be automatic. An automatic transition triggers immediately following another transition provided its guard conditions pass. It will not necessarily trigger as soon as the guard condition becomes true, as that would involve continually re-evaluating guards for all active workflows on all objects!

When a transition is triggered, the IBeforeTransitionEvent and IAfterTransitionEventevents are triggered. These are low-level events from Products.DCWorkflow that can tell you a lot about the previous and current states. There is a higher level IActionSucceededEvent in Products.CMFCore that is more commonly used to react after a workflow action has completed.

In addition to the events, you can configure workflow scripts. These are either created through-the-web or (more commonly) as External Methods, and may be set to execute before a transition is complete (i.e. before the object enters the target state) or just after it has been completed (the object is in the new state). Note that if you are using event handlers, you'll need to check the event object to find out which transition was invoked, since the events are fired on all transitions. The per-transition scripts are only called for the specific transitions for which they were configured.

Multi-chain workflows

Workflows are mapped to types via the portal_workflow tool. There is a default workflow, indicated by the string (Default). Some types have no workflow, which means that they hold no state information and typically inherit permissions from their parent. It is also possible for types to have multiple workflows. You can list multiple workflows by separating their names by commas. This is called a workflow chain.

Note that in Plone, the workflow chain of an object is looked up by multi-adapting the object and the workflow to the IWorkflowChain interface. The adapter factory should return a tuple of string workflow names (IWorkflowChain is a specialisation of IReadSequence, i.e. a tuple). The default obviously looks at the mappings in the portal_workflow tool, but it is possible to override the mapping, e.g. by using a custom adapter registered for some marker interface, which in turn could be provided by a type-specific behavior.

Multiple workflows applied in a single chain co-exist in time. Typically, you need each workflow in the chain to have a different state variable name. The standard portal_workflow API (in particular, doActionFor(), which is used to change the state of an object) also asumes the transition ids are unique. If you have two workflows in the chain and both currently have a submit action available, only the first workflow will be transitioned if you do portal_workflow.doActionFor(context, 'submit'). Plone will show all available transitions from all workflows in the current object's chain in the State drop-down, so you do not need to create any custom UI for this. However, Plone always assumes the state variable is called review_state (which is also the variable indexed in portal_catalog). Therefore, the state of a secondary workflow won't show up unless you build some custom UI.

In terms of security, remember that the role-to-permission (and group-to-local-role) mappings are event-driven and are set after each transition. If you have two concurrent workflows that manage the same permissions, the settings from the last transition invoked will apply. If they manage different permissions (or there is a partial overlap) then only the permissions managed by the most-recently-invoked workflow will change, leaving the settings for other permissions untouched.

Multiple workflows can be very useful in case you have concurrent processes. For example, an object may be published, but require translation. You can track the review state in the main workflow and the translation state in another. If you index the state variable for the second workflow in the catalog (the state variable is always available on the indexable object wrapper so you only need to add an index with the appropriate name to portal_catalog) you can search for all objects pending translation, for example using a Collection.

Creating a new workflow

With the theory out of the way, let's show how to create a new workflow.

Workflows are managed in the portal_workflow tool. You can use the ZMI to create new workflows and assign them to types. However, it is usually preferable to create an installable workflow configuration using GenericSetup. By default, each workflow as well as the workflow assignments are imported and exported using an XML syntax. This syntax is comprehensive, but rather verbose if you are writing it manually.

For the purposes of this manual, we will show an alternative configuration syntax based on spreadsheets (in CSV format). This is provided by the collective.wtf package. You can read more about the details of the syntax in its documentation. Here, we will only show how to use it to create a simple workflow for the Session type, allowing members to submit sessions for review.

To use collective.wtf, we need to depend on it. In setup.py, we have:

      install_requires=[
          ...
          'collective.wtf',
      ],

As before, the <includeDependencies /> line in configure.zcml takes care of configuring the package for us.

A workflow definition using collective.wtf consists of a CSV file in the profiles/default/workflow_csv directory, which we will create, and a workflows.xml file in profiles/default which maps types to workflows.

The workflow mapping in profiles/default/workflows.xml looks like this:

<?xml version="1.0"?>
<object name="portal_workflow">
    <bindings>
        <type type_id="example.conference.session">
            <bound-workflow workflow_id="example.conference.session_workflow"/>
        </type>
    </bindings>
</object>

The CSV file itself is found in profiles/default/workflow_csv/example.conference.session_workflow.csv. It contains the following, which was exported to CSV from an OpenOffice spreadsheet. You can find the original spreadsheet with the example.conference source code. This applies some useful formatting, which is obviously lost in the CSV version.

For your own workflows, you may want to use this template as a starting point.

"[Workflow]"
"Id:","example.conference.session_workflow"
"Title:","Conference session workflow"
"Description:","Allows members to submit session proposals for review"
"Initial state:","draft"

"[State]"
"Id:","draft"
"Title:","Draft"
"Description:","The proposal is being drafted."
"Transitions","submit"
"Permissions","Acquire","Anonymous","Authenticated","Member","Manager","Owner","Editor","Reader","Contributor","Reviewer"
"View","N",,,,"X","X","X","X",,
"Access contents information","N",,,,"X","X","X","X",,
"Modify portal content","N",,,,"X","X","X",,,


"[State]"
"Id:","pending"
"Title:","Pending"
"Description:","The proposal is pending review"
"Worklist:","Pending review"
"Worklist label:","Conference sessions pending review"
"Worklist guard permission:","Review portal content"
"Transitions:","reject, publish"
"Permissions","Acquire","Anonymous","Authenticated","Member","Manager","Owner","Editor","Reader","Contributor","Reviewer"
"View","N",,,,"X","X","X","X",,"X"
"Access contents information","N",,,,"X","X","X","X",,"X"
"Modify portal content","N",,,,"X","X","X",,,"X"

"[State]"
"Id:","published"
"Title:","Published"
"Description:","The proposal has been accepted"
"Transitions:","reject"
"Permissions","Acquire","Anonymous","Authenticated","Member","Manager","Owner","Editor","Reader","Contributor","Reviewer"
"View","Y","X",,,,,,,,
"Access contents information","Y","X",,,,,,,,
"Modify portal content","Y",,,,"X","X","X",,,

"[Transition]"
"Id:","submit"
"Title:","Submit"
"Description:","Submit the session for review"
"Target state:","pending"
"Guard permission:","Request review"

"[Transition]"
"Id:","reject"
"Title:","Reject"
"Description:","Reject the session from the program"
"Target state:","draft"
"Guard permission:","Review portal content"

"[Transition]"
"Id:","publish"
"Title:","Publish"
"Description:","Accept and publish the session proposal"
"Target state:","published"
"Guard permission:","Review portal content"

Here, you can see several states and transitions. Each state contains a role/permission map, and a list of the possible exit transitions. Each transition contains a target state and other meta-data such as a title and a description, as well as guard permissions.

Note that like most other GenericSetup import steps, the workflow uses the Zope 2 permission title when referring to permissions.

When the package is (re-)installed, this workflow should be available under portal_workflow and mapped to the Session type.

If you have existing instances, don't forget to go to portal_workflow in the ZMI and click Update security settings at the bottom of the page. This ensures that existing objects reflect the most recent security settings in the workflow.

A note about add permissions

This workflow assumes that regular members can add Session proposals to Programs, which are then reviewed. Previously, we granted the example.conference: Add session permission to the Member role. This is necessary, but not sufficient to allow memers to add sessions to programs. The user will also need the generic Add portal content permission in the Program folder.

There are two ways to achieve this:

• Build a workflow for the Program type that manages this permission
• Use the Sharing tab to grant Can add to the Authenticated Users group. This grants the Contributor local role to members. By default, this role is granted the Add portal content permission.

The ZODB is a hierarchical object store where objects of different schemata and sizes can live side by side. This is great for managing individual content items, but not optimal for searching across the content repository. A naive search would need to walk the entire object graph, loading each object into memory and comparing object metadata with search criteria. On a large site, this would quickly become prohibitive.

Luckily, Zope comes with a technology called the ZCatalog, which is basically a table structure optimised for searching. In Plone, there's a ZCatalog instance called portal_catalog. Standard event handlers will index content in the catalog when it is created or modified, and unindex when the content is removed.

The catalog manages indexes, which can be searched, and metadata (also known as columns), which are object attributes for which the value is copied into the catalog. When we perform a search, the result is a lazily loaded list of objects known as catalog brains. Catalog brains contain the value of metadata columns (but not indexes) as attributes. The functions getURL(), getPath() and getObject() can be used to get the URL and path of the indexed content item, and to load the full item into memory.

Dexterity objects are more lightweight than Archetypes objects. This means that loading objects into memory is not quite as undesirable as is sometimes assumed. If you're working with references, parent objects, or a small number of child objects, it is usually OK to load objects directly to work with them. However, if you are working with a large or unknown-but-potentially-large number of objects, you should consider using catalog searches to find them and use catalog metadata to store frequently used values. There is an important trade-off to be made between limiting object access and bloating the catalog with unneeded indexes and metadata, though. In particular, large strings (such as the body text of a document) or binary data (such as the contents of image or file fields) should not be stored as catalog metadata.

Plone comes with a number of standard indexes and metadata columns. These correspond to much of the Dublin Core set of metadata as well as several Plone-specific attributes. You can view the indexes, columns and the contents of the catalog through the ZMI pages of the portal_catalog tool. If you've never done this, it is probably instructive to have a look, both to understand how the indexes and columns may apply to your own content types, and to learn what searches are already possible.

Indexes come in various types. The most common ones are:

• FieldIndex, the most common type, used to index a single value.
• KeywordIndex, used to index lists of values where you want to be able to search for a subset of the values. As the name implies, commonly used for keyword fields, such as the Subject Dublin Core metadata field.
• DateIndex, used to index Zope 2 DateTime objects. Note that if your type uses a Python datetime object, you'll need to convert it to a Zope 2 DateTime using a custom indexer!
• DateRangeIndex, used mainly for the effective date range.
• ZCTextIndex, used mainly for the SearchableText index. This is the index used for full-text search.
• ExtendedPathIndex, a variant of PathIndex, which is used for the path index. This is used to search for content by path and optionally depth.

Adding new indexes and metadata columns

When an object is indexed, the catalog will by default attempt to find attributes and methods that match index and column names on the object. Methods will be called (with no arguments) in an attempt to get a value. If a value is found, it is indexed.

Objects are normally acquisition-wrapped when they are indexed, which means that an indexed value may be acquired from a parent. This can be confusing, especially if you are building container types and creating new indexes for them. If child objects don't have attributes/methods with the same name, the parent object's value will be indexed for all children as well.

Catalog indexes and metadata can be installed with the catalog.xml GenericSetup import step. It is useful to look at the one in Plone (parts/omelette/Products/CMFPlone/profiles/default/catalog.xml).

As an example, let's index the track property of a Session in the catalog, and add a metadata column for this property as well. In profiles/default/catalog.xml, we have:

<?xml version="1.0"?>
<object name="portal_catalog">
    <index name="track" meta_type="FieldIndex">
        <indexed_attr value="track"/>
    </index>
    <column value="track"/>
</object>

Notice how we specify both the index name and the indexed attribute. It is possible to use an index name (the key you use when searching) that is different to the indexed attribute, although they are usually the same. The metadata column is just the name of an attribute. 

Creating custom indexers

Indexing based on attributes can sometimes be limiting. First of all, the catalog is indiscriminate in that it attempts to index every attribute that's listed against an index or metadata column for every object. Secondly, it is not always feasible to add a method or attribute to a class just to calculate an indexed value.

Plone 3.3 and later ships with a package called plone.indexer to help make it easier to write custom indexers: components that are invoked to calculate the value the catalog sees when it tries to index a given attribute. Indexers can be used to index a different value to the one stored on the object, or to allow indexing of a "virtual" attribute that does not actually exist on the object is question. Indexers are usually registered on a per-type basis, so you can have different implementations for different types of content.

To illustrate indexers, we will add three indexers to program.py: Two will provide values for the start and end indexes, normally used by Plone's Event type. We actually have attributes with the correct name for these already, but they use Python datetime objects whereas the DateIndex requires a Zope 2 DateTime.DateTime object. (Python didn't have a datetime module when this part of Zope was created!) The third indexer will be used to provide a value for the Subject index that takes its value from the tracks list.

from DateTime import DateTime
from plone.indexer import indexer

...

@indexer(IProgram)
def startIndexer(obj):
    if obj.start is None:
        return None
    return DateTime(obj.start.isoformat())
grok.global_adapter(startIndexer, name="start")

@indexer(IProgram)
def endIndexer(obj):
    if obj.end is None:
        return None
    return DateTime(obj.end.isoformat())
grok.global_adapter(endIndexer, name="end")

@indexer(IProgram)
def tracksIndexer(obj):
    return obj.tracks
grok.global_adapter(tracksIndexer, name="Subject")

Here, we use the @indexer decorator to create an indexer. This doesn't register the indexer component, though, so we need to use grok.global_adapter() to finalise the registration. Crucially, this is where the indexer's name is defined. This is the name of the indexed attribute for which the indexer is providing a value.

Since all of these indexes are part of a standard Plone installation, we won't register them in catalog.xml. If you are creating custom indexers and need to add new catalog indexes or columns for them, remember that the "indexed attribute" name (and the column name) must match the name of the indexer as set in its adapter registration.

Searching using your indexes

Once we have registered our indexers and re-installed our product (to ensure that the catalog.xml import step is allowed to install new indexes in the catalog), we can use our new indexes just like we would any of the default indexes. The pattern is always the same:

from Products.CMFCore.utils import getToolByName
# get the tool
catalog = getToolByName(context, 'portal_catalog')
# execute a search
results = catalog(track='Track 1')
# examine the results
for brain in results:
    start = brain.start
    url = brain.getURL()
    obj = brain.getObject() # Performance hit!

This shows a simple search using the portal_catalog tool, which we look up from some context object. We call the tool to perform a search, passing search criteria as keyword arguments, where the left hand side refers to an installed index and the right hand side is the search term.

Some of the more commonly used indexes are:

• Title, the object's title.
• Description, the object's description.
• path, the object's path. The argument is a string like '/foo/bar'. To get the path of an object (e.g. a parent folder), do '/'.join(folder.getPhysicalPath()). Searching for an object's path will return the object and any children. To depth-limit the search, e.g. to get only those 1 level deep, use a compound query, e.g. path={'query': '/'.join(folder.getPhysicalPath()), 'depth': 1}. If a depth is specified, the object at the given path is not returned (but any children within the depth limit are).
• object_provides, used to match interfaces provided by the object. The argument is an interface name or list of interface names (of which any one may match). To get the name of a given interface, you can call ISomeInterface.__identifier__.
• portal_type, used to match the portal type. Note that users can rename portal types, so it is often better not to hardcode these. Often, using an object_provides search for a type-specific interface will be better. Conversely, if you are asking the user to select a particular type to search for, then they should be choosing from the currently installed portal_types.
• SearchableText, used for full-text searches. This supports operands like AND and OR in the search string.
• Creator, the username of the creator of a content item
• Subject, a KeywordIndex of object keywords
• review_state, an object's workflow state

In addition, the search results can be sorted based on any FieldIndex, KeywordIndex or DateIndex using the following keyword arguments:

• Use sort_on='<index name>' to sort on a particular index. For example, sort_on='sortable_title' will produce a sensible title-based sort. sort_on='Date' will sort on the publication date, or the creation date if this is not set.
• Add sort_order='reverse' to sort in reverse. The default is sort_order='ascending'. 'descending' can be used as an alias for 'reverse'.
• Add sort_limit=10 to limit to approximately 10 search results. Note that it is possible to get more results due to index optimisations. Use a list slice on the catalog search results to be absolutely sure that you have got the maximum number of results, e.g. results = catalog(..., sort_limit=10)[:10]. Also note that the use of sort_limit requires a sort_on as well.

Some of the more commonly used metadata columns are:

• Creator, the user who created the content object
• Date, the publication date or creation date, whichever is later
• Title, the object's title
• Description, the object's description
• getId, the object's id (note that this is an attribute, not a function)
• review_state, the object's workflow state
• portal_type, the object's portal type

For more information about catalog indexes and searching, see the ZCatalog chapter in the Zope 2 book.

Until now, we have used Dexterity's default content add and edit forms, supplying form hints in our schemata to influence how the forms are built. For most types, that is all that's ever needed. In some cases, however, we want to build custom forms, or supply additional forms.

Dexterity uses the z3c.form library to build its forms, via the plone.z3cform integration package.

Note that the plone.z3cform package requires that standard z3c.form forms are used via a form wrapper view. In Dexterity, this wrapper is normally applied automatically by the form grokkers in plone.directives.form and plone.directives.dexterity.

Dexterity also relies on plone.autoform, in particular its AutoExtensibleForm base class, which is responsible for processing form hints and setting up z3c.form widgets and groups (fieldsets). A custom form, therefore, is simply a view that uses these libraries, although Dexterity provides some helpful base classes that make it easier to construct forms based on the schema and behaviors of a Dexterity type.

If you want to build standalone forms not related to content objects, see the z3c.form documentation. For convenience, you may want to use the base classes and schema support in plone.directives.form.

Edit forms

An edit form is just a form that is registered for a particular type of content and knows how to register its fields. If the form is named edit, it will replace the default edit form, which is registered with that name for the more general IDexterityContent interface.

Dexterity provides a standard edit form base class that provides sensible defaults for buttons, labels and so on. This should be registered for a type schema (not a class). To create an edit form that is identical to the default, we could do:

class EditForm(dexterity.EditForm):
    grok.context(IFSPage)

The dexterity module is plone.directives.dexterity and the grok module is five.grok.

The default name for the form is edit, but we could supply a different name using grok.name(). The default permission is cmf.ModifyPortalContent, but we could require a different permission with grok.require(). We could also register the form for a particular browser layer, using grok.layer().

This form is of course not terribly interesting, since it is identical to the default. However, we can now start changing fields and values. For example, we could:

• Override the schema property to tell plone.autoform to use a different schema interface (with different form hints) than the content type schema
• Override the additional_schemata property to tell plone.autoform to use different supplemental schema interfaces. The default is to use all behavior interfaces that provide the IFormFieldProvider marker from plone.directives.form.
• Override the label and description properties to provide different a different title and description for the form.
• Set the z3c.formfields and groups attributes directly.
• Override the updateWidgets() method to modify widget properties, or one of the other update*() methods, to perform additional processing on the fields. In most cases, these require us to call the super version at the beginning. See the plone.autoform and z3c.form documentation to learn more about the sequence of calls that eminate from the form update() method in the z3c.form.form.BaseForm class.

Content add sequence

Add forms are similar to edit forms in that they are built from a type's schema and the schemata of its behaviors. However, for an add form to be able to construct a content object, it needs to know the portal_type to use.

You should realise that the FTIs in the portal_types tool can be modified through the web. It is even possible to create new types through the web that re-use existing classes and factories.

For this reason, add forms are looked up via a namespace traversal adapter alled ++add++. You may have noticed this in the URLs to add forms already. What actually happens is this:

• Plone renders the add menu.
• To do so, it looks, among other places, for actions in the folder/add category. This category is provided by the portal_types tool.
• The folder/add action category is constructed by looking up the add_view_expr property on the FTIs of all addable types. This is a TALES expression telling the add menu which URL to use.
• The default add_view_expr in Dexterity (and CMF 2.2) is string:${folder_url}/++add++${fti/getId}. That is, it uses the ++add++ traversal namespace with an argument containing the FTI name.
• A user clicks on an entry in the menu as is taken to a URL like /path/to/folder/++add++my.type.
• The ++add++ namespace adapter looks up the FTI with the given name, and gets its factory property.
• The factory property of an FTI gives the name of a particular zope.component.interfaces.IFactory utility, which is used later to construct an instance of the content object. Dexterity automatically registers a factory instance for each type, with a name that matches the type name, although it is possible to use an existing factory name in a new type. This allows administrators to create new "logical" types that are functionally identical to an existing type.
• The ++add++ namespace adapter looks up the actual form to render as a multi-adapter from (context, request, fti) to Interface with a name matching the factory property. Recall that a standard view is a multi-adapter from (context, request) to Interface with a name matching the URL segment for which the view is looked up. As such, add forms are not standard views, because they get the additional fti parameter when constructed.
• If this fails, there is no custom add form for this factory (as is normally the case). The fallback is an unnamed adapter from (context, request, fti). The default Dexterity add form is registered as such an adapter, specific to the IDexterityFTI interface.
• The form is rendered like any other z3c.form form instance, and is subject to validation, which may cause it to be loaded several times.
• Eventually, the is successfully submitted. At this point: 
• The standard AddForm base class will look up the factory from the FTI reference it holds and call it to create an instance.
• The default Dexterity factory looks at the klass attribute of the FTI (class is a reserved word in Python...) to determine the actual content class to use, creates an object and initialises it.
• The portal_type attribute of the newly created instance is set to the name of the FTI. Thus, if the FTI is a "logical type" created through the web, but using an existing factory, the new instance's portal_type will be set to the "logical type".
• The object is initialised with the values submitted in the form
• An IObjectCreatedEvent is fired
• The object is added to its container
• The user is redirected to the view specified in the immediate_view property of the FTI

This sequence is pretty long, but thankfully we rarely have to worry about it. In most cases, we can use the default add form, and when we can't, creating a custom add form is no more difficult than creating a custom edit form. The add form grokker take care of registering the add view appropriately.

Custom add forms

As with edit forms, Dexterity provides a sensible base class for add forms that knows how to deal with the Dexterity FTI and factory.

A custom form replicating the default would look like this:

class AddForm(dexterity.AddForm):
    grok.name('example.fspage')

The name here should match the factory name. By default, Dexterity types have a factory called the same as the FTI name. If no such factory exists (i.e. you have not registered a custom IFactory utility), a local factory utility will be created and managed by Dexterity when the FTI is installed.

Also note that we do not specify a context here. Add forms are always registered for any IFolderish context. We can specify a layer with grok.layer() and a permission other than the default cmf.AddPortalContent with grok.require().

If the permission used for the add form is different to the add_permission set in the FTI, the user needs to have both permissions to be able to see the form and add content. For this reason, most add forms will use the generic cmf.AddPortalContent permission. The add menu will not render links to types where the user does not have the add permission stated in the FTI, even if this is different to cmf.AddPortalContent.

As with edit forms, we can customise this form by overriding z3c.form and plone.autoform properties and methods. See the z3c.form documentation on add forms for more details.

When we learned about configuring the Dexterity FTI, we saw the klass attribute and how it could be used to refer to either the Container or Item content classes. These classes are defined in the plone.dexterity.content module, and represent container (folder) and item (non-folder) types, respectively.

For most applications, these two classes will suffice. We will normally use behaviors, adapters, event handlers and schema interfaces to build additional functionality for our types. In some cases, however, it is useful or necessary to override the class, typically to override some method or property provided by the base class that cannot be implemented with an adapter override. A custom class may also be able to provide marginally better performance by side-stepping some of the schema-dependent dynamic behavior found in the base classes. In real life, you are very unlikely to notice, though.

Creating a custom class is simple: simply derive form one of the standard ones, e.g.:

from plone.dexterity.content import Item

class MyItem(Item):
    """A custom content class"""
    ...

For a container type, we'd do:

from plone.dexterity.content import Container

class MyContainer(Container):
    """A custom content class"""
    ...

You can now add any required attributes or methods to this class.

To make use of this class, set the klass attribute in the FTI to its dotted name, e.g.

    <property name="klass">my.package.myitem.MyItem</property>

This will cause the standard Dexterity factory to instantiate this class when the user submits the add form.

As an alternative to setting klass in the FTI, you amy provide your own IFactory utility for this type in lieu of Dexterity's default factory (see plone.dexterity.factory). However, you need to be careful that this factory performs all necessary initialisation, so it is normally better to use the standard factory.

Custom class caveats

There are a few important caveats when working with custom content classes:

• Make sure you use the correct base class: either plone.dexterity.content.Item or plone.dexterity.content.Container.
• If you mix in other base classes, it is safer to put the Item or Container class first. If another class comes first, it may override the __name__, __providedBy__,  __allow_access_to_unprotected_subobjects__ and/or isPrincipiaFolderish properties, and possibly the __getattr__() and __getitem__() methods, causing problems with the dynamic schemata and/or folder item security. In all cases, you may need to explicitly set these attributes to the ones from the correct base class.
• If you define a custom constructor, make sure it can be called with no arguments, and with an optional id argument giving the name.

Zope supports WebDAV, a protocol that allows content objects to be viewed, modified, copied, renamed, moved and deleted as if they were files on the filesystem. WebDAV is also used to support saving to remote locations from various desktop programs. In addition, WebDAV powers the External Editor product, which allows users to launch a desktop program from within Plone to edit a content object.

To configure a WebDAV server, you can add the following option to the [instance] section of your buildout.cfg and re-run buildout.

webdav-address = 9800

See the documentation for plone.recipe.zope2instance for details. When Zope is started, you should now be able to mount it as a WebDAV server on the given port.

Most operating systems support mounting WebDAV servers as folders. Unfortunately, not all WebDAV implementations are very good. Dexterity content should work with Windows Web Folders (open Internet Explorer, go to File | Open, type in a WebDAV address, e.g. http://localhost:9800, and then select "Open as web folder" before hitting OK) and well-behaved clients such as Novell NetDrive. 

On Mac OS X, the Finder claims to support WebDAV, but the implementation is so flakey that it is just as likely to crash Mac OS X as it is to let you browse files and folders. Use a dedicated WebDAV client instead, such as Cyberduck.

Default WebDAV behaviour

By default, Dexterity content can be downloaded and uploaded using a text format based on RFC (2)822, the same standard used to encode email messages. Most fields are encoded in headers, whilst the field marked as "primary" will be contained in the body of the message. If there is more than one primary field, a multi-part message is created.

A field can be marked as "primary" using the primary() directive from plone.directives.form. For example:

class ISession(form.Schema):
    """A conference session. Sessions are managed inside Programs.
    """
    
    title = schema.TextLine(
            title=_(u"Title"),
            description=_(u"Session title"),
        )
    
    description = schema.Text(
            title=_(u"Session summary"),
        )
    
    form.primary('details')
    details = RichText(
            title=_(u"Session details"),
            required=False
        )

    form.widget(presenter=AutocompleteFieldWidget)
    presenter = RelationChoice(
            title=_(u"Presenter"),
            source=ObjPathSourceBinder(object_provides=IPresenter.__identifier__),
            required=False,
        )
    
    dexterity.write_permission(track='example.conference.ModifyTrack')
    track = schema.Choice(
            title=_(u"Track"),
            source=possibleTracks,
            required=False,
        )

This will actually apply the IPrimaryField marker interface from the plone.rfc822 package to the given field(s).

A WebDAV download of this content item will by default look like this:

title: Test session
description: First session
presenter: 713399904
track: Administrators
MIME-Version: 1.0
Content-Type: text/html; charset="utf-8"
Portal-Type: example.conference.session

<p>Details <b>here</b></p>

Notice how most fields are encoded as header strings. The presenter relation field stores a number, which is the integer id of the target object. Note that this id is generated when the content object is created, and so is unlikely to be valid on a different site. The details field, which we marked as primary, is encoded in the body of the message.

It is also possible to upload such a file to create a new session. In order to do that, the content_type_registry tool needs to be configured with a predicate that can detect the type of content from the uploaded file and instantiate the correct type of object. Such predicates could be based on an extension or a filename pattern. Below, we will see a different approach that uses a custom "file factory" for the containing Program type.

Containers

Container objects will be shown as collections (WebDAV-speak for folders) for WebDAV purposes. This allows the WebDAV client to open the container and list its contents. However, representing containers as collections makes it impossible to access the data contained in the various fields of the content object.

To allow access to this information, a pseudo-file called _data will be exposed inside a Dexterity container. This file can be read and written like any other, to access or modify the container's data. It cannot be copied, moved, renamed or deleted: those operations should be performed on the container itself.

Customising WebDAV behaviour

There are several ways in which you can influence the WebDAV behaviour of your type.

• If you are happy with the RFC 2822 format, you can provide your own plone.rfc822.interfaces.IFieldMarshaler adapters to provide alternate serialisations and parsers for fields. See the plone.rfc822 documentation for details.
• If you want to use a different file representation, you can provide your own IRawReadFile and IRawWriteFile adapters. For example, if you have a content object that stores binary data, you could return this data directly, with an appropriate MIME type, to allow it to be edited in a desktop program (e.g. an image editor if the MIME type is image/jpeg). The file plone.dexterity.filerepresentation contains two base classes, ReadFileBase and WriteFileBase, which you may be able to use to make it easier to implement these interfaces.
• If you want to control how content objects are created when a new file or directory is dropped into a particular type of container, you can provide your own IFileFactory or IDirectoryFactory adapters. See plone.dexterity.filerepresentation for the default implementations.

As an example, let's register a custom IFileFactory adapter for the IProgram type. This adapter will not rely on the content_type_registry tool to determine which type to construct, but will instead create a Session object, since that is the only type that is allowed inside a Program container.

The code, in program.py, looks like this:

from five import grok

...

from zope.component import createObject
from zope.event import notify
from zope.lifecycleevent import ObjectCreatedEvent
from zope.filerepresentation.interfaces import IFileFactory

...

class ProgramFileFactory(grok.Adapter):
    """Custom file factory for programs, which always creates a Session.
    """
    
    grok.implements(IFileFactory)
    grok.context(IProgram)
    
    def __call__(self, name, contentType, data):
        session = createObject('example.conference.session')
        notify(ObjectCreatedEvent(session))
        return session

This adapter overrides the DefaultFileFactory found in plone.dexterity.filerepresentation. It creates an object of the designated type, fires an IObjectModifiedEvent and then returns the object, which will then be populated with data from the uploaded file.

To test this, you could write a text file like the one shown above in a text editor and save it on your desktop, then drag it into the folder in your WebDAV client representing a Program.

Here is a simple automated integration test for the same component:

    def test_file_factory(self):
        self.folder.invokeFactory('example.conference.program', 'p1')
        p1 = self.folder['p1']
        fileFactory = IFileFactory(p1)
        newObject = fileFactory('new-session', 'text/plain', 'dummy')
        self.failUnless(ISession.providedBy(newObject))

How it all works

The rest of this section describes in some detail how the various WebDAV related components interact in Zope 2, CMF and Dexterity. This may be helpful if you are trying to customise or debug WebDAV behaviour.

Background

Basic WebDAV support can be found in the webdav package. This defines two base classes, webdav.Resource.Resource and webdav.Collection.Collection. Collection extends Resource. These are mixed into item and container content objects, respectively.

The webdav package also defines the NullResource object. A NullResource is a kind of placeholder, which supports the HTTP verbs HEAD, PUT, and MKCOL.

Contains based on ObjectManager (including those in Dexterity) will return a NullResource if they cannot find the requested object and the request is a WebDAV request.

The zope.filerepresentation package defines a number of interfaces which are intended to help manage file representations of content objects. Dexterity uses these interfaces to allow the exact file read and write operations to be overridden without subclassing.

HEAD

A HEAD request retrieves headers only.

Resource.HEAD() sets Content-Type based on self.content_type(), Content-Length based on self.get_size(), Last-Modified based on self._p_mtime, and an ETag based on self.http__etag(), if available.

Collection.HEAD() looks for self.index_html.HEAD() and returns its value if that exists. Otherwise, it returns a 405 Method Not Allowed response. If there is no index_html object, it returns 404 Not Found.

GET

A GET request retrieves headers and body.

Zope calls manage_DAVget() to retrieve the body. The default implementation calls manage_FTPget().

In Dexterity, manage_FTPget() adapts self to IRawReadFile and uses its mimeType and encoding properties to set the Content-Type header, and its size() method to set Content-Length.

If the IRawReadFile adapter is also an IStreamIterator, it will be returned for the publisher to consume directly. This provides for efficient serving of large files, although it does require that the file can be read in its entirety with the ZODB connection closed. Dexterity solves this problem by writing the file content to a temporary file on the server.

If the IRawReadFile adapter is not a stream iterator, its contents are returned as a string, by calling its read() method. Note that this loads the entire file contents into memory on the server.

The default IRawReadFile implementation for Dexterity content returns an RFC 2822 style message document. Most fields on the object and any enabled behaviours will be turned into UTF-8 encoded headers. The primary field, if any, will be returned in the body, also most likely encoded as an UTF-8 encoded string. Binary data may be base64 encoded instead.

A type which wishes to override this behaviour can provide its own adapter. For example, an image type could return the raw image data.

PUT

A PUT request reads the body of a request and uses it to update a resource that already exists, or to create a new object.

By default Resource.PUT() fails with 405 Method Not Allowed. That is, it is not by default possible to PUT to a resource that already exists. The same is true of Collection.PUT().

In Dexterity, the PUT() method is overridden to adapt self to zope.filerepresentation.IRawWriteFile, and call its write() method one or more times, writing the contents of the request body, before calling close(). The mimeType and encoding properties will also be set based on the value of the Content-Type header, if available.

The default implementation of IRawWriteFile for Dexterity objects assumes the input is an RFC 2822 style message document. It will read header values and use them to set fields on the object or in behaviours, and similarly read the body and update the corresponding primary field.

NullResource.PUT() is responsible for creating a new content object and initialising it (recall that a NullResource may be returned if a WebDAV request attempts to traverse to an object which does not exist). It sniffs the content type and body from the request, and then looks for the PUT_factory() method on the parent folder.

In Dexterity, PUT_factory() is implemented to look up an IFileFactory adapter on self and use it to create the empty file. The default implementation will use the content_type_registry tool to determine a type name for the request (e.g. based on its extension or MIME type), and then construct an instance of that type.

Once an instance has been constructed, the object will be initialised by calling its PUT() method, as above.

Note that when content is created via WebDAV, an IObjectCreatedEvent will be fired from the IFileFactory adapter, just after the object has been constructed. At this point, none of its values will be set. Subsequently, at the end of the PUT() method, an IObjectModifiedEvent will be fired. This differs from the event sequence of an object created through the web. Here, only an IObjectCreatedEvent is fired, and only after the object has been fully initialised.

DELETE

A DELETE request instructs the WebDAV server to delete a resource.

Resource.DELETE() calls manage_delObjects() on the parent folder to delete an object.

Collection.DELETE() does the same, but checks for write locks of all children of the collection, recursively, before allowing the delete.

PROPFIND

A PROPFIND request returns all or a set of WebDAV properties. WebDAV properties are metadata used to describe an object, such as the last modified time or the author.

Resource.PROPFIND() parses the request and then looks for a propertysheets attribute on self.

If an 'allprop' request is received, it calls dav__allprop(), if available, on each property sheet. This method returns a list of name/value pairs in the correct WebDAV XML encoding, plus a status.

If a 'propnames' request is received, it calls dav__propnames(), if available, on each property sheet. This method returns a list of property names in the correct WebDAV XML encoding, plus a status.

If a 'propstat' request is received, it calls dav__propstats(), if available, on each property sheet, for each requested property. This method returns a property name/value pair in the correct WebDAV XML encoding, plus a status.

The PropertyManager mixin class defines the propertysheets variable to be an instance of DefaultPropertySheets. This in turn has two property sheets, default, a DefaultProperties instance, and webdav, a DAVProperties instance.

The DefaultProperties instance contains the main property sheet. This typically has a title property, for example.

DAVProperties will provides various core WebDAV properties. It defines a number of read-only properties: creationdate, displayname, resourcetype, getcontenttype, getcontentlength, source, supportedlock, and lockdiscovery. These in turn are delegated to methods prefixed with dav__, so e.g. reading the creationdate property calls dav__creationdate() on the property sheet instance. These methods in turn return values based on the the property manager instance (i.e. the content object). In particular:

• creationdate returns a fixed date (January 1st, 1970).
• displayname returns the value of the title_or_id() method
• resourcetype returns an empty string or <n:collection/>
• getlastmodified returns the ZODB modification time
• getcontenttype delegates to the content_type() method, falling back on the default_content_type() method. In Dexterity, content_type() is implemented to look up the IRawReadFile adapter on the context and return the value of its mimeType property.
• getcontentlength delegates to the get_size() method (which is also used for the "size" column in Plone folder listings). In Dexterity, this looks up a zope.size.interfaces.ISized adapter on the object and calls sizeForSorting(). If this returns a unit of 'bytes', the value portion is used. Otherwise, a size of 0 is returned.
• source returns a link to /document_src, if that attribute exists
• supportedlock indicates whether IWriteLock is supported by the content item
• lockdiscovery returns information about any active locks

Other properties in this and any other property sheets are returned as stored when requested.

If the PROPFIND request specifies a depth of 1 or infinity (i.e. the client wants properties for items in a collection), the process is repeated for all items returned by the listDAVObjects() methods, which by default returns all contained items via the objectValues() method.

PROPPATCH

A PROPPATCH request is used to update the properties on an existing object.

Resource.PROPPATCH() deals with the same types of properties from property sheets as PROPFIND(). It uses the PropertySheet API to add or update properties as appropriate.

MKCOL

A MKCOL request is used to create a new collection resource, i.e. create a new folder.

Resource.MKCOL() raises 405 Method Not Allowed, because the resource already exists (remember that in WebDAV, the MKCOL request, like a PUT for a new resource, is sent with a location that specifies the desired new resource location, not the location of the parent object).

NullResource.MKCOL() handles the valid case where a MKCOL request has been sent to a new resource. After checking that the resource does not already exist, that the parent is indeed a collection (folderish item), and that the parent is not locked, it calls the MKCOL_handler() method on the parent folder.

In Dexterity, MKCOL()_handler is overridden to adapt self to an IDirectoryFactory from zope.filerepresentation and use this to create a directory. The default implementation simply calls manage_addFolder() on the parent. This will create an instance of the Folder type.

COPY

A COPY request is used to copy a resource.

Resource.COPY() implements this operation using the standard Zope content object copy semantics.

MOVE

A MOVE request is used to relocate or rename a resource.

Resource.MOVE() implements this operation using the standard Zope content object move semantics.

LOCK

A LOCK request is used to lock a content object.

All relevant WebDAV methods in the webdav package are lock aware. That is, they check for locks before attempting any operation that would violate a lock.

Also note that plone.locking uses the lock implementation from the webdav package by default.

Resource.LOCK() implements locking and lock refresh support.

NullResource.LOCK() implements locking on a NullResource. In effect, this means locking the name of the non-existent resource. When a NullResource is locked, it is temporarily turned into a LockNullResource object, which is a persistent object set onto the parent (remember that a NullResource is a transient object returned when a child object cannot be found in a WebDAV request).

UNLOCK

An UNLOCK request is used to unlock a locked object.

Resource.UNLOCK() handles unlock requests.

LockNullResource.UNLOCK() handles unlocking of a LockNullResource. This deletes the LockNullResource object from the parent container.

Fields on container objects

When browsing content via WebDAV, a container object (folderish item) will appear as a folder. Most likely, this object will also have content in the form of schema fields. To make this accessible, Dexterity containers expose a pseudo-file with the name '_data', by injecting this into the return value of listDAVObjects() and adding a special traversal hook to allow its contents to be retrieved.

This file supports HEAD, GET, PUT, LOCK, UNLOCK, PROPFIND and PROPPATCH requests (an error will be raised if the user attempts to rename, copy, move or delete it). These operate on the container object, obviously. For example, when the data object is updated via a PUT request, the PUT() method on the container is called, by default delegating to an IRawWriteFile adapter on the container.

As all good developers know, automated tests are very important! If you are not comfortable with automated testing and test-driven development, you should read the Plone testing tutorial. In this section, we will assume you are familiar with Plone testing basics, and show some tests that are particularly relevant to our example types.

Firstly, we will add a few unit tests. Recall that unit tests are simple tests for a particular function or method, and do not depend on an outside environment being set up. As a rule of thumb, if something can be tested with a simple unit test, do so, because:

• Unit tests are quick to write
• They are also quick to run
• Because they are more isolated, you are less likely to have tests that pass or fail due to incorrect assumptions or by luck
• You can usually test things more thoroughly and exhaustively with unit tests than with (slower) integration tests

You'll typically supplement a larger number of unit tests with a smaller number of integration tests, to ensure that your application's correctly wired up and working.

That's the theory, at least. When we're writing content types, we're often more interested in integration test, because a type schema and FTI are more like configuration of the Plone and Dexterity frameworks than imperative programming. We can't "unit test" the type's schema interface, but we can and should test that the correct schema is picked up and used when our type is installed. We will often write unit tests (with mock objects, where required) for custom event handlers, default value calculation functions and other procedural code.

In that spirit, let's write some unit tests for the default value handler and the invariant in program.py. We'll add the directory tests, with an __init__.py and a file test_program.py that looks like this:

import unittest
import datetime

from example.conference.program import startDefaultValue
from example.conference.program import endDefaultValue
from example.conference.program import IProgram
from example.conference.program import StartBeforeEnd

class MockProgram(object):
    pass

class TestProgramUnit(unittest.TestCase):
    """Unit test for the Program type
    """
    
    def test_start_defaults(self):
        data = MockProgram()
        default_value = startDefaultValue(data)
        today = datetime.datetime.today()
        delta = default_value - today
        self.assertEquals(6, delta.days)

    def test_end_default(self):
        data = MockProgram()
        default_value = endDefaultValue(data)
        today = datetime.datetime.today()
        delta = default_value - today
        self.assertEquals(9, delta.days)
    
    def test_validate_invariants_ok(self):
        data = MockProgram()
        data.start = datetime.datetime(2009, 1, 1)
        data.end = datetime.datetime(2009, 1, 2)
        
        try:
            IProgram.validateInvariants(data)
        except:
            self.fail()
    
    def test_validate_invariants_fail(self):
        data = MockProgram()
        data.start = datetime.datetime(2009, 1, 2)
        data.end = datetime.datetime(2009, 1, 1)
        
        try:
            IProgram.validateInvariants(data)
            self.fail()
        except StartBeforeEnd:
            pass
    
    def test_validate_invariants_edge(self):
        data = MockProgram()
        data.start = datetime.datetime(2009, 1, 2)
        data.end = datetime.datetime(2009, 1, 2)
        
        try:
            IProgram.validateInvariants(data)
        except:
            self.fail()

def test_suite():
    return unittest.defaultTestLoader.loadTestsFromName(__name__)

This is a simple test using the Python standard library's unittest module. There are a few things to note here:

• We have created a dummy class to simulate a Program instance. It doesn't contain anything at all, but we set some attributes onto it for certain tests. This is a very simple way to do mocks. There are much more sophisticated mock testing approaches, but starting simple is good.
• Each test is self contained. There is no test layer or test case setup/tear-down.
• We use the defaultTestLoader to load all test classes in the module automatically. The test runner will look for modules in the tests package with names starting with test* that have a test_suite() method to get test suites.

To run the tests, we can do:

$ ./bin/text example.conference

Hopefully it should show five passing tests.

This uses the testrunner configured via the [test] part in our buildout.cfg. This provides better test reporting and a few more advanced options (like output colouring). We could also use the built-in test runner in the instance script, e.g. with ./bin/instance test -s example.conference.

To run just this test suite, we can do:

$ ./bin/test example.conference -t TestProgramUnit

This is useful when we have other test suites that we don't want to run, e.g. because they are integration tests and require lengthy setup.

To get a report about test coverage, we can run:

$ ./bin/test example.conference --coverage

Test coverage reporting is important. If you have a module with low test coverage, it means that your tests do not cover many of the code paths in those modules, and so are less useful for detecting bugs or guarding against future problems. Aim for 100%.

We'll now add some integration tests for our type. These should ensure that the package installs cleanly, and that our custom types are addable in the right places and have the right schemata, at the very least.

To help manage test setup, we'll make use of the Zope test runner's concept of layers. Layers allow common test setup (such as configuring a Plone site and installing a product) to take place once and be re-used by multiple test cases. Those test cases can still modify the environment, but their changes will be torn down and the environment reset to the layer's initial state between each test, facilitating test isolation.

As the name implies, layers are, erm, layered. One layer can extend another. If two test cases in the same test run use two different layers with a common ancestral layer, the ancestral layer is only set up and torn down once.

We'll use collective.testcaselayer to write and manage layers. We need to depend on this, so in setup.py, we have:

      install_requires=[
          ...
          'collective.testcaselayer',
      ],

Don't forget to re-run buildout after making changes to setup.py.

We then add our own layer to tests/layer.py:

from Products.PloneTestCase import ptc
import collective.testcaselayer.ptc

ptc.setupPloneSite()

class IntegrationTestLayer(collective.testcaselayer.ptc.BasePTCLayer):

    def afterSetUp(self):
        # Install the example.conference product
        self.addProfile('example.conference:default')

Layer = IntegrationTestLayer([collective.testcaselayer.ptc.ptc_layer])

This extends a base layer that sets up Plone, and adds some custom layer setup for our package, in this case installing the example.conference extension profile. We could also perform additional setup here, such as creating some initial content or setting the default roles for the test run. See the collective.testcaselayer documentation for more details.

To use the layer, we can create a new test case based on PloneTestCase that uses our layer. We'll add one to test_program.py first. (In the code snippet below, the unit test we created previously has been removed to conserve space.)

import unittest

from zope.component import createObject
from zope.component import queryUtility

from plone.dexterity.interfaces import IDexterityFTI

from Products.PloneTestCase.ptc import PloneTestCase
from example.conference.tests.layer import Layer

from example.conference.program import IProgram

class TestProgramIntegration(PloneTestCase):
    
    layer = Layer
    
    def test_adding(self):
        self.folder.invokeFactory('example.conference.program', 'program1')
        p1 = self.folder['program1']
        self.failUnless(IProgram.providedBy(p1))
    
    def test_fti(self):
        fti = queryUtility(IDexterityFTI, name='example.conference.program')
        self.assertNotEquals(None, fti)
    
    def test_schema(self):
        fti = queryUtility(IDexterityFTI, name='example.conference.program')
        schema = fti.lookupSchema()
        self.assertEquals(IProgram, schema)
    
    def test_factory(self):
        fti = queryUtility(IDexterityFTI, name='example.conference.program')
        factory = fti.factory
        new_object = createObject(factory)
        self.failUnless(IProgram.providedBy(new_object))
    
    def test_view(self):
        self.folder.invokeFactory('example.conference.program', 'program1')
        p1 = self.folder['program1']
        view = p1.restrictedTraverse('@@view')
        sessions = view.sessions()
        self.assertEquals(0, len(sessions))
    
    def test_start_end_dates_indexed(self):
        self.folder.invokeFactory('example.conference.program', 'program1')
        p1 = self.folder['program1']
        p1.start = datetime.datetime(2009, 1, 1, 14, 01)
        p1.end = datetime.datetime(2009, 1, 2, 15, 02)
        p1.reindexObject()
        
        result = self.portal.portal_catalog(path='/'.join(p1.getPhysicalPath()))
        
        self.assertEquals(1, len(result))
        self.assertEquals(result[0].start, DateTime('2009-01-01T14:01:00'))
        self.assertEquals(result[0].end, DateTime('2009-01-02T15:02:00'))
    
    def test_tracks_indexed(self):
        self.folder.invokeFactory('example.conference.program', 'program1')
        p1 = self.folder['program1']
        p1.tracks = ['Track 1', 'Track 2']
        p1.reindexObject()
        
        result = self.portal.portal_catalog(Subject='Track 2')
        
        self.assertEquals(1, len(result))
        self.assertEquals(result[0].getURL(), p1.absolute_url())

def test_suite():
    return unittest.defaultTestLoader.loadTestsFromName(__name__)

This illustrates a basic set of tests that make sense for most content types. There are many more things we could test (for example, we could test the add permissions more thoroughly, and we ought to test the sessions() method on the view with some actual content!), but even this small set of integration tests tells us that our product has installed, that the content type is addable, that it has the right factory, and that instances of the type provide the right schema interface.

There are some important things to note about this test case:

• We extend PloneTestCase, which means we have access to a full Plone integration test environment. See the testing tutorial for more details.
• We set the layer attribute to our custom layer. This means that all tests in our test case will have the example.conference:default profile installed.
• We test that the content is addable (here, as a normal member in their member folder, since that is the default security context for the test - use self.setRoles(['Manager']) to get the Manager role and self.portal to access the portal root), that the FTI is installed and can be located, and that both the FTI and instances of the type know about the correct type schema.
• We also test that the view can be looked up and has the correct methods. We've not included a full functional test (e.g. using zope.testbrowser) or any other front-end testing here. If you require those, take a look at the testing tutorial.
• We also test that our custom indexers are working, by creating an appropriate object and searching for it again. Note that we need to reindex the object after we've modified it so that the catalog is up to date.
• The defaultTestLoader will find this test and load it, just as it found the TestProgramUnit test case.

To run our tests, we can still do.

$ ./bin/test example.conference

You should now notice layers being set up and torn down. Again, use the -t option to run a particular test case (or test method) only.

The other tests are similar. We have tests/test_session.py to test the Session type:

import unittest

from zope.component import createObject
from zope.component import queryUtility

from plone.dexterity.interfaces import IDexterityFTI

from Products.PloneTestCase.ptc import PloneTestCase
from example.conference.tests.layer import Layer

from example.conference.session import ISession
from example.conference.session import possible_tracks

class TestSessionIntegration(PloneTestCase):
    
    layer = Layer
    
    def test_adding(self):
        
        # We can't add this directly
        self.assertRaises(ValueError, self.folder.invokeFactory, 'example.conference.session', 'session1')
        
        self.folder.invokeFactory('example.conference.program', 'program1')
        p1 = self.folder['program1']
        
        p1.invokeFactory('example.conference.session', 'session1')
        s1 = p1['session1']
        self.failUnless(ISession.providedBy(s1))
    
    def test_fti(self):
        fti = queryUtility(IDexterityFTI, name='example.conference.session')
        self.assertNotEquals(None, fti)
    
    def test_schema(self):
        fti = queryUtility(IDexterityFTI, name='example.conference.session')
        schema = fti.lookupSchema()
        self.assertEquals(ISession, schema)
    
    def test_factory(self):
        fti = queryUtility(IDexterityFTI, name='example.conference.session')
        factory = fti.factory
        new_object = createObject(factory)
        self.failUnless(ISession.providedBy(new_object))
    
    def test_tracks_vocabulary(self):
        self.folder.invokeFactory('example.conference.program', 'program1')
        p1 = self.folder['program1']
        p1.tracks = ['T1', 'T2', 'T3']
        
        p1.invokeFactory('example.conference.session', 'session1')
        s1 = p1['session1']
        
        vocab = possible_tracks(s1)
        
        self.assertEquals(['T1', 'T2', 'T3'], [t.value for t in vocab])
        self.assertEquals(['T1', 'T2', 'T3'], [t.token for t in vocab])
    
    def test_catalog_index_metadata(self):
        self.failUnless('track' in self.portal.portal_catalog.indexes())
        self.failUnless('track' in self.portal.portal_catalog.schema())
    
    def test_workflow_installed(self):
        self.folder.invokeFactory('example.conference.program', 'program1')
        p1 = self.folder['program1']
        
        p1.invokeFactory('example.conference.session', 'session1')
        s1 = p1['session1']
        
        chain = self.portal.portal_workflow.getChainFor(s1)
        self.assertEquals(('example.conference.session_workflow',), chain)

def test_suite():
    return unittest.defaultTestLoader.loadTestsFromName(__name__)

Notice here how we test that the Session type cannot be added directly to a folder, and that it can be added inside a program. We also add a test for the possible_tracks() vocabulary method, as well as tests for the installation of the track index and metadata column and the custom workflow.

And in tests/test_presenter.py, we test the Presenter type:

import unittest

from zope.component import createObject
from zope.component import queryUtility

from plone.dexterity.interfaces import IDexterityFTI

from Products.PloneTestCase.ptc import PloneTestCase
from example.conference.tests.layer import Layer

from example.conference.presenter import IPresenter

class TestPresenterIntegration(PloneTestCase):
    
    layer = Layer
    
    def test_adding(self):
        self.folder.invokeFactory('example.conference.presenter', 'presenter1')
        p1 = self.folder['presenter1']
        self.failUnless(IPresenter.providedBy(p1))
    
    def test_fti(self):
        fti = queryUtility(IDexterityFTI, name='example.conference.presenter')
        self.assertNotEquals(None, fti)
    
    def test_schema(self):
        fti = queryUtility(IDexterityFTI, name='example.conference.presenter')
        schema = fti.lookupSchema()
        self.assertEquals(IPresenter, schema)
    
    def test_factory(self):
        fti = queryUtility(IDexterityFTI, name='example.conference.presenter')
        factory = fti.factory
        new_object = createObject(factory)
        self.failUnless(IPresenter.providedBy(new_object))
    
def test_suite():
    return unittest.defaultTestLoader.loadTestsFromName(__name__)

Faster tests with Roadrunner

You will have noticed that running unit tests was much quicker than running integration tests. That is unfortunate, but to be expected: the integration test setup basically requires starting all of Zope and configuring a Plone site.

Luckily, there is a tool that we can use to speed things up, and if you've been following along the tutorial, you already have it in your buildout: Roadrunner. This is a command that takes the place of ./bin/instance test that preloads the Zope environment and allows you to re-run tests much faster.

To run our tests with roadrunner, we would do:

$ ./bin/roadrunner -s example.conference

This runs the tests once, and then drops to the Roadrunner prompt:

rr>

Simply hitting enter here, or typing a command like test -s example.conference will re-run your tests, this time taking much less time.

Roadrunner works best when you are adding and debugging your tests. For example, it's a very quick way to get to a pdb prompt: just set a breakpoint in your test with import pdb; pdb.set_trace() and re-run it in roadrunner. You can then step into your test code and the code under test.

Roadrunner should pick up changes to your tests automatically. However, it may not pick up changes to your application code, grokked components or ZCML files. If it doesn't, you'll need to exit the Roadrunner prompt and restart.

Mock testing is a powerful approach to testing that lets you make assertions about how the code under test is interacting with other system modules. It is often useful when the code you want to test is performing operations that cannot be easily asserted by looking at its return value.

In our example product, we have an event handler like this:

@grok.subscribe(IPresenter, IObjectAddedEvent)
def notifyUser(presenter, event):
    acl_users = getToolByName(presenter, 'acl_users')
    mail_host = getToolByName(presenter, 'MailHost')
    portal_url = getToolByName(presenter, 'portal_url')
    
    portal = portal_url.getPortalObject()
    sender = portal.getProperty('email_from_address')
    
    if not sender:
        return
    
    subject = "Is this you?"
    message = "A presenter called %s was added here %s" % (presenter.title, presenter.absolute_url(),)
    
    matching_users = acl_users.searchUsers(fullname=presenter.title)
    for user_info in matching_users:
        email = user_info.get('email', None)
        if email is not None:
            mail_host.secureSend(message, email, sender, subject)

If we want to test that this sends the right kind of email message, we'll need to somehow inspect what is passed to secureSend(). The only way to do that is to replace the MailHost object that is acquired when getToolByName(presenter, 'MailHost') is called, with something that performs that assertion for us.

If we wanted to write an integration test, we could use PloneTestCase to execute this event handler, e.g. by firing the event manually, and temporarily replace the MailHost object in the root of the test case portal (self.portal) with a dummy that raised an exception if the wrong value was passed.

However, such integration tests can get pretty heavy handed, and sometimes it is difficult to ensure that it works in all cases. In the approach outlined above, for example, we would miss cases where no mail was sent at all.

Enter mock objects. A mock object is a "test double" that knows how and when it ought to be called. The typical approach is as follows:

• Create a mock object.
• The mock object starts out in "record" mode.
• Record the operations that you expect the code under test perform on the mock object. You can make assertions about the type and value of arguments, the sequence of calls, or the number of times a method is called or an attribute is retrieved or set.
• You can also give your mock objects behaviour, e.g. by specifying return values or exceptions to be raised in certain cases.
• Initialise the code under test and/or the environment it runs in so that it will use the mock object rather than the real object. Sometimes this involves temporarily "patching" the environment.
• Put the mock framework into "replay" mode.
• Run the code under test.
• Apply any assertions as you normally would.
• The mock framework will raise exceptions if the mock objects are called incorrectly (e.g. with the wrong arguments, or too many times) or insufficiently (e.g. an expected method was not called).

There are several Python mock object frameworks. Dexterity itself users a powerful one called mocker, via the plone.mocktestcase integration package. You are encouraged to read the documentation for those two packages to better understand how mock testing works, and what options are available.

Take a look at the tests in plone.dexterity if you're looking for more examples of mock tests using plone.mocktestcase.

To use the mock testing framework, we first need to depend on plone.mocktestcase. As usual, we add it to setup.py and re-run buildout.

      install_requires=[
          ...
          'plone.mocktestcase',
      ],

As an example test case, consider the following class in test_presenter.py:

import unittest

...

from plone.mocktestcase import MockTestCase
from zope.app.container.contained import ObjectAddedEvent
from example.conference.presenter import notifyUser

class TestPresenterUnit(MockTestCase):
    
    def test_notify_user(self):
        
        # dummy presenter
        presenter = self.create_dummy(
                __parent__=None,
                __name__=None,
                title="Jim",
                absolute_url = lambda: 'http://example.org/presenter',
            )
        
        # dummy event
        event = ObjectAddedEvent(presenter)
        
        # search result for acl_users
        user_info = [{'email': 'jim@example.org', 'id': 'jim'}]
        
        # email data
        message = "A presenter called Jim was added here http://example.org/presenter"
        email = "jim@example.org"
        sender = "test@example.org"
        subject = "Is this you?"
        
        # mock tools/portal
        
        portal_mock = self.mocker.mock()
        self.expect(portal_mock.getProperty('email_from_address')).result('test@example.org')
        
        portal_url_mock = self.mocker.mock()
        self.mock_tool(portal_url_mock, 'portal_url')
        self.expect(portal_url_mock.getPortalObject()).result(portal_mock)
        
        acl_users_mock = self.mocker.mock()
        self.mock_tool(acl_users_mock, 'acl_users')
        self.expect(acl_users_mock.searchUsers(fullname='Jim')).result(user_info)
        
        mail_host_mock = self.mocker.mock()
        self.mock_tool(mail_host_mock, 'MailHost')
        self.expect(mail_host_mock.secureSend(message, email, sender, subject))
        
        
        # put mock framework into replay mode
        self.replay()
        
        # call the method under test
        notifyUser(presenter, event)

        # we could make additional assertions here, e.g. if the function
        # returned something. The mock framework will verify the assertions
        # about expected call sequences.

...

def test_suite():
    return unittest.defaultTestLoader.loadTestsFromName(__name__)

Note that the other tests in this module have been removed for the sake of brevity.

If you are not familiar with mock testing, it may take a bit of time to get your head around what's going on here. Let's run though the test:

• First, we create a dummy presenter object. This is not a mock object, it's just a class with the required minimum set of attributes, created using the create_dummy() helper method from the MockTestCase base class. We use this type of dummy because we are not interested in making any assertions on the presenter object: it is used as an "input" only.
• Next, we create a dummy event. Here we have opted to use a standard implementation from zope.app.container.
• We then define a few variables that we will use in the various assertions and mock return values: the user data that will form our dummy user search results, and the email data passed to the mail host.
• Next, we create mocks for each of the tools that our code needs to look up. For each, we use the expect() method from MockTestCase to make some assertions. For example, we expect that getPortalObject() will be called (once) on the portal_url tool, and it should return another mock object, the portal_mock. On this, we expect that getProperty() is called with an argument equal to "email_from_address". The mock will then return "test@example.org". Take a look at the mocker and plone.mocktestcase documentation to see the various other types of assertions you can make.
• The most important mock assertion is the line self.expect(mail_host_mock.secureSend(message, email, sender, subject)). This asserts that the secureSend() method gets called with the required message, recipient address, sender address and subject, exactly once.
• We then put the mock into replay mode, using self.replay(). Up until this point, any calls on our mock objects have been to record expectations and specify behaviour. From now on, any call will count towards verifying those expectations.
• Finally, we call the code under test with our dummy presenter and event.
• In this case, we don't have any "normal" assertions, although the usual unit test assertion methods are all available if you need them, e.g. to test the return value of the method under test. The assertions in this case are all coming from the mock objects. The tearDown() method of the MockTestCase class will in fact check that all the various methods were called exactly as expected.

To run these tests, use the normal test runner, e.g.:

$ ./bin/test example.conference -t TestPresenterMock

Note that mock tests are typically as fast as unit tests, so there is typically no need for something like roadrunner.

Mock testing caveats

Mock testing is a somewhat controversial topic. On the one hand, it allows you to write tests for things that are often difficult to test, and a mock framework can - once you are familiar with it - make child's play out of the often laborious task of creating reliable test doubles. On the other hand, mock based tests are inevitably tied to the implementation of the code under test, and sometimes this coupling can be too tight for the test to be meaningful. Using mock objects normally also means that you need a very good understanding of the external APIs you are mocking. Otherwise, your mock may not be a good representation of how these systems would behave in the real world. Much has been written on this, for example by Martin Fowler.

As always, it pays to be pragmatic. If you find that you can't write a mock based test without reading every line of code in the method under test and reverse engineering it for the mocks, then an integration test may be more appropriate. In fact, it is prudent to have at least some integration tests in any case, since you can never be 100% sure your mocks are valid representations of the real objects they are mocking.

On the other hand, if the code you are testing is using well-defined APIs in a relatively predictable manner, mock objects can be a valuable way to test the "side effects" of your code, and a helpful tool to simulate things like exceptions and input values that may be difficult to produce otherwise.

Remember also that mock objects are not necessarily an "all or nothing" proposition. You can use simple dummy objects or "real" instances in most cases, and augment them with a few mock objects for those difficult-to-replicate test cases.

The following tables shows the most common field types for use in Dexterity schemata. See the documentation on creating schemata for information about how to use these.

Field properties

Fields are initialised with properties passed in their constructors. To avoid having to repeat the available properties for each field, we'll list them once here, grouped into the interfaces that describe them. You'll see those interfaces again in the tables below that describe the various field types. Refer to the table below to see what properties a particular interface implies.

Field types

The following tables describe the most commonly used field types, grouped by the module from which they can be imported.

Fields in zope.schema

Fields in plone.namedfile.field

See plone.namedfile and plone.formwidget.namedfile for more details.

Fields in z3c.relationfield.schema

See z3c.relationfield for more details

Fields in plone.app.textfield

See plone.app.textfield for more details

Most of the time, you will use the standard widgets provided by z3c.form. To learn more about z3c.form widgets, see the z3c.form documentation. To learn about setting custom widgets for Dexterity content types, see the schema introduction.

The table below shows some commonly used custom widgets.

Dexterity ships with several standard behaviors. The following table shows the interfaces you can list in the FTI behaviors properties and the resultant form fields and interfaces.

Dexterity uses the plone.autoform package to configure its z3c.form-based add and edit forms. This allows a schema to be annotated with "form hints", which are used to configure the form.

The easiest way to apply form hints in Python code is to use the directives from plone.directives.form and plone.directives.dexterity. These directives are used when the package is "grokked" (via the <grok:grok package="." /> ZCML directive) to apply the form hints to the interface where they are found. For this process to work, the schema must derive from plone.directives.form.Schema. Directives can be placed anywhere in the class body. By convention they are kept next to the fields they apply to.

For example, here is a schema that omits a field:

from plone.directives import form
from zope import schema

class ISampleSchema(form.Schema):
    
    title = schema.TextLine(title=u"Title")

    form.omitted('additionalInfo')
    additionalInfo = schema.Bytes()

The form directives take parameters in the form of a list of field names, or a set of field name/value pairs as keyword arguments. Each directive can be used zero or more times.

Form directives

The form directives in the plone.directives.form package are shown below.

The code sample below illustrates each of these directives:

from plone.directives import form
from zope import schema
from plone.app.z3cform.wysiwyg import WysiwygFieldWidget

class ISampleSchema(form.Schema):

    # A fieldset with id 'extra' and label 'Extra information' containing
    # the 'footer' and 'dummy' fields. The label can be omitted if the
    # fieldset has already been defined.
    
    form.fieldset('extra',
            label=u"Extra information",
            fields=['footer', 'dummy']
        )

    # Here a widget is specified as a dotted name.
    # The body field is also designated as the priamry field for this schema
    
    form.widget(body='plone.app.z3cform.wysiwyg.WysiwygFieldWidget')
    form.primary('body')
    body = schema.Text(
            title=u"Body text",
            required=False,
            default=u"Body text goes here"
        )

    # The widget can also be specified as an object
    
    form.widget(footer=WysiwygFieldWidget)
    footer = schema.Text(
            title=u"Footer text",
            required=False
        )
    
    # An omitted field. Use form.omitted('a', 'b', 'c') to omit several fields
        
    form.omitted('dummy')
    dummy = schema.Text(
            title=u"Dummy"
        )
    
    # A field in 'hidden' mode
    
    form.mode(secret='hidden')
    secret = schema.TextLine(
            title=u"Secret",
            default=u"Secret stuff"
        )
        
    # This field is moved before the 'description' field of the standard
    # IBasic behaviour, if this is in use.

    form.order_before(importantNote='IBasic.description')
    importantNote = schema.TextLine(
            title=u"Important note",
        )

Security directives

The security directives in the plone.directives.dexterity package are shown below. Note that these are also used to control reading and writing of fields on content instances.

The code sample below illustrates each of these directives:

from plone.directives import form, dexterity
from zope import schema

class ISampleSchema(form.Schema):

    # This field requires the 'cmf.ReviewPortalContent' to be read and
    # written
    
    dexterity.read_permission(reviewNotes='cmf.ReviewPortalContent')
    dexterity.write_permission(reviewNotes='cmf.ReviewPortalContent')
    reviewNotes = schema.Text(
            title=u"Review notes",
            required=False,
        )

In this section, we will describe some of the more commonly used APIs that can be used to inspect and manipulate Dexterity content objects. In most cases, the content object is referred to as context, its parent folder is referred to as folder, and the type name is example.type. Relevant imports are shown with each code snippet, though of course you are more likely to place those at the top of the relevant code module.

Content object creation and folder manipulation

This section describes means to create objects and manipulate folders.

Creating a content object

The simplest way to create a content item is via its factory:

from zope.component import createObject
context = createObject('example.type')

At this point, the object is not acquisition wrapped. You can wrap it explicitly by calling:

wrapped = context.__of__(folder)

However, it's normally better to add the item to a folder and then re-get it from the folder.

Note that the factory is normally installed as a local utility, so the createObject() call will only work once you've traversed over the Plone site root.

There is a convenience method that can be used to create a Dexterity object. It is mostly useful in tests:

from plone.dexterity.utils import createContent
context = createContent('example.type', title=u"Foo")

Any keyword arguments are used to set properties on the new instance (via setattr() on the newly created object). This method relies on being able to look up the FTI as a local utility, so again you must be inside the site for it to work.

Adding an object to a container

Once an object has been created, it can be added to a container. If the container is a Dexterity container, or another container that supports a dict API (e.g. a Large Plone Folder in Plone 3 or a container based on plone.folder), you can do:

folder['some_id'] = context

You should normally make sure that the id property of the object is the same as the id used in the container.

If the object only supports the basic OFS API (as is the case with standard Plone Folders in Plone 3), you can use the _setObject() method:

folder._setObject('some_id') = context

Note that both of these approaches bypass any type checks, i.e. you can add items to containers that would not normally allow this type of content. Dexterity comes with a convenience function, useful in tests, to simulate the checks performed when content is added through the web:

from plone.dexterity.utils import addContentToContainer
addContentToContainer(folder, context)

This will also invoke a name chooser and set the object's id accordingly, so things like the title-to-id behavior should work. As before, this relies on local components, so you must have traversed into a Plone site (PloneTestCase takes care of this for you).

To bypass folder constraints, you can use this function and pass checkConstraints=False.

You can also both create and add an object in one call:

from plone.dexterity.utils import createContentInContainer
createContentInContainer(folder, 'example.type', title=u"Foo")

Again, you can pass checkConstraints=False to bypass folder constraints, and pass object properties as keyword arguments.

Finally, you can use the invokeFactory() API, which is similar, but more generic in that it can be used for any type of content, not just Dexterity content:

new_id = folder.invokeFactory('example.type', 'some_id')
context = folder['new_id']

This always respects add constraints, including add permissions and the current user's roles.

Getting items from a folder

Dexterity containers and other containers based on plone.folder support a dict-like API to obtain and manipulate items in folders. For example, to obtain an (acquisition-wrapped) object by name:

context = folder['some_id']

Folders can also be iterated over, and you can all items(), keys(), values() and so on, treating the folder as a dict with string keys and content objects as values.

Dexterity containers also support the more basic OFS API. You can call objectIds() to get keys, objectValues() to get a list of content objects, objectItems() to get an items()-like dict,and hasObject(id) to check if an object exists in a container.

Removing items from a folder

Again, Dexterity containers act like dictionaries, and so implement __delitem__:

del folder['some_id']

The OFS API uses the _delObject() function for the same purpose:

folder._delObject('some_id')

Object introspection

This section describes means of getting information about an object.

Obtaining an object's schema interface

A content object's schema is an interface, i.e. an object of type zope.interface.interface.InterfaceClass.

from zope.app.content import queryContentType
schema = queryContentType(context)

The schema can now be inspected. For example:

from zope.schema import getFieldsInOrder
fields = getFieldsInOrder(schema)

Finding an object's behaviors

To find all behaviors supported by an object, use the plone.behavior API:

from plone.behavior.interfaces import IBehaviorAssignable
assignable = IBehaviorAssignable(context)
for behavior in assignable.enumerate_behaviors():
    behavior_schema = behavior.interface
    adapted = behavior_schema(context)
    ...

The objects returned are instances providing plone.behavior.interfaces.IBehavior. To get the behavior schema, use the interface property of this object. You can inspect this and use it to adapt the context if required.

Getting the FTI

To obtain a Dexterity FTI, look it up as a local utility:

from zope.component import getUtility
from plone.dexterity.interfaces import IDexterityFTI
fti = getUtility(IDexterityFTI, name='example.type')

The returned object provides plone.dexterity.interfaces.IDexterityFTI. To get the schema interface for the type from the FTI, you can do:

schema = fti.lookupSchema()

Getting the object's parent folder

A Dexterity item in a Dexterity container should have the __parent__ property set, pointing to its containment parent:

folder = context.__parent__

Items in standard Plone folders won't have this property set, at least not in Plone 3.x.

The more general approach relies on acquisition:

from Acquisition import aq_inner, aq_parent
folder = aq_parent(aq_inner(context))

Workflow

This section describes ways to inspect an object's workflow state and invoke transitions.

Obtaining the workflow state of an object

To obtain an object's workflow state, ask the portal_workflow tool:

from Products.CMFCore.utils import getToolByName
portal_workflow = getToolByName(context, 'portal_workflow')
review_state = portal_workflow.getInfoFor(context, 'review_state')

This assumes that the workflow state variable is called review_state, as is the case for almost all workflows.

Invoking a workflow transition

To invoke a transition:

portal_workflow.doActionFor(context, 'some_transition')

The transition must be available in the current workflow state, for the current user. Otherwise, an error will be raised.

Cataloging and indexing

This section describes ways of indexing an object in the portal_catalog tool.

Reindexing the object

Objects may need to be reindexed if they are modified in code. The best way to reindex them is actually to send an event and let Dexterity's standard event handlers take care of this:

from zope.lifecycleevent import modified
modified(context)

In tests, it is sometimes necessary to reindex explicitly. This can be done with:

context.reindexObject()

You can also pass specific index names to reindex, if you don't want to reindex everything:

context.reindexObject(idxs=['Title', 'sortable_title'])

This method comes from the Products.CMFCore.CMFCatalogAware.CMFCatalogAware mix-in class.

Security

This section describes ways to check and modify permissions. For more information, see the section on permissions.

Checking a permission

To check a permission by its Zope 3 name:

from zope.security import checkPermission
checkPermission('zope2.View', context)

Note: In a test, you may get an AttributeError when calling this method. To resolve this, call newInteraction() from Products.Five.security in your test setup (e.g. the afterSetUp() method).

To use the Zope 2 permission title:

from AccessControl import getSecurityManager
getSecurityManager().checkPermission('View', context)

Sometimes, normally in tests, you want to know which roles have a particular permission. To do this, use:

roles = [r['name'] for r in context.rolesOfPermission('View') if r['selected']]

Again, note that this uses the Zope 2 permission title.

Changing permissions

Normally, permissions should be set with workflow, but in tests it is often useful to manipulate security directly:

context.manage_permission('View', roles=['Manager', 'Owner'], acquire=True)

Again note that this uses the Zope 2 permission title.

Content object properties and methods

The following table shows the more important properties and methods available on Dexterity content objects. In addition, any field described in the type's schema will be available as a property, and can be read and set using normal attribute access.