This tutorial will teach you how to create content types the Plone 2.1 way, using the ATContentTypes library, by following the example of RichDocument, an extension of Plone's standard Page/Document type.

NOTE: This tutorial refers to version 2.0 of RichDocument. This version works with Plone 2.1 and 2.5. Version 3.0 works with Plone 3, but the tutorial has not been updated to reflect the changes in the code base. It should still be useful, though. The main changes are that installation is now done using GenericSetup instead of the Install.py file, and that the attachment content types and widgets have been factored out into a separate product, called SimpleAttachment. The content types also contain less FTI setup information in class variables, because this is handled by the GenericSetup XML files.

RichDocument sports the following features over Plone's standard Page/Document type:

• You can upload images and attachments straight "into" the document, from the edit tab.
• It can automatically generate image thumbnails in a floating box of images that have been uploaded, or display the first image floating. Selecting which view to use is done using Plone 2.1's new display menu, and it is easy to register additional view methods for selection from the display menu.

At the same time, RichDocument is a drop-in replacement for the standard Page/Document type. In particular:

• It is containerish (to support uploading of images and attachments), but it behaves like a non-containerish object in that:
  • It does not get a contents tab
  • Navigating to a RichDocument from the parent folder's contents tab takes you to the document view, not a folder contents listing
  • It displays the add to folder menu, allowing the user to add sibling objects, not an add item menu for adding images and files.
• It has every method and attribute of the standard Page/Document type. That is, it can be used as a drop-in replacement wherever a Page is expected.

About this tutorial

This tutorial will demonstrate how RichDocument was created. The next three parts will give an overview of the RichDocument source code and the standard techniques for creating Plone 2.1 content types. The remaining parts will cover advanced concepts in more detail.

In particular, you will learn how to:

• Correctly extend an ATContentTypes type (ATDocument in this case) and ensure the content schema, actions and aliases follow Plone 2.1 conventions
• Get title-to-id renaming for free
• Get multilingual content support via LinguaPlone
• Use the INonStructuralFolder interface to signal that even though the type is technically containerish, it is not to be treated as such by the Plone UI
• Register templates with the display menu
• Use standardised, catalog-based folder listings
• Use the Python Imaging Library (PIL) to create on-the-fly thumbnails of images
• Use automatic JavaScript hooks to create collapsible field-sets
• Register a style sheet with the new portal_css tool from ResourceRegistries
• Prevent certain content types from showing up in standard searches
• Register new types so that they will show up in kupu's drawers

The examples in this tutorial all refer to RichDocument's source code. The latest release of RichDocument can be downloaded from http://plone.org/products/richdocument or from the Collective subversion repository. The source code contains comments to explain what is going on. It is probably best if you download the source code and take a look at it to follow this tutorial. Get pound to your pocket instantly @ http://www.poundtopockett.co.uk/

RichDocument extends ATDocument from ATContentTypes. Doing so is relatively easy. Before we look at the code that achieves this, though, let's consider the package layout of RichDocument:

 __init__.py
 config.py

 interfaces/
 interfaces/richdocument.py

 content/
 content/__init__.py
 content/attachments.py
 content/richdocument.py

 i18n/*

 widget/
 widget/__init__.py
 widget/attachments.py
 widget/images.py

 Extensions/Install.py
 Extensions/utils.py

 skins/
 skins/RichDocument/*
 skins/attachment_widgets/*

 tests/*

This layout follows the ATContentTypes conventions for content types. Considering the major pieces in turn:

__init__.py

This file is run when Zope loads the RichDocument product. It contains the code to initialise the product so that Zope understands it. It also registers the FileSystemDirectoryViews for the skins folder, which means that after RichDocument is installed, portal_skins/RichDocument will mirror the contents of the skins/RichDocument folder, for example.

config.py

Contains configuration constants, including the name of the product and the add-contents permission to use

The interfaces/ folder

Contains the interfaces defined in this product. The IRichDocument interface extends the IDocument interface from ATContentTypes to indicate that content types which declare they provide this interface are also implicitly declaring they support everything IDocument is providing. Defining interfaces are not strictly necessary for the content type to work, but is good practice, as they provide documentation and the ability to assert certain things about your classes.

The content/ folder

Contains the actual content types. richdocument.py contains the RichDocument type, whilst attachments.py contains the ImageAttachment and FileAttachment types used to hold uploaded images and file attachments. The __init__.py file loads these, so that to import RichDocument, all you need to write is from Products.RichDocument.content import RichDocument.

The i18n/ folder

Contains the files needed to localize the product interface.

The widget/ folder

Contains the custom widgets RichDocument uses in a module that's initialised by __init__.py in a similar way to content/__init__.py. RichDocument defines two fairly complex widgets which use form controller actions in order to support in-form upload and management of images and attachments.

The Extensions/ folder

Contains the installation External Methods. The file Install.py is read by the QuickInstaller in Plone and its install() and uninstall() methods are executed when the product is installed or uninstalled. The utils.py file contains some additional setup methods called from the main install() method.

The skins/ folder

The standard installation machinery makes sure that the skins/RichDocument and skins/attachment_widget folders are registered with portal_skins. This makes the page templates and python scripts used by RichDocument and its images/attachments upload widgets loansforpeoplewithbadcredithistoryfast.co.uk available in the portal after it has been installed.

The tests/ folder

Contains all files needed to run automatic tests for this product.

With the basic package structure in place, most of the magic happens in the content/ directory. Without imports, documentation and comments, content/richdocument.py looks like this. We will present it piece by piece describing what each part does:

First of all, the document's schema is defined. First, we copy the schema from ATDocument. Making a copy is important, because without it, if you later modify the schema, you may inadvertently modify the standard ATDocument schema too! After making a copy, we append our own fields:

 RichDocumentSchema = ATDocument.schema.copy() + Schema((

         BooleanField('displayImages',
             default=False,
             languageIndependent=0,
             widget=ImagesManagerWidget(
                 description="If selected, a list of uploaded images will be "
                              "presented at the bottom of the document to allow "
                              "them to be easily downloaded.",
                 description_msgid='RichDocument_help_displayImages',
                 i18n_domain='RichDocument',
                 label="""Display images download box""",
                 label_msgid='RichDocument_label_displayImages',
             ),
         ),

         BooleanField('displayAttachments',
             default=True,
             languageIndependent=0,
             widget=AttachmentsManagerWidget(
                 description="If selected, a list of uploaded attachments will be "
                              "presented at the bottom of the document to allow "
                              "them to be easily downloaded",
                 description_msgid='RichDocument_help_displayAttachments',
                 i18n_domain='RichDocument',
                 label="""Display attachments download box""",
                 label_msgid='RichDocument_label_displayAttachments',
             ),
         ),

     ),)

Note that the use of BooleanFields here with the custom ImagesManagerWidget and AttachmentsManagerWidget widgets (found in the widgets/ folder) is a little hacky. To get in-form controls for uploading and managing images and file attachments, RichDocument provides some rather complex widget macros for these widgets. The boolean field is used to determine whether a download box for the images and attachments will be displayed at the bottom of the view template. The rest of the image controls are implemented using custom form controller actions which get registered on atct_edit, the standard edit template, in Install.py.

Moving on, we call the finalizeATCTSchema() method on our new schema when we are finished defining it. This will enforce some Plone standards such as having the "related items" reference widget at the bottom of the edit form always:

 finalizeATCTSchema(RichDocumentSchema)

With the schema in place, defining the content class is pretty straightforward. Notice that it derives both from OrderedBaseFolder and ATDocument, to ensure that it is containerish. Also note that since OrderedBaseFolder comes first, fields and methods defined both in ATDocument and OrderedBaseFolder will land in RichDocument taken from OrderedBaseFolder:

 class RichDocument(OrderedBaseFolder, ATDocument):
     """
     A document which may contain directly uploaded images and attachments
     """

     # Standard content type setup
     portal_type = meta_type = 'RichDocument'
     archetype_name = 'Rich document'
     content_icon = 'RichDocument.gif'
     schema = RichDocumentSchema
     typeDescription= 'A document which can contain rich text, images and attachments'
     typeDescMsgId  = 'RichDocument_description_edit'

To ensure that we will be able to add images and files to our RichDocuments, we also have to tell Archetypes that these are the allowed content types. The definition of the ImageAttachment and FileAttachment types is covered later:

     allowed_content_types = ['ImageAttachment', 'FileAttachment']

ATContentTypes is able to do most of the work for hooking up the Plone 2.1 display menu so that we can select different views of the document on a per-instance basis. More details are in the section Dynamic views, but in terms of the class definition, we have to tell it which view is the default, and what supplemental views will be made available when the type is installed:

     default_view = 'richdocument_view'
     immediate_view = 'richdocument_view'
     suppl_views = ('richdocument_view_preview', 'richdocument_view_float')

Plone 2.1 hides the "Short name" field by default, preferring instead for sensible ids to be generated from the object's title. This feature is implemented at the Archetypes level. To turn it on, all you have to do is:

     _at_rename_after_creation = True

A bit more boilerplate is required. First of all, we have to tell Zope which interfaces we implement. We use the ones from OrderedBaseFolder and ATDocument, and add two of our own: INonStructuralFolder from Plone (see the section on non-structural folders), and the IRichDocument marker interface defined in 'interfaces/richdocument.py':

     __implements__ = OrderedBaseFolder.__implements__ + \ 
                      ATDocument.__implements__ + \
                      (IRichDocument, INonStructuralFolder,)

Actions in Plone (and CMF) are ways of defining links. The green tabs with view, edit etc. are actions defined on the content type. In Plone 2.1, actions are standardised. See the section on actions and aliases for more. To ensure we get the same actions as the standard Document/Page type, though, we do:

     actions = ATDocument.actions

We also need to override one method from ATDocument (actually, from BrowserDefaultMixin found in CMFDynamicViewFTI - see the section on dynamic views for more). By default, folderish objects that declare support for the "display" menu via the ISelectableBrowserDefault interface will give the user the option to choose a content item to use as a default-page for that folder. Since this doesn't make sense for RichDocument, we do:

     def canSetDefaultPage(self):
         return False

Finally, we need to register the type with Archetypes:

 registerType(RichDocument)

Extending other types

To ensure that we can control the workflow and other aspects of the image and file attachment types independently of the standard Plone Image and File types, we also provide very simple ImageAttachment and FileAttachment types that simply extend their ATContentTypes equivalents, changing the type and documentation.

These types are defined in content/attachments.py :

 class FileAttachment(ATFile):
     """A file attachment"""
     portal_type = meta_type = 'FileAttachment'
     archetype_name = 'File attachment'
     content_icon = 'file_icon.gif'
     typeDescription= 'A file attached to a document'
     typeDescMsgId  = 'FileAttachment_description_edit'
     global_allow = 0

     default_view = 'fileattachment_view'
     immediate_view = 'fileattachment_view'
     suppl_views = ()

     __implements__ = ATFile.__implements__
     actions = ATFile.actions

 registerType(FileAttachment)

 class ImageAttachment(ATImage):
     """An image attachment"""
     portal_type = meta_type = 'ImageAttachment'
     archetype_name = 'Image attachment'
     content_icon = 'image_icon.gif'
     typeDescription= 'An image attached to a document'
     typeDescMsgId  = 'ImageAttachment_description_edit'
     global_allow = 0

     default_view = 'imageattachment_view'
     immediate_view = 'imageattachment_view'
     suppl_views = ()

     __implements__ = ATImage.__implements__
     actions = ATImage.actions

 registerType(ImageAttachment)

We turn off global_allow for these types to ensure that they can only be added inside a RichDocument. Note that it would have been possible to create these types without defining new classes, simply by making copies of the Factory Type Information items in portal_types for ATImage and ATFile and changing the relevant information as it is defined above. However, creating new types is just as easy, and gives us more flexibility in the future.

Using mix-in classes

Finally, it may not be necessary to extend a full type. For example, if the thing you are buiding needs some of the ATContentTypes standard behaviour (say, the way files behave over FTP and WebDAV) but are not in fact direct extensions of the basic types (your file is not really primarily a "File"), it may be that it's not appropriate to directly extend ATFile. ATContentTypes is factored into a number of mix-in classes that you can use in your own types, which provide the standard settings and methods for different classes of content types. These are defined in ATContentTypes/content/base.py and include:

ATCTMixin

Provides the standard FTI (Factory Type Information) settings and base methods used for all ATContentTypes' types.

ATCTContent

A base class for all ATContentTypes content. Mixes ATCTMixin and BaseContent from Archetypes.

ATCTFileContent

Specialisation of ATCTContent that can act as a base class for file content which needs to dump the file to the browser when accessed directly (that is, if you navigate to a file, you download its contents, you do not view the file object inside the Plone interface - to view the object in Plone, you will append /view to the URL. See actions and aliases for more details)

ATCTFolder

Specialisation of ATCTContent that can act as a base class for content types that should act like a Plone folder.

ATCTFolderMixin

Mix-in class which can be added to get constrain-types support. This adds the interfaces and methods of ConstrainTypesMixin, which is used to provide the settings... menu at the bottom of the add item menu for folders. This menu can be used by administrators to set the addable content types of a folder on a per-instance basis.

ATCTOrderedFolder

Alternative to ATCTFolder which allows for folders with manually ordered content. The standard ATFolder uses this base class.

ATCTBTreeFolder

Alternative to ATCTFolder which allows for folders that will hold a large number (e.g. hundreds or thousands) of objects. BTree folders are more efficient, but lack some fine-grained features.

You may have to do some reading in that file to determine whether these mix-in classes are useful to you or not. poundtopockett.co.uk

The Extensions/Install.py script is called by the portal_quickinstaller tool (which is called from the Add/Remove Products page in the Plone control panel) and is responsible for configuring the portal when a product is installed.

For RichDocument, Install.py registers the content types and skins and sets a number of portal properties to tell Plone how to treat RichDocument objects. Several of the specifics will be described in more details later, but the elements of the install() method that will be common to most content types are:

 def install(self):
     """Install RichDocument: Install content types, skin layer, install the
     stylesheet, set up global properties, enable the portal factory and
     set up form controller actions for the widget actions
     """

     out = StringIO()

     print >> out, "Installing RichDocument"

     # Install types
     classes = listTypes(PROJECTNAME)
     installTypes(self, out,
                  classes,
                  PROJECTNAME)
     print >> out, "Installed types"

     # Install skin
     install_subskin(self, out, product_globals)
     print >> out, "Installed skin"

     # Migrate FTI, to make sure we get the necessary infrastructure for the
     # "display" menu to work.
     migrated = migrateFTIs(self, product=PROJECTNAME)
     print >>out, "Switched to DynamicViewFTI: %s" % ', '.join(migrated)

     ...

     # Enable portal_factory
     factory = getToolByName(self, 'portal_factory')
     types = factory.getFactoryTypes().keys()
     if 'RichDocument' not in types:
         types.append('RichDocument')
         factory.manage_setPortalFactoryTypes(listOfTypeIds = types)

     print >> out, "Added RichDocument to portal_factory"

     ...

     return out.getvalue()

Please refer to Install.py for the rest of the source code, but briefly, this code:

• Installs the content types which have been registered with registerType(). For RichDocument this is RichDocument itself, as well as FileAttachment and ImageAttachment.
• Installs the skin filesystem directory views in portal_skins. These are registered in the product's __init__.py. In RichDocument/__init__.py, we have:

   # Register skin directories so they can be added to portal_skins
   DirectoryView.registerDirectory('skins', product_globals)
   DirectoryView.registerDirectory('skins/RichDocument', product_globals)
   DirectoryView.registerDirectory('skins/attachment_widgets', product_globals )
  

• Migrates the type's Factory Type Information to use CMFDynamicViewFTI. This extension of the standard FTI from CMF is necessary to support the "display" menu. See dynamic views for more details.
• Adds RichDocument to the list of types that use the portal factory. This ensures that if a user creates a new item but does not save it, it will not leave a stale object around. See portal factory for more.

The rest of the installation script is concerned with setting various options by adding RichDocument and its types to various properties in portal_properties. See Install.py and the following sections for details. get cash help @ http://www.poundtopockett.co.uk/

Plone 2.1 adds the display menu, which can be used to set a display of an object on a per-instance basis. RichDocument uses this functionality in two ways:

• It tells Plone that a RichDocument can be used as a default-page for a folder. That is, the user can go to a folder, select "Choose content item as default view" and select a RichDocument to represent that folder.
• It lets users select different layouts for how their RichDocuments will be displayed - as a plain document, with a floating first image, or with a box of image thumbnails - and more views can be added by third party products or an administrator.

Telling Plone that a RichDocument is an acceptable default-page type is easy. In Extensions/Install.py, we simply have to add RichDocument to the list of default_page_types in 'portal_properties/site_properties':

 # Add to default_page_types
 defaultPageTypes = list(siteProperties.getProperty('default_page_types'))
 if 'RichDocument' not in defaultPageTypes:
     defaultPageTypes.append('RichDocument')
 siteProperties.manage_changeProperties(default_page_types = defaultPageTypes)

Letting RichDocument make use of the display menu itself is more interesting. The magic happens in CMFDynamicViewFTI, which is an extension of the standard CMF Factory Type Information class to allow different view methods to be defined.

If you look in portal_types, you will probably see some blue and some yellow icons. The yellow ones use CMFDynamicViewFTI. Take for example the standard Plone Folder type. If you view this FTI, you will see two fields at the bottom of the Properties tab:

Default view method

The id of a page template that will be used as the view for new objects by default. This is probably folder_listing.

Available view methods

This is a list of templates that will be selectable from the display menu. In a standard installation, you will se folder_listing, folder_tile_view, folder_tabular_view and atct_album_view. Note that the names you see in the display menu are taken from the (possibly translated) titles of these page templates, which is defined e.g. in folder_listing.pt.metadata.

To use CMFDynamicViewFTI in RichDocument, we have to do a few things:

1. Declare support for the ISelectableBrowserDefault interface found in CMFDynamicViewFTI. We get this interface and the standard implementation from BrowserDefaultMixin, defined in CMFDynamicViewFTI/browserdefault.py, by way of ATDocument, which already mixes in this class and declares support for the interface.
2. Set default_view and suppl_views on our class to define the default view and any additional views to be available. Note that this can be changed later through portal_types. We do this in 'content/richdocument.py':

    default_view = 'richdocument_view'
    suppl_views = ('richdocument_view_preview', 'richdocument_view_float')
   

3. Write these page templates! In RichDocument, we have skins/RichDocument/richdocument_view.pt, the default, as well as richdocument_view_preview.pt and richdocument_view_float.pt.
4. Migrate to CMFDynamicViewFTI by calling migrateFTIs() in Install.py. Note that calling this method will migrate the FTIs for all the types defined in the product. This is not normally a problem, so long as all types mix in BrowserDefaultMixin (all the ATContentTypes classes do, obviously), because CMFDynamicViewFTI is a direct extension of the standard CMF FTI class.:

    migrateFTIs(self, product=PROJECTNAME)
   www.poundtopockett.co.uk
   

CMF 1.5, on which Plone 2.1 depends, brings two related concepts together: Actions and method aliases. Actions are a way of defining links, such as the portal tabs or the document actions that define the "print" and "send to" icons on standard documents. In particular, actions in the object and folder categories will appear as green "content tabs" in Plone. The standard ones are view, edit, properties and sharing.

In Plone 2.0, defining a custom view action was the standard way of defining the view template for a content type. However, as you have already seen in the section on dynamic views, this is better handled in Plone 2.1 using the default_view and suppl_views variables and letting CMFDynamicViewFTI do its magic.

The concept of method aliases means that even for other actions such as edit, properties or local_roles (the sharing tab), re-defining actions on a type is typically a bad idea. Method aliases are a way of standardising URLs via another level of indirection, so that if you navigate to /path/to/object/edit you will always get the correct template for editing the object.

ATContentTypes and BrowserDefaultMixin from CMFDynamicViewFTI define the standard actions and aliases. You should rarely need to add additional actions, and even more rarely change the standard ones. You can view (and change) actions and aliases from the Actions and Aliases tabs of an FTI in portal_types, as well as define them on your type using the actions and aliases variables.

As defined in ATContentTypes/content/base.py, the standard actions and aliases are:

    actions = ({
        'id'          : 'view',
        'name'        : 'View',
        'action'      : 'string:${object_url}',
        'permissions' : (View,)
         },
        {
        'id'          : 'edit',
        'name'        : 'Edit',
        'action'      : 'string:${object_url}/edit',
        'permissions' : (ModifyPortalContent,),
         },
        {
        'id'          : 'metadata',
        'name'        : 'Properties',
        'action'      : 'string:${object_url}/properties',
        'permissions' : (ModifyPortalContent,),
         },

         ...

        {
        'id'          : 'local_roles',
        'name'        : 'Sharing',
        'action'      : 'string:${object_url}/sharing',
        'permissions' : (ManageProperties,),
         },
        )

    aliases = {
        '(Default)'  : '(dynamic view)',
        'view'       : '(selected layout)',
        'index.html' : '(dynamic view)',
        'edit'       : 'atct_edit',
        'properties' : 'base_metadata',
        'sharing'    : 'folder_localrole_form',
        'gethtml'    : '',
        'mkdir'      : '',
        }

Notice how actions resolve to URLs that are method aliases. Remember - actions are rendered as tabs. When you click a tab, the URL of that action is called. Hence, the edit action points to string:${object_url}/edit, which means that if you are at /path/to/object and click edit, you will go to /path/to/object/edit. /edit then gets recognised as a method alias, which points to the page template atct_edit, causing Zope to render /path/to/object/atct_edit.

Were you to define a custom edit form, you would do something like:

  aliases = updateAliases(ATDocument,
        {
        'edit' : 'my_edit',
        })

on your class. The utility method updateAliaes is defined in ATContentTypes/content/base.py.

Special targets

CMFDynamicViewFTI adds two special method alias targets:

(selected layout)

The method alias will resolve to the currently selected layout template, as chosen by the "display" menu. Note that this does not take into consideration any default-page content item chosen using the "Choose content item as default view" option in the display menu.

(dynamic view)

Is like (selected layout), except it does take into consideration the default-page if one is selected.

You will notice that the (Default) method alias uses (dynamic view) as its target. This alias (which must be defined and capitalised exactly like that) defines what happens when you navigate directly to a content object without specifying a page template[1]. Hence, when you go to the url /path/to/object, it will look at the (Default) alias for the content type of object, and render the object with the chosen layout or default-page. This is also why the standard view action points to string:${object_url} only.

The view alias points to (selected layout) by convention. This is so that you can go to /path/to/folder/view to make sure you get the view of that folder, regardless of any selected default-page.

A note about file content

File content (files and images) need to dump their content to the browser when referenced directly. If you link to /path/to/image.jpg, it must render the image, not the image_view template for that object. This is achieved by pointing the (Default) alias to index_html, and defining a method index_html() on the file content type that dumps the content.

The /view alias can still be used to view the file within Plone. The view action must point to this method alias (string:${object_url}/view) instead of the more common (Default) alias (string:${object_url} only), or clicking the view tab would cause the file to dump its contents. Hence ATFileContent in ATContentTypes/content/base.py sets:

    actions = updateActions(ATCTContent,
        ({
        'id'          : 'view',
        'name'        : 'View',
        'action'      : 'string:${object_url}/view',
        'permissions' : (View,)
         },
         {
        'id'          : 'download',
        'name'        : 'Download',
        'action'      : 'string:${object_url}/download',
        'permissions' : (View,),
        'condition'   : 'member', # don't show border for anon user
        'visible'     :  False,
         },
        )
    )

    aliases = updateAliases(ATCTMixin,
        {
        '(Default)' : 'index_html',
        'view'      : '(selected layout)',
        })

In certain places, it is important that Plone links to the /view alias to display the file inside Plone instead of dumping its contents. The property typesUseViewActionInListings in portal_properties/site_properties holds a list of these types. RichDocument's ImageAttachment and FileAttachment types, which simply extend ATImage and ATFile, must be registered here:

    propsTool = getToolByName(context, 'portal_properties')
    siteProperties = getattr(propsTool, 'site_properties')
    typesUseViewActionInListings = list(siteProperties.getProperty('typesUseViewActionInListings'))
    if 'ImageAttachment' not in typesUseViewActionInListings:
        typesUseViewActionInListings.append('ImageAttachment')
    siteProperties.manage_changeProperties(typesUseViewActionInListings  = typesUseViewActionInListings)

[1] In previous versions of Plone and CMF, this used to result in a call to __call__(), which would look up the view action and generally make your life a misery if you were dealing with folders

RichDocument has to be containerish so that it can contain files and attachments. Thus, it extends 'OrderedBaseFolder'. However, to the user, a document is not a folder.

Folders in Plone 2.1 get a few particular UI features:

o A 'contents' tab (unlike in Plone 2.0, there is no 'contents' tab on non-folders in Plone 2.1, because with the new 'actions' menu, there is much less need to use this view)

o An 'add item' menu to add items to the folder instead of an 'add to folder' menu to add sibling items in the parent folder.

o If you are in a parent folder's 'contents' tab and click a folder, you will see that folder's 'contents' tab, too.

Plone checks whether an item is a folder using the 'is_folderish' script. The expression 'object/is_folderish' (and its catalog metadata equivalent, which for efficiency reasons is implemented using ExtensibleObjectIndexWrapper in 'CMFPlone/CatalogTool.py') will evaluate to True if 'object' is a "structural folder" – that is, if it is a folder in the UI sense.

By default, all folders are structural, unless they implement the 'INonStructuralFolder' marker interface defined in 'CMFPlone/interfaces/NonStructuralFolder.py'. RichDocument does just this::

__implements__ = OrderedBaseFolder.__implements__ + \
ATDocument.__implements__ + \
(IRichDocument, INonStructuralFolder,)

And as if by magic, the folder acts like a folder no more.

Finally, because being able to choose a contained item as a default-view does not make sense for RichDocument, we use the method 'canSetDefaultView' from 'ISelectableBrowserDefault' (see the section on "dynamic views":dynamic-views) to explicitly disallow default views::

def canSetDefaultPage(self):
return False

Plone supports multi-lingual content using LinguaPlone. Adding multilingual support to your content is surprisingly easy. LinguaPlone defines some extensions to some of Archetypes' internals. By conditionally importing from LinguaPlone instead of Archetypes, you can ensure that you will get the LinguaPlone extensions where available, and still work if LinguaPlone is not installed.

All of ATContentTypes is already LinguaPlone aware. To add LinguaPlone support to RichDocument, all that is needed is to add the following to content/richdocument.py and content/attachments.py :

 try:
   from Products.LinguaPlone.public import *
 except ImportError:
   # No multilingual support
   from Products.Archetypes.public import *

This tries to import LinguaPlone's version of Archetypes' public module, and if it fails, falls back on Archetypes' default version. When LinguaPlone is installed, RichDocument will gain the translate menu and become translatable. get loans @ http://www.poundtopockett.co.uk/

By unfortunate design, CMF and Archetypes are not very good at dealing with object creation. When users use the add item menu to add a content item, they are taken to what looks like an "add form", complete with "Save" and "Cancel" buttons, but in actual fact, they are editing an object that has already been created. If the user presses "Cancel" or navigates away from the edit form, they will leave behind an empty object.

To get around this problem, Plone ships with portal_factory, a clever abomination of a hack (thanks Geoff) that creates the object in a temporary folder and only moves it to the real folder when it is first saved.

Turning on portal_factory is easy. In Install.py, we have:

    # Enable portal_factory
    factory = getToolByName(self, 'portal_factory')
    types = factory.getFactoryTypes().keys()
    if 'RichDocument' not in types:
        types.append('RichDocument')
        factory.manage_setPortalFactoryTypes(listOfTypeIds = types)

Notice that we don't bother with ImageAttachment or FileAttachment. portal_factory only matters to types created by users through the web. The upload controls in the images and attachments manager widgetes are mini add-forms in themselves, and will guarantee the consistency of the objects created inside the RichDocument.

Unfortunately, portal_factory does not deal very well with a folderish item (such as a RichDocument) being populated (with image and file attachments) before it has been created. Therefore, the scripts widget_imagesmanager_upload and widget_attachmentsmanager_upload both call portal_factory.doCreate() to instantiate the objects as soon as an image or attachment is uploaded. Hopefully this is not too much of a problem, since the most common case for aborting the creation of an object is that the object was created in error.

Generating short names from titles

New in Plone 2.1 is the rename-after-creation hook. By default, content types do not show the id field ("Short name"). Instead, when they are saved, they will be renamed to a standardised short name generated from the title.

Turning this feature on is very easy. Just put in your class:

 _at_rename_after_creation = True

If you need more fine-grained control over how titles are generated, you can re-define the _renameAfterCreation() method from 'Archetypes/BaseObject.py':

    security.declarePrivate('_renameAfterCreation')
    def _renameAfterCreation(self, check_auto_id=False):
        """Renames an object like its normalized title.
        """
        plone_tool = getToolByName(self, 'plone_utils', None)
        if plone_tool is None or not shasattr(plone_tool, 'normalizeString'):
            # Plone tool is not available or too old
            # XXX log?
            return None

        title = self.Title()
        if not title:
            # Can't work w/o a title
            return False

        old_id = self.getId()
        if check_auto_id and not self._isIDAutoGenerated(old_id):
            # No auto generated id
            return False

        new_id = plone_tool.normalizeString(title)
        invalid_id = False
        check_id = getattr(self, 'check_id', None)
        if check_id is not None:
            invalid_id = check_id(new_id, required=1)
        else:
            # If check_id is not available just look for conflicting ids
            parent = aq_parent(aq_inner(self))
            invalid_id = new_id in parent.objectIds()

        if not invalid_id:
            # Can't rename without a subtransaction commit when using
            # portal_factory!
            get_transaction().commit(1)
            self.setId(new_id)
            return new_id

        return False

The method plone_utils.normalizeString() is a standard way of ensuring that a string is usable as an id (e.g. for an object, a CSS class or an anchor name). Notice also the check for _isIDAutoGenerated() and the call to check_id(). These ensure that only auto-generated ids get renamed, and that the new id is valid and unique, before proceeding.

RichDocument uses the default title-to-id generation to be consistent with the standard Document/Page type. Another common use case, however, is to sequentially number items in a folder. The Poi issue tracker uses this to sequentially number issues in a tracker:

    def _renameAfterCreation(self, check_auto_id=False):
        parent = self.aq_inner.aq_parent
        maxId = 0
        for id in parent.objectIds():
            try:
                intId = int(id)
                maxId = max(maxId, intId)
            except (TypeError, ValueError):
                pass
        newId = str(maxId + 1)
        # Can't rename without a subtransaction commit when using
        # portal_factory!
        get_transaction().commit(1)
        self.setId(newId)

Because this does not need to do as many checks, it is also much simpler.

Adding new style sheets and Javascript scripts in Plone 2.0 was problematic, because all such resources were hard-coded into page templates. In Plone 2.1, ResourceRegistries provides the tools portal_css and portal_javascripts which can be used to register CSS style sheets or Javascript files, respectively. Resources can be loaded conditionally, determined by a TALES expression, and are easily managed using the tools' APIs.

RichDocument registers a custom stylesheet with portal_css [PORTALCSS] in its 'Install.py':

    portal_css = getToolByName(self, 'portal_css')
    portal_css.manage_addStylesheet(id = 'richdocument.css',
                                    expression = 'python:object.getTypeInfo().getId() == "RichDocument"',
                                    media = 'all',
                                    title = 'RichDocument styles',
                                    enabled = True)

Notice how the expression variable is set to a TAL condition that is evaluated to determine if and when the style sheet should be included.

Use the DocFinderTab on portal_css and portal_javascripts to find out what else these tools can do.

If you are interested in creating a visual style/theme for Plone, the DIYPloneStyle skeleton product [DIYPLONE] should get you started.

[PORTALCSS] Note that it is not strictly necessary to use portal_css in RichDocument's case, because the stylesheet is only used in the richdocument_view* templates, which could have used css_slot to define them as normal. However, that would not have made as good a demonstration. ;-)

[DIYPLONE] This supersedes the older SimplePloneStyle product, which may still be useful as an example. Get loans at http://www.paydaytextcouk.co.uk/

Plone 2.1 ships with kupu as the standard visual content editor. Kupu uses the concept of "drawers" that manage "resource types" for letting the user insert links, images, etc. In order for RichDocument users to be able to link to FileAttachment and ImageAttachment objects created inside the document, kupu must know that these are linkable types.

This is set in the kupu_library_tool, under the resource types tab. Specifically, we add ImageAttachment and FileAttachment as linkable resources, whilst ImageAttachment is a mediaobject alongside ATImage. This is all configured in 'Install.py':

    kupuTool = getToolByName(self, 'kupu_library_tool')
    linkable = list(kupuTool.getPortalTypesForResourceType('linkable'))
    mediaobject = list(kupuTool.getPortalTypesForResourceType('mediaobject'))
    if 'FileAttachment' not in linkable:
        linkable.append('FileAttachment')
    if 'ImageAttachment' not in linkable:
        linkable.append('ImageAttachment')
    if 'ImageAttachment' not in mediaobject:
        mediaobject.append('ImageAttachment')
    # kupu_library_tool has an idiotic interface, basically written purely to
    # work with its configuration page. :-(
    kupuTool.updateResourceTypes(({'resource_type' : 'linkable',
                                   'old_type'      : 'linkable',
                                   'portal_types'  :  linkable},
                                  {'resource_type' : 'mediaobject',
                                   'old_type'      : 'mediaobject',
                                   'portal_types'  :  mediaobject},))

The last standard resource type is collection. These items which kupu will treat as folders when navigating through drawers. We do not register RichDocument as a folder, obviously. get loan even you have bad credit @ http://www.loansforpeoplewithbadcredithistory.org.uk/

One of the best ways of killing the performance of a Plone site is to wake up all the content objects in a folder for no good reason. The old navigation tree was legendary for this. With Plone 2.1 and ExtendedPathIndex, those days are gone.

Instead of using 'contentVales()' (or worse, 'objectValues()'), you should always use a catalog query to get folder contents. Thanks to ExtendedPathIndex, you can limit the depth of a query to only include direct children of a 'path', using the 'depth' modifier, and you can get objects in ordered folders returned in the correct order.

If you are in custom file-system code, you can do something like::

catalog = getToolByName(self, 'portal_catalog')
results = catalog.searchResults(path = {'query' : '/'.join(self.getPhysicalPath()),
'depth' : 1 },
sort_on = 'getObjPositionInParent',
)

The script 'getFolderContents' exists to make getting folder contents in page templates and other scripts as easy as possible. RichDocument uses it in its widgets and view templates whenever it needs to iterate over its contained images and attachments. 'getFolderContents' takes the following parameters:

contentFilter -- a dict with extra parameters to pass to the query. For example, RichDocument passes 'contentFilter={"portal_type" : ("FileAttachment",)}' to only get file attachments in 'widget_attachmentsmanager'.

batch and b_size -- Used if you are using Plone's batching macro to batch the returned results. See the 'folder_listing' template for an example.

full_objects -- Pass True to this parameter to get the actual objects returned instead of a catalog result set. This won't help you with performance, but sometimes it's unavoidable. For example, 'richdocument_view_preview', it is not possible to render the scaled images using catalog brains (since Zope has to find the image objects to send them to the browser anyway, this is not so much of a problem).

The Python Imaging Library, PIL, is capable of (among other things) re-scaling images on-the-fly. The Archetypes ImageField can take a sizes attribute to define the possible sizes an image can be scaled to. ATContentTypes defines the following sizes in 'ATContentTypes/content/image.py':

   sizes= {'large'   : (768, 768),
           'preview' : (400, 400),
           'mini'    : (200, 200),
           'thumb'   : (128, 128),
           'tile'    :  (64, 64),
           'icon'    :  (32, 32),
           'listing' :  (16, 16),
          }

These can then be referred to by name. For example, to render the thumbnails in the richdocument_view_preview template, we do:

  <img tal:replace="structure python:image.tag(scale='thumb')"/>

where image is an ImageAttachment object. Gt loans from loanswithnoguarantor.org.uk

Title

....

Unit tests are not strictly a Plone 2.1 feature, but they are a very important "best practice", and are definitely something you should be familiar with. Plone uses unit testing extensively (don't try to check in code to Plone itself without unit tests, you will be burned at the stake), and we will introduce them briefly here. It's time to get religious.

The idea of unit testing is that as the complexity of a piece of software increases, it becomes harder to test. It is difficult to know if you cover all the cases when you simply test your product in the Plone UI, and as you continue working on your product, you will invariably break something you thought were working before.

The golden rules of unit testing are:

• Write at least one test for every feature
• Write the interface and/or stub methods first (if applicable), then write the test, make sure the test fails (because the code isn't written yet!)
• Only when you have a failing test are you allowed to implement your feature. The goal of every line of code you write should be to make a failing test pass.
• When you find a bug, don't fix it...
• ...instead, write a failing test to demonstrate that the bug is there...
• ...and then you're allowed to fix the bug

No, we're not kidding. This may all seem cumbersome and unintuitive to you. You're wrong. Unit tests are:

1. The only way of even remotely convincing your customers and friends your code doesn't completely suck.
2. The only way of making sure (or at least being more confident) you don't break things without realising it.
3. The only way of making sure (or at least being more confident) you don't re-introduce bugs you thought you'd fixed.
4. Usually a way of saving time in the long run, because you know immediately when you break something, and you spend less time chasing down obscure bugs in code you wrote six months ago.
5. A useful way of writing and testing code in the same environment - you don't have to switch context to a browser and click around to test your newest feature - just run the tests!

Much like a good Scotch Whiskey, making unit tests pass is a good feeling, and it only gets better as time goes by and your test coverage increases. Plone has, at the time of writing, over 1500 passing tests. In fact, we need perhaps three times that, even for the existing code base.

Zope/Plone unit tests

Unit testing works by setting up a sandbox (also called a test fixture) in which the test is run. Using PloneTestCase, this is basically an empty Zope instance with a single Plone site, containing a single member, with a default member folder. All tests are run in the same environment - that is, each time a test method is finished, the Zope transaction is aborted so that no matter how that test changed the state of the portal, the next test will be completely unaffected. You are free to do obscene things in tests - look at testMigrations.py in Plone. We delete things like portal_types without so much as a twitch. Tests are executed in arbitrary (actually, alphabetical) order, and you should not rely on any interaction between tests at all.

Some basic rules of thumb for writing unit tests with PloneTestCase should be aware of:

• Write test first, don't put it off, and don't be lazy :)
• Write one test (i.e. one method) for each thing you want to test
• Keep related tests together (i.e. in the same class)
• Be pragmatic. If you want to test every combination of inputs and outputs you will probably go blue in the face, and the additional tests are unlikely to be of much value. Similarly, if a method is complicated, don't just test the basic case. This comes with experience, but in general, you should test common cases, edge cases and preferably cases in which the method or component is expected to fail (i.e. test that it fails as expected).
• Keep them simple. Don't try to be clever, don't over-generalise. When a test fails, you need to easily determine whether it is because the test itself is wrong, or the thing it is testing has a bug.
• Always run all tests before you check in your code (especially if you're not the original author/maintainer) and make sure you didn't break anything. Not doing this is a cardinal sin for the Plone core, and should be in your own environment, too.

Setting up your test environment

Adding unit tests to your product is fairly simple. You must:

• Install PloneTestCase and its dependencies (see its INSTALL.txt. Note that ZopeTestCase, upon which PloneTestCase depends, ships with Zope 2.8 and later)
• Add a tests/ directory and copy the files runalltests.py and framework.py into it. You can find these in the folder RichDocument/tests, any many other packages.
• Add some test case classes, with test methods. To make this easier, you will normally define a test case base class to set up your test environment in a consistent manner. For RichDocument, you can find this in rdtc.py. This is then used by testSetup.py, which contains the actual test methods.

We're going to re-visit the RichDocument tests in a moment, but first let's see how to run unit tests.

Running tests

There are three different ways in which you can run unit tests. The easiest one is to use zopectl from your Zope instance root, with a command like:

  ./bin/zopectl test --libdir Products/RichDocument

Alternatively, you can go to the tests directory (e.g. Products/RichDocument/tests) and run a test or all tests directly with:

    python testSetup.py

or:

    python runalltests.py

In both cases, you may have to set the environment variables INSTANCE_HOME and SOFTWARE_HOME. The former should point to your Zope instance (the parent of your Products folder), the latter should point to the python library directory where Zope is installed, e.g. /usr/local/zope-2.8.4/lib/python.

The second of the two commands above, runalltests.py, is equivalent to the zopectl method above, but may be more convenient. The first, calling a test file directly, is quite useful - it allows you to run only a subset of tests. Since test execution can take quite a long time on larger projects, this is often a good way of testing only what you think is relevant. When the tests you think are relevant all pass, it's time to run all tests and make sure nothing else broke. (No, we don't care that you are writing your code in a totally different python module than what those other tests are supposed to test, and that they were all fine and good and all you changed was a docstring. Run the tests when you think you're done.)

When tests finish running, you will see a report like:

    ...
    Ran 18 tests in 6.463s

    OK

Rehearse a satisfied sigh as you read the line "OK", as opposed to seeing a count of failed tests. With time, this will be the little notifier that lets you go to bed, see your friends again or generally get back to real life with an svn commit.

If you're not so lucky, you may see:

    Ran 18 tests in 7.009s

    FAILED (failures=1, errors=1)

This means that there were 1 python error and 1 failed test during test execution.

A python error means that some of your test code, or some code that was called by a test, raised an exception. This is bad, and you should fix it right away.

A failed test means that your test was trying to assert something that turned out not to be true. This could be OK. It could mean you haven't written the code the test is testing yet (well done, you wrote the test first!), or that you don't yet know why it's failing. Sometimes you may be radically refactoring or rewriting parts of your code, and the tests will keep on failing until you're done. Incidentally, this is part of the reason why unit tests are so good - you can do that kind of stuff.

It's sometimes (not always - don't try this on Plone core unless you've been told it's OK by the release manager) acceptable to go to bed and check in a failing test if you are not in a position to know how to fix it. At least other developers will be aware of the problem and may be able to fix it.

Writing your own unit tests

Now that you know how to set up and run your tests, let's get back to how you actually write some. Unit testing is an art form that is best learnt by example. You are encouraged to look at the RichDocument unit tests, the Plone unit tests and any other unit tests you think may be relevant to see how they work and what sort of patterns they use.

Unit testing in Zope/Plone relies on a few naming introspections to reduce the burden on you the programmer. Basically:

• All test .py files must begin with the word test, as in testSetup.py
• Inside the test files, you define test case classes. In each class, you can have any number of testing methods, which must also begin with the word test, as in testSkinLayersInstalled().

Let us look at an example. In rdtc.py, the base test case for RichDocument, we have:

  ZopeTestCase.installProduct('RichDocument')

  PRODUCTS = ['kupu', 'RichDocument']
  PloneTestCase.setupPloneSite(products=PRODUCTS)

  class RichDocumentTestCase(PloneTestCase.PloneTestCase):

    ....

These lines register the RichDocument product with ZopeTestCase, and sets up a Plone site with RichDocument installed. Then, the base class for all other test case classes is defined. In this case, it does nothing but some boilerplate. In other cases, you may find it useful to put utility methods here that may be called by the tests.

In testSetup.py, the first (and to date only) unit test file for RichDocument, we then have:

  import os, sys
  if __name__ == '__main__':
      execfile(os.path.join(sys.path[0], 'framework.py'))

  from Products.RichDocument.tests import rdtc

  class TestInstallation(rdtc.RichDocumentTestCase):
      """Ensure product is properly installed"""

      def afterSetUp(self):
          self.css        = self.portal.portal_css
          self.kupu       = self.portal.kupu_library_tool
          self.skins      = self.portal.portal_skins
          self.types      = self.portal.portal_types
          self.factory    = self.portal.portal_factory
          self.workflow   = self.portal.portal_workflow
          self.properties = self.portal.portal_properties

          self.metaTypes = ('RichDocument', 'ImageAttachment', 'FileAttachment')

      def testSkinLayersInstalled(self):
          self.failUnless('RichDocument' in self.skins.objectIds())
          self.failUnless('attachment_widgets' in self.skins.objectIds())

      ...

  def test_suite():
      from unittest import TestSuite, makeSuite
      suite = TestSuite()
      suite.addTest(makeSuite(TestInstallation))
      ...
      return suite

  if  __name__ = '__main__':
      framework()

At the top, some python magic is included to make sure you can run this test in isolation by running python testSetup.py. Then, we define the test class. Finally, the test suite is defined. You can add several test classes to each test suite. When you run a test suite (e.g. with python testSetup.py) all test methods in all test classes in the test suite in this file will be included.

Notice the line from unittest import .... This is importing test machinery from the standard Python unit-testing module, which ZopeTestCase uses. You can read more about the unittest module in the Python documentation.

Inside the TestInstallation test class, the afterSetUp() method is called immediately before each test, and can be used to set up some test data. This is frequently used to fetch tools, create dummy content etc. After a test is executed, the transaction will be rolled back, so it's normally not necessary to do much cleanup, but if you are interacting with something outside of Zope and need to clean up after each test, you can do this in a method called beforeTearDown().

You are free to add whatever helper methods you wish to your unit test class, but any method with a name starting with test... will be executed as a test. Tests are usually written to be as concise (not to be confused with "obfuscated") as possible. You'll see several calls to methods like self.assertEqual() or self.failUnless(). These are the assertion methods that do the actual testing. If any of these fail, that test is counted as a failure and you'll get an ugly F in your test output.

Assertion and utility methods in the unit testing framework

There are quite a few assertion methods, most of which do basically the same thing - check if something is True or False. Having a variety of names allows you to make your tests read the way you want. In fact, tests should be viewed as stories of how your code is expected to behave. It's often useful for other developers to read your tests to see what your actual code is capable of. The list of assertion methods can be found in the Python documentation for unittest.TestCase. The most common ones are:

failUnless(expr)

Ensure expr is true

assertEqual(expr1, expr2)

Ensure expr1 is equal to expr2

assertRaises(exception, callable, ...)

Make sure exception is raised by the callable. Note that callable here should be the name of a method or callable object, not an actual call, so you write e.g. self.assertRaises(AttributeError, myMethod, someParameter). Note lack of () after myMethod. If you included it, you'd get the exception raised in your test method, which is probably not what you want. Instead, the statement above will cause the unit testing framework to call myMethod(someParameter) (you can pass along any parameters you want after the calalble) and check for an AttributeError.

fail()

Simply fail. This is useful if a test has not yet been completed, or in an if statement inside a test where you know the test has failed.

In addition to the unit testing framework assertion methods, ZopeTestCase and PloneTestCase include some helper methods and variables to help you interact with Zope. It's instructive to read the source code for these two products, but briefly, the key variables you can use in unit tests are:

self.portal

The Plone portal the test is executing in

self.folder

The member folder of the member you are executing as

And the key methods are:

self.logout()

Log out, i.e. become anonymous

self.login()

Log in again. Pass a username to log in as a different user.

self.setRoles(roles)

Pass in a list of roles you want to have. For example, self.setRoles((Manager,)) lets you be manager for a while. How nice.

self.setPermissions(permissions)

Similarly, set a number of permissions for the test user in self.folder

self.setGroups(groups)

Set a list of groups for the current user.

Tips & Tricks

Good unit testing comes with experience. It's always useful to read the unit tests of code with which you are fairly familiar, to see how other people unit test. We'll cover a few hints here to get you thinking about how you approach your own tests:

• Don't be timid! Python, being a dynamic, weakly typed scripting language, lets you do all kinds of crazy things. You can rip a function right out from the Plone core and replace it with your own implementation in afterSetUp() or a test if that serves your testing purposes.
• Similarly, replacing things like the MailHost with dummy implementations may be the only way to test certain features. Look at CMFPlone/tests/dummy.py for some examples of dummy objects.
• Use tests to try things out. They are a safe environment. If you need to try something a bit out of the ordinary, writing them in a test is often the easiest way of seeing how something works.
• During debugging, you can insert print statements in tests to get traces in your terminal when you execute the tests. Don't check in code with printing tests, though. :)
• Similarly, the python debugger is very valuable inside tests. Putting import pdb; pdb.set_trace() inside your test methods lets you step through testing code and step into the code it calls. If you're not familiar with the python debugger, your life is incomplete. Read more in the python documentation.

Migrations

Sometimes we don't get it right on the first attempt. In fact, we rarely do. The first evidence of this are the following lines in 'RichDocument/__init__.py':

    sys.modules['Products.RichDocument.RichDocument'] = content.richdocument
    sys.modules['Products.RichDocument.FileAttachemnt'] = content.attachments
    sys.modules['Products.RichDocument.ImageAttachemnt'] = content.attachments
    sys.modules['Products.RichDocument.widgets'] = widget
    sys.modules['Products.RichDocument.ImagesManagerWidget'] = widget.images
    sys.modules['Products.RichDocument.AttachmentsManagerWidget'] = widget.attachments

These lines are necessary because we moved some of the RichDocument python files around to get in line with best practice. Unfortunately, there are still ZODBs out there which depend on the old layout, so we have to provide aliases to make sure that code doesn't break.

Things can move around inside content types, too, and sometimes you want to replace a whole content type with another, better version. Doing so is called "migration". You may have come across this term if you ever upgraded Plone. Because Plone has lots of state in the ZODB, we need to write migrations to ensure upgraded sites are consistent with freshly installed new ones. Migrations can be tricky, mostly because there are so many different ways in which you can mess up your site that we just didn't think about, and you'll see plenty of bloody, sweat and tears in Products/CMFPlone/migrations. Now that you've learnt about unit testing, you can also check out Products/CMFPlone/tests/testMigrations.py and see how we test migrations to make sure they do what we think they do, and that they fail gracefully when something unexpected happens.

Luckily, most products are not quite as complex as Plone itself and hence can get away with simpler migrations. Even more luckily, if what you are migrating are content types, ATContentTypes contains a migration framework that takes most of the work out of it for you.

Using Walkers and Migrators

ATContentTypes uses a registry of migrators, managed in the portal_atct tool to keep track of its migrators. This is quite flexible, and enables the migrators to be hidden behind a sensible user interface. However, it's also a lot of work. In most cases, a manually created External Method will suffice, since migrations are typically only called once.

The ATContentTypes migration framework uses the concept of walkers and migrators. A walker, as found in Products/ATContentTypes/migration/walker.py is responsible for finding content to migrate. The simplest migrator is the CatalogWalker, which does a catalog query for all content of a given type. This has two implications:

• The old content must (still) be in the catalog
• It finds and migrates all content of the source type, no special rules.

Later on, we'll see an example of a more flexible walker.

Migrators are simple classes that know how to migrate a given content type. The framework contains base classes to make writing migrators surprisingly easy.

RichDocument 1.0 was a very different beast internally to what you see now. It didn't use ATContentTypes, the display menu or many of the other advanced techniques you have learnt about in this tutorial. It was, however, the base type for a number of other types, including a Page type (not to be confused with Plone's Page type, which is really just a renamed Document... hey, Limi, I had the name first!) in a customer project. Page was like an old-fashioned RichDocument with an added subtitle field. We later decided we didn't care about the subtitle, and we wanted to migrate it to a standard RichDocument to make it easier to maintain. This posed a few challenges:

• We had to change all the Page objects to RichDocument ones. There were far too many Pages in the portal to do this manually
• RichDocument didn't use to use the "display" menu for the float/preview images options. Instead, it had an IntegerField displayImages that took on different values: 0 for basic view, 1 for float and 2 for display preview box.
• RichDocument used to store plain old Images instead of ImageAttachment objects for its attached images.

The migration thus had to deal with all of this. We used the following external method, called 'Page/Extensions/migrate.py':

  from Products.CMFCore.utils import getToolByName
  from StringIO import StringIO
  from Products.ATContentTypes.migration.walker import CatalogWalker
  from Products.ATContentTypes.migration.migrator import CMFFolderMigrator, CMFItemMigrator

  class RichDocumentMigrator(CMFItemMigrator):
      """Base class to migrate to RichDocument or a derivative. Takes care of 
      contained images.
      """

      def migrate_imageAttachments(self):
          """Previously, we used standard images. Now we use our own
          ImageAttachment type.
          """
          for img in self.old.objectValues(('Image', 'ATImage',)):
              self.new.invokeFactory('ImageAttachment', img.getId())
              newImg = getattr(self.new, img.getId())
              newImg.setImage(img.getImage())

      def migrate_imageSettings(self):
          """Previously, we used an integer variable to determine whether to
          display floats or previews or the standard document view. Now, we
          use the "display" menu for this.
          """
          displayImages = self.old.getDisplayImages()
          self.new.setDisplayImages(False)
          if displayImages == 1:
              self.new.setLayout('richdocument_view_float')
          elif displayImages == 2:
              self.new.setLayout('richdocument_view_preview')
          else:
              self.new.setLayout('richdocument_view')

  class PageMigrator(RichDocumentMigrator):
      walkerClass = CatalogWalker
      src_meta_type = 'Page'
      src_portal_type = 'Page'
      dst_meta_type = 'RichDocument'
      dst_portal_type = 'RichDocument'
      map = {'getRawText' : 'setText'}

  def migrate(self):
      """Run the migration"""

      out = StringIO()
      print >> out, "Starting migration"

      portal_url = getToolByName(self, 'portal_url')
      portal = portal_url.getPortalObject()

      migrators = (PageMigrator,)

      for migrator in migrators:
          walker = migrator.walkerClass(portal, migrator)
          walker.go(out=out)
          print >> out, walker.getOutput()

      print >> out, "Migration finished"
      return out.getvalue()

This method was set up in portal_skins/custom as a new External Method, with the id migrateTypes, module Page.migrate and function name migrate. To run it, we simply had to hit the Test tab of the External Method in the ZMI, or call it on http://localhost:8080/plone-site/migrateTypes.

WARNING: Migrations are typically not easy to Undo in the ZMI, because they tend to span many transactions. It's therefore vital that you keep a backup of your site pre-migration to ensure you can roll back if something goes wrong!

The migration framework works by instantiating a walker with a migrator to apply to all the objects it finds. This is what happens in the migrate() function. Migrators are classes usually derived from CMFFolderMigrator or CMFItemMigrator, or an intermediary. You can see these and the various ATContentTypes migration base classes in Products/ATContentTypes/migration/migrator.py and Products/ATContentTypes/migration/atctmigrator.py. Basically, a CMFItemMigrator migrates a standard CMF type (including those made with Archetypes), including all standard metadata, local roles etc. A CMFFolderMigrator does this as well as ensure folder contents are migrated with the folder.

There are three ways of performing custom migration:

• Any method in the class with a name starting with migrate_ will be called automatically. This is similar to how test... methods are called during unit testing. If you follow the hierarchy of base classes from CMFItemMigrator, you will see a number of such methods taking care of various parts of the migration process. In these methods, as well as the other automatically called methods described below, you have two basic variables available: self.old is the old object being replaced, and self.new is the new object being created.

  If you need more control there are some additional method prefixes you can use:

  • beforeChange_... methods are called before the migration takes place, meaning before self.new is created. Hence, they must work in terms of self.old.
  • last_migrate_... methods are called just before the migrator finishes migrating an object.
• The method custom() is called after migrate_... methods, but before the last_migrate_... methods. The default implementation is empty, but you can override it to perform any custom migration.
• The easiest way of performing migrations is with the map class variable. Here, you can define a map of attributes and/or methods that should get migrated. From the docstring of migrate_withMap() in 'Products/ATContentTypes/migrate/migrator.py':

      """Migrates other attributes from obj.__dict__ using a map
  
      The map can contain both attribute names and method names
  
      'oldattr' : 'newattr'
          new.newattr = oldattr
      'oldattr' : ''
          new.oldattr = oldattr
      'oldmethod' : 'newattr'
          new.newattr = oldmethod()
      'oldattr' : 'newmethod'
          new.newmethod(oldatt)
      'oldmethod' : 'newmethod'
          new.newmethod(oldmethod())
      """
  

In the code above, we define a base class for old-to-new RichDocument migrations. Note that it uses CMFItemMigrator instead of CMFFolderMigrator even though RichDocument is folderish, because we take care of the Image-to-ImageAttachment translation in the automatically called migrate_imageAttachments(). The method migrate_imageSettings() takes care of the change from using an integer to hold the display mode to using the "display" menu.

The PageMigrator now becomes very simple. It sets the source and destination portal- and meta-types (in most cases, these are the same - check the definition of your Archetypes class to be sure; in ATContentTypes core, they are actually different: the meta type is ATImage for the portal type Image and so on) for the walker to search for, and defines the map that migrates the value of the getRawText accessor to the setText mutator.

Migrating between different versions of the same type

Perhaps more common than migrating from one content type to a completely different one is the case where you have changed the internal organisation of a content type. You may have renamed fields, or changed the internal conventions for how your data is stored.

archetype_tool provides some means of performing basic migrations, via its Update schema tab. This will make sure that the in-ZODB schemata of your types are synced with the current story on the filesystem. However, you may end up losing data this way. If you rename a field, Archetypes' storage will store it in a different location (e.g. a different attribute). Hence, existing instances will have a blank/default value for these fields. To retain the old data, you will probably need migration.

The contentmigration product was written to help this scenario. It extends the ATContentTypes migrator in a few ways. From its README:

• A CustomQueryWalker can be used to specify a more specific catalog query for a walker to use (e.g. which content to actually migrate). This can be used with any migrator.
• A BaseInlineMigrator is similar to BaseMigrator, but does not migrate by copying the old object to a temporary location, creating a new object and applying migration methods. Instead, migration methods are applied in-place. This simplifies the code significantly, because attributes, local roles etc. does not need to be copied over.

  Note that whereas BaseMigrator works in terms of self.old and self.new as the objects being migrated, BaseInlineMigrator only has a single object, stored in self.obj. This can be used with any walker.

• An extension of this class called FieldActionMigrator uses the action-based migration framework for Archetypes fields, found in field.py. Please refer to that file for full details, but briefly, you specify a list of attributes to migrate at the storage level, instructing the migrator whether to rename, transform, unset or change the storage for an attribute.

You can get contentmigration from the Collective svn repository. Note that it is not a product to install in Plone. Instead, you can call it from your own migration methods, with statements like:

  from Products.contentmigration.walker import CustomQueryWalker
  from Products.contentmigration.migrator import FieldActionMigrator
  from Products.Archetypes.public import *

  class MyMigrator(FieldActionMigrator):
      src_portal_type = 'MyType'
      src_meta_type = 'MyType'

      fieldActions = ({'fieldName'    : 'someField',
                       'storage'      : AttributeStorage(),
                       'newFieldName' : 'renamedField',
                       'newStorage'   : AnnotationStorage(),
                       'transform'    : lambda obj, val, **kw: val + 10,
                       })

The CustomQueryWalker is a generic walker that you can use in other migration too. It allows you specify a specific catalog query to use to find objects. In this way, you can restrict migration to a subset of the content in your site. You initialise it with a query like this:

  walker = CustomQueryWalker(portal, MyMigrator, 
                             query = {'path' : '/some/path'})
  walker.go()

Note that the src_portal_type and src_meta_type still exist in the migrator (MyMigrator in this case) and are inserted in the query. In fact, they override any setting of meta_type or portal_type in the query itself.

The BaseInlineMigrator and its extension FieldActionMigrator are the base classes you can use for "inline" migration - that is, migration that doesn't migrate from one type to another, but alters the fields in the current object. You can use migrate_..., beforeChange_..., last_migrate_... methods, and the custom() override here too. Note that the map class variable is not available, because it is superseded by the fieldActions variable and is generally less useful for inline migrations. Also note that any custom logic you implement should use self.obj instead of self.old and self.new, since there is only one object taking part in the migration.

The fieldActions method of migration is similar to the map feature of the BaseMigrator, but is more powerful. It uses a list of actions, specified as dictionaries. Actions work on fields, accessed with storages. Note that the fields and storages do not need to exist in the current schema. Hence, if a field name has been changed, and/or the storage of the field changed, you can use actions like the one above to specify the change. Actions can also apply transformations via a callback method (the lambda above), and execute a callback method before or after the migration of a specific action.

To learn more about field actions, see contentmigration/field.py. For examples, see contentmigration/tests/testATFieldMigration.py and contentmigration/tests/cmtc.py.

The following techniques may be worth adding to this tutorial:

o i18n - how to properly internationalise your templates (especially after Sebastien has patched RichDocument to fix i18n support)

o More details on using IConstrainTypes to restrict addable types to a folder

o Writing custom widgets - what was involved in writing the ImagesManagerWidget and AttachmentsManagerWidget widgets, including:

o Registering new form controller actions on atct_edit