Test-driven development

One of the greatest things that Zope 3 has established is a culture of test-driven development. Because Zope 3 components tend to be small and not dependent on a large framework or (typically) a running application server, tests are easier to write and execute faster. Most Zope 3 testing happens in the form of testable documentation - DocTests - which tell the story of how a component should be used along with testable examples.

The testing tutorial explains the philosophy behind test-driven development and the tools and techniques available in Zope. It is required reading if you are not familiar with testing in Zope, and probably quite useful even if you are.

Testing strategy

Tests were (largely) written against interfaces and stub implementations, before the actual functionality was written. One of the first test cases to be created was test_adapters.py, which simply verifies that the various adapter registrations are in effect. This is obviously an integration test (using PloneTestCase), since it is verifying what happens on a "normal" Zope start-up.

You will also notice tests named after the three content types, test_department.py, test_employee.py and test_project.py. Each of these contains tests that verify the given type is available and can be instantiated and edited. This catches errors in Archetypes registrations or schemas. There are then further tests for the membrane integration and for the adapters to the canonical interfaces IDepartment, IEmployee and IProject. Lastly, non-trivial methods in content types and relevant adapters are given their own test fixtures.

By being systematic and diligent with tests, many, many bugs were caught and dealt with before they ever hit a live system. Of course, this does not replace in-browser acceptance testing, which was also performed regularly.

At the time of writing, there are no zope.testbrowser based functional tests for the user interface. That is regrettable - and this is an open source project after all, so feel free to contribute some!

Test set-up

You will find b-org's tests in the tests module. Most of these use are DocTest integration tests, using PloneTestCase. Make sure you use a recent version of PloneTestCase (or svn trunk) since there have been some recent changes in how Zope 3 components (or rather, ZCML registrations) are loaded for test runs. The upshot is that with PloneTestCase, things should "just work" for integration testing - components you have defined in ZCML in your products will be loaded as they would when Zope is started.

The file base.py contains an insulating base class for b-org tests, called BorgTestCase and its sister-class BorgFunctionalTesetCase. When imported, this file will trigger the setup of a Plone site with the membrane and borg extension profiles installed, as such:

from Testing import ZopeTestCase

# Let Zope know about the two products we require above-and-beyond a basic
# Plone install (PloneTestCase takes care of these).
ZopeTestCase.installProduct('membrane')
ZopeTestCase.installProduct('borg')

# Import PloneTestCase - this registers more products with Zope as a side effect
from Products.PloneTestCase.PloneTestCase import PloneTestCase
from Products.PloneTestCase.PloneTestCase import FunctionalTestCase
from Products.PloneTestCase.PloneTestCase import setupPloneSite

# Set up a Plone site, and apply the membrane and borg extension profiles
# to make sure they are installed.
setupPloneSite(extension_profiles=('membrane:default', 'borg:default'))

Integration and unit tests

Most of the tests are integration test that are set up like so:

import unittest
from Testing.ZopeTestCase import ZopeDocTestSuite

from base import BorgTestCase
from utils import optionflags

def test_creation():
    """Test that departments can be created an initiated.

    >>> self.setRoles(('Manager',))
    >>> id = self.portal.invokeFactory('Department', 'dept')
    >>> dept = self.portal.dept

    Set roles.

    >>> dept.setRoles(('Reviewer',))
    >>> tuple(dept.getRoles())
    ('Reviewer',)

    Add an employee and set it as a manager.

    >>> id = dept.invokeFactory('Employee', 'emp')
    >>> dept.setManagers((dept.emp.UID(),))
    >>> tuple(dept.getManagers())
    (<Employee at ...>,)
    """

...

def test_suite():
    return unittest.TestSuite((
            ZopeDocTestSuite(test_class=BorgTestCase,
                             optionflags=optionflags),
        ))

There is also a plain-python (no loading of Zope necessary, which is much faster) unit test for the password digest in test_passwords.py. This is appropriate because the functionality under test does not depend on the Zope application server or database being loaded. Use plain-python (or perhaps rather, plain Zope 3) tests whenever you can to reduce interdependencies and test load times:

import unittest

from zope.testing.doctestunit import DocTestSuite
from utils import configurationSetUp, configurationTearDown, optionflags

def test_passwords_hashed():
    """Check that passwords are hashed

    We expect that the password will be saved as a SHA-1 digest.

    >>> import sha
    >>> digest = sha.sha('secret').digest()

    Set a password.

    >>> from Products.borg.content.employee import Employee
    >>> e = Employee('emp')
    >>> e.setPassword('secret')

    The value is stored in an annotation, and there is no direct way to
    access it (deliberately). Thus, check the annotation directly.

    >>> from zope.app.annotation.interfaces import IAnnotations
    >>> from Products.borg.config import PASSWORD_KEY
    >>> annotations = IAnnotations(e)
    >>> password = annotations[PASSWORD_KEY]

    Ensure it is what we expected:

    >>> password == digest
    True
    """

...

def test_suite():
    return unittest.TestSuite((
            DocTestSuite(setUp=configurationSetUp,
                         tearDown=configurationTearDown,
                         optionflags=optionflags),
        ))

The functions configurationSetUp() and configurationTearDown() are defined in utils.py and are used to load specific ZCML files that enable the test environment to function. This is necessary because without PloneTestCase's integration test layer in effect, there will be no compnent registrations when the tests are run! This may be more cumbersome (though in reality, the same set of components tend to be used), but also allows better control over the environment in which test are run, in addition to (much) faster test execution times.

From utils.py:

import doctest
from zope.app.tests import placelesssetup
from zope.configuration.xmlconfig import XMLConfig

# Standard options for DocTests
optionflags =  (doctest.ELLIPSIS |
                doctest.NORMALIZE_WHITESPACE |
                doctest.REPORT_ONLY_FIRST_FAILURE)


def configurationSetUp(self):
    """Set up Zope 3 test environment
    """

    placelesssetup.setUp()

    # Ensure that the ZCML registrations in membrane and borg are in effect
    # Also ensure the Five directives and permissions are available

    import Products.Five
    import Products.membrane
    import Products.borg

    XMLConfig('configure.zcml', Products.Five)()
    XMLConfig('meta.zcml', Products.Five)()

    XMLConfig('configure.zcml', Products.membrane)()
    XMLConfig('configure.zcml', Products.borg)()

def configurationTearDown(self):
    """Tear down Zope 3 test environment
    """

    placelesssetup.tearDown()

You will also find a regular unit test in test_setup.py, simply because this was quicker to write:

from base import BorgTestCase

from Products.membrane.interfaces import ICategoryMapper
from Products.membrane.config import ACTIVE_STATUS_CATEGORY
from Products.membrane.utils import generateCategorySetIdForType

from Products.borg.config import LOCALROLES_PLUGIN_NAME, PLACEFUL_WORKFLOW_POLICY

class TestProductInstall(BorgTestCase):

    def afterSetUp(self):
        self.types = ('Department', 'Employee', 'Project',)

    def testTypesInstalled(self):
        for t in self.types:
            self.failUnless(t in self.portal.portal_types.objectIds(),
                            '%s content type not installed' % t)

    ...

def test_suite():
    from unittest import TestSuite, makeSuite
    suite = TestSuite()
    suite.addTest(makeSuite(TestProductInstall))
    return suite

Finally, there is an docstring DocTest for the ExtensibleSchemaSupport class. This is because this class if largely standalone (it probably shouldn't be b-org at all, but in a more general module, except Archetypes will gain similar functionality of its own for Plone 3.0) and the test provided important documentation in the class' docstring.

The class looks like this:

class ExtensibleSchemaSupport(Base):
    """Mixin class to support instance-based schemas.

    Note: you must mix this in before BaseFolder or BaseContent, e.g.:

    class Foo(ExtensibleSchemaSupport, BaseContent):
        ...

    This is based on Archetype's VariableSchemaSupport.

    Define a content type with a marker interface:

    >>> from zope.interface import Interface, implements
    >>> class IMyType(Interface):
    ...     pass

    >>> from Products.Archetypes.atapi import *
    >>> from Products.borg.content.schema import ExtensibleSchemaSupport
    >>> class MyType(ExtensibleSchemaSupport, BaseObject):
    ...     implements(IMyType)
    ...     schema = BaseSchema.copy() + Schema((StringField('foo'),))
    >>> registerType(MyType, 'testing')

    Create a schema extender:

    ...

    """
    implements(IExtensibleSchemaProvider)

    ...

And the test runner, in test_schema.py, contains:

import unittest
from Testing.ZopeTestCase import ZopeDocTestSuite

from base import BorgTestCase
from utils import optionflags

def test_suite():
    return unittest.TestSuite((
            ZopeDocTestSuite('Products.borg.content.schema',
                             test_class=BorgTestCase,
                             optionflags=optionflags),
        ))