As all good developers know, automated tests are very important! If you are not comfortable with automated testing and test-driven development, you should read the Plone testing tutorial. In this section, we will assume you are familiar with Plone testing basics, and show some tests that are particularly relevant to our example types.

Firstly, we will add a few unit tests. Recall that unit tests are simple tests for a particular function or method, and do not depend on an outside environment being set up. As a rule of thumb, if something can be tested with a simple unit test, do so, because:

• Unit tests are quick to write
• They are also quick to run
• Because they are more isolated, you are less likely to have tests that pass or fail due to incorrect assumptions or by luck
• You can usually test things more thoroughly and exhaustively with unit tests than with (slower) integration tests

You'll typically supplement a larger number of unit tests with a smaller number of integration tests, to ensure that your application's correctly wired up and working.

That's the theory, at least. When we're writing content types, we're often more interested in integration test, because a type schema and FTI are more like configuration of the Plone and Dexterity frameworks than imperative programming. We can't "unit test" the type's schema interface, but we can and should test that the correct schema is picked up and used when our type is installed. We will often write unit tests (with mock objects, where required) for custom event handlers, default value calculation functions and other procedural code.

In that spirit, let's write some unit tests for the default value handler and the invariant in program.py. We'll add the directory tests, with an __init__.py and a file test_program.py that looks like this:

import unittest
import datetime

from example.conference.program import startDefaultValue
from example.conference.program import endDefaultValue
from example.conference.program import IProgram
from example.conference.program import StartBeforeEnd

class MockProgram(object):
    pass

class TestProgramUnit(unittest.TestCase):
    """Unit test for the Program type
    """
    
    def test_start_defaults(self):
        data = MockProgram()
        default_value = startDefaultValue(data)
        today = datetime.datetime.today()
        delta = default_value - today
        self.assertEquals(6, delta.days)

    def test_end_default(self):
        data = MockProgram()
        default_value = endDefaultValue(data)
        today = datetime.datetime.today()
        delta = default_value - today
        self.assertEquals(9, delta.days)
    
    def test_validate_invariants_ok(self):
        data = MockProgram()
        data.start = datetime.datetime(2009, 1, 1)
        data.end = datetime.datetime(2009, 1, 2)
        
        try:
            IProgram.validateInvariants(data)
        except:
            self.fail()
    
    def test_validate_invariants_fail(self):
        data = MockProgram()
        data.start = datetime.datetime(2009, 1, 2)
        data.end = datetime.datetime(2009, 1, 1)
        
        try:
            IProgram.validateInvariants(data)
            self.fail()
        except StartBeforeEnd:
            pass
    
    def test_validate_invariants_edge(self):
        data = MockProgram()
        data.start = datetime.datetime(2009, 1, 2)
        data.end = datetime.datetime(2009, 1, 2)
        
        try:
            IProgram.validateInvariants(data)
        except:
            self.fail()

def test_suite():
    return unittest.defaultTestLoader.loadTestsFromName(__name__)

This is a simple test using the Python standard library's unittest module. There are a few things to note here:

• We have created a dummy class to simulate a Program instance. It doesn't contain anything at all, but we set some attributes onto it for certain tests. This is a very simple way to do mocks. There are much more sophisticated mock testing approaches, but starting simple is good.
• Each test is self contained. There is no test layer or test case setup/tear-down.
• We use the defaultTestLoader to load all test classes in the module automatically. The test runner will look for modules in the tests package with names starting with test* that have a test_suite() method to get test suites.

To run the tests, we can do:

$ ./bin/text example.conference

Hopefully it should show five passing tests.

This uses the testrunner configured via the [test] part in our buildout.cfg. This provides better test reporting and a few more advanced options (like output colouring). We could also use the built-in test runner in the instance script, e.g. with ./bin/instance test -s example.conference.

To run just this test suite, we can do:

$ ./bin/test example.conference -t TestProgramUnit

This is useful when we have other test suites that we don't want to run, e.g. because they are integration tests and require lengthy setup.

To get a report about test coverage, we can run:

$ ./bin/test example.conference --coverage

Test coverage reporting is important. If you have a module with low test coverage, it means that your tests do not cover many of the code paths in those modules, and so are less useful for detecting bugs or guarding against future problems. Aim for 100%.

We'll now add some integration tests for our type. These should ensure that the package installs cleanly, and that our custom types are addable in the right places and have the right schemata, at the very least.

To help manage test setup, we'll make use of the Zope test runner's concept of layers. Layers allow common test setup (such as configuring a Plone site and installing a product) to take place once and be re-used by multiple test cases. Those test cases can still modify the environment, but their changes will be torn down and the environment reset to the layer's initial state between each test, facilitating test isolation.

As the name implies, layers are, erm, layered. One layer can extend another. If two test cases in the same test run use two different layers with a common ancestral layer, the ancestral layer is only set up and torn down once.

We'll use collective.testcaselayer to write and manage layers. We need to depend on this, so in setup.py, we have:

      install_requires=[
          ...
          'collective.testcaselayer',
      ],

Don't forget to re-run buildout after making changes to setup.py.

We then add our own layer to tests/layer.py:

from Products.PloneTestCase import ptc
import collective.testcaselayer.ptc

ptc.setupPloneSite()

class IntegrationTestLayer(collective.testcaselayer.ptc.BasePTCLayer):

    def afterSetUp(self):
        # Install the example.conference product
        self.addProfile('example.conference:default')

Layer = IntegrationTestLayer([collective.testcaselayer.ptc.ptc_layer])

This extends a base layer that sets up Plone, and adds some custom layer setup for our package, in this case installing the example.conference extension profile. We could also perform additional setup here, such as creating some initial content or setting the default roles for the test run. See the collective.testcaselayer documentation for more details.

To use the layer, we can create a new test case based on PloneTestCase that uses our layer. We'll add one to test_program.py first. (In the code snippet below, the unit test we created previously has been removed to conserve space.)

import unittest

from zope.component import createObject
from zope.component import queryUtility

from plone.dexterity.interfaces import IDexterityFTI

from Products.PloneTestCase.ptc import PloneTestCase
from example.conference.tests.layer import Layer

from example.conference.program import IProgram

class TestProgramIntegration(PloneTestCase):
    
    layer = Layer
    
    def test_adding(self):
        self.folder.invokeFactory('example.conference.program', 'program1')
        p1 = self.folder['program1']
        self.failUnless(IProgram.providedBy(p1))
    
    def test_fti(self):
        fti = queryUtility(IDexterityFTI, name='example.conference.program')
        self.assertNotEquals(None, fti)
    
    def test_schema(self):
        fti = queryUtility(IDexterityFTI, name='example.conference.program')
        schema = fti.lookupSchema()
        self.assertEquals(IProgram, schema)
    
    def test_factory(self):
        fti = queryUtility(IDexterityFTI, name='example.conference.program')
        factory = fti.factory
        new_object = createObject(factory)
        self.failUnless(IProgram.providedBy(new_object))
    
    def test_view(self):
        self.folder.invokeFactory('example.conference.program', 'program1')
        p1 = self.folder['program1']
        view = p1.restrictedTraverse('@@view')
        sessions = view.sessions()
        self.assertEquals(0, len(sessions))
    
    def test_start_end_dates_indexed(self):
        self.folder.invokeFactory('example.conference.program', 'program1')
        p1 = self.folder['program1']
        p1.start = datetime.datetime(2009, 1, 1, 14, 01)
        p1.end = datetime.datetime(2009, 1, 2, 15, 02)
        p1.reindexObject()
        
        result = self.portal.portal_catalog(path='/'.join(p1.getPhysicalPath()))
        
        self.assertEquals(1, len(result))
        self.assertEquals(result[0].start, DateTime('2009-01-01T14:01:00'))
        self.assertEquals(result[0].end, DateTime('2009-01-02T15:02:00'))
    
    def test_tracks_indexed(self):
        self.folder.invokeFactory('example.conference.program', 'program1')
        p1 = self.folder['program1']
        p1.tracks = ['Track 1', 'Track 2']
        p1.reindexObject()
        
        result = self.portal.portal_catalog(Subject='Track 2')
        
        self.assertEquals(1, len(result))
        self.assertEquals(result[0].getURL(), p1.absolute_url())

def test_suite():
    return unittest.defaultTestLoader.loadTestsFromName(__name__)

This illustrates a basic set of tests that make sense for most content types. There are many more things we could test (for example, we could test the add permissions more thoroughly, and we ought to test the sessions() method on the view with some actual content!), but even this small set of integration tests tells us that our product has installed, that the content type is addable, that it has the right factory, and that instances of the type provide the right schema interface.

There are some important things to note about this test case:

• We extend PloneTestCase, which means we have access to a full Plone integration test environment. See the testing tutorial for more details.
• We set the layer attribute to our custom layer. This means that all tests in our test case will have the example.conference:default profile installed.
• We test that the content is addable (here, as a normal member in their member folder, since that is the default security context for the test - use self.setRoles(['Manager']) to get the Manager role and self.portal to access the portal root), that the FTI is installed and can be located, and that both the FTI and instances of the type know about the correct type schema.
• We also test that the view can be looked up and has the correct methods. We've not included a full functional test (e.g. using zope.testbrowser) or any other front-end testing here. If you require those, take a look at the testing tutorial.
• We also test that our custom indexers are working, by creating an appropriate object and searching for it again. Note that we need to reindex the object after we've modified it so that the catalog is up to date.
• The defaultTestLoader will find this test and load it, just as it found the TestProgramUnit test case.

To run our tests, we can still do.

$ ./bin/test example.conference

You should now notice layers being set up and torn down. Again, use the -t option to run a particular test case (or test method) only.

The other tests are similar. We have tests/test_session.py to test the Session type:

import unittest

from zope.component import createObject
from zope.component import queryUtility

from plone.dexterity.interfaces import IDexterityFTI

from Products.PloneTestCase.ptc import PloneTestCase
from example.conference.tests.layer import Layer

from example.conference.session import ISession
from example.conference.session import possible_tracks

class TestSessionIntegration(PloneTestCase):
    
    layer = Layer
    
    def test_adding(self):
        
        # We can't add this directly
        self.assertRaises(ValueError, self.folder.invokeFactory, 'example.conference.session', 'session1')
        
        self.folder.invokeFactory('example.conference.program', 'program1')
        p1 = self.folder['program1']
        
        p1.invokeFactory('example.conference.session', 'session1')
        s1 = p1['session1']
        self.failUnless(ISession.providedBy(s1))
    
    def test_fti(self):
        fti = queryUtility(IDexterityFTI, name='example.conference.session')
        self.assertNotEquals(None, fti)
    
    def test_schema(self):
        fti = queryUtility(IDexterityFTI, name='example.conference.session')
        schema = fti.lookupSchema()
        self.assertEquals(ISession, schema)
    
    def test_factory(self):
        fti = queryUtility(IDexterityFTI, name='example.conference.session')
        factory = fti.factory
        new_object = createObject(factory)
        self.failUnless(ISession.providedBy(new_object))
    
    def test_tracks_vocabulary(self):
        self.folder.invokeFactory('example.conference.program', 'program1')
        p1 = self.folder['program1']
        p1.tracks = ['T1', 'T2', 'T3']
        
        p1.invokeFactory('example.conference.session', 'session1')
        s1 = p1['session1']
        
        vocab = possible_tracks(s1)
        
        self.assertEquals(['T1', 'T2', 'T3'], [t.value for t in vocab])
        self.assertEquals(['T1', 'T2', 'T3'], [t.token for t in vocab])
    
    def test_catalog_index_metadata(self):
        self.failUnless('track' in self.portal.portal_catalog.indexes())
        self.failUnless('track' in self.portal.portal_catalog.schema())
    
    def test_workflow_installed(self):
        self.folder.invokeFactory('example.conference.program', 'program1')
        p1 = self.folder['program1']
        
        p1.invokeFactory('example.conference.session', 'session1')
        s1 = p1['session1']
        
        chain = self.portal.portal_workflow.getChainFor(s1)
        self.assertEquals(('example.conference.session_workflow',), chain)

def test_suite():
    return unittest.defaultTestLoader.loadTestsFromName(__name__)

Notice here how we test that the Session type cannot be added directly to a folder, and that it can be added inside a program. We also add a test for the possible_tracks() vocabulary method, as well as tests for the installation of the track index and metadata column and the custom workflow.

And in tests/test_presenter.py, we test the Presenter type:

import unittest

from zope.component import createObject
from zope.component import queryUtility

from plone.dexterity.interfaces import IDexterityFTI

from Products.PloneTestCase.ptc import PloneTestCase
from example.conference.tests.layer import Layer

from example.conference.presenter import IPresenter

class TestPresenterIntegration(PloneTestCase):
    
    layer = Layer
    
    def test_adding(self):
        self.folder.invokeFactory('example.conference.presenter', 'presenter1')
        p1 = self.folder['presenter1']
        self.failUnless(IPresenter.providedBy(p1))
    
    def test_fti(self):
        fti = queryUtility(IDexterityFTI, name='example.conference.presenter')
        self.assertNotEquals(None, fti)
    
    def test_schema(self):
        fti = queryUtility(IDexterityFTI, name='example.conference.presenter')
        schema = fti.lookupSchema()
        self.assertEquals(IPresenter, schema)
    
    def test_factory(self):
        fti = queryUtility(IDexterityFTI, name='example.conference.presenter')
        factory = fti.factory
        new_object = createObject(factory)
        self.failUnless(IPresenter.providedBy(new_object))
    
def test_suite():
    return unittest.defaultTestLoader.loadTestsFromName(__name__)

Faster tests with Roadrunner

You will have noticed that running unit tests was much quicker than running integration tests. That is unfortunate, but to be expected: the integration test setup basically requires starting all of Zope and configuring a Plone site.

Luckily, there is a tool that we can use to speed things up, and if you've been following along the tutorial, you already have it in your buildout: Roadrunner. This is a command that takes the place of ./bin/instance test that preloads the Zope environment and allows you to re-run tests much faster.

To run our tests with roadrunner, we would do:

$ ./bin/roadrunner -s example.conference

This runs the tests once, and then drops to the Roadrunner prompt:

rr>

Simply hitting enter here, or typing a command like test -s example.conference will re-run your tests, this time taking much less time.

Roadrunner works best when you are adding and debugging your tests. For example, it's a very quick way to get to a pdb prompt: just set a breakpoint in your test with import pdb; pdb.set_trace() and re-run it in roadrunner. You can then step into your test code and the code under test.

Roadrunner should pick up changes to your tests automatically. However, it may not pick up changes to your application code, grokked components or ZCML files. If it doesn't, you'll need to exit the Roadrunner prompt and restart.

Mock testing is a powerful approach to testing that lets you make assertions about how the code under test is interacting with other system modules. It is often useful when the code you want to test is performing operations that cannot be easily asserted by looking at its return value.

In our example product, we have an event handler like this:

@grok.subscribe(IPresenter, IObjectAddedEvent)
def notifyUser(presenter, event):
    acl_users = getToolByName(presenter, 'acl_users')
    mail_host = getToolByName(presenter, 'MailHost')
    portal_url = getToolByName(presenter, 'portal_url')
    
    portal = portal_url.getPortalObject()
    sender = portal.getProperty('email_from_address')
    
    if not sender:
        return
    
    subject = "Is this you?"
    message = "A presenter called %s was added here %s" % (presenter.title, presenter.absolute_url(),)
    
    matching_users = acl_users.searchUsers(fullname=presenter.title)
    for user_info in matching_users:
        email = user_info.get('email', None)
        if email is not None:
            mail_host.secureSend(message, email, sender, subject)

If we want to test that this sends the right kind of email message, we'll need to somehow inspect what is passed to secureSend(). The only way to do that is to replace the MailHost object that is acquired when getToolByName(presenter, 'MailHost') is called, with something that performs that assertion for us.

If we wanted to write an integration test, we could use PloneTestCase to execute this event handler, e.g. by firing the event manually, and temporarily replace the MailHost object in the root of the test case portal (self.portal) with a dummy that raised an exception if the wrong value was passed.

However, such integration tests can get pretty heavy handed, and sometimes it is difficult to ensure that it works in all cases. In the approach outlined above, for example, we would miss cases where no mail was sent at all.

Enter mock objects. A mock object is a "test double" that knows how and when it ought to be called. The typical approach is as follows:

• Create a mock object.
• The mock object starts out in "record" mode.
• Record the operations that you expect the code under test perform on the mock object. You can make assertions about the type and value of arguments, the sequence of calls, or the number of times a method is called or an attribute is retrieved or set.
• You can also give your mock objects behaviour, e.g. by specifying return values or exceptions to be raised in certain cases.
• Initialise the code under test and/or the environment it runs in so that it will use the mock object rather than the real object. Sometimes this involves temporarily "patching" the environment.
• Put the mock framework into "replay" mode.
• Run the code under test.
• Apply any assertions as you normally would.
• The mock framework will raise exceptions if the mock objects are called incorrectly (e.g. with the wrong arguments, or too many times) or insufficiently (e.g. an expected method was not called).

There are several Python mock object frameworks. Dexterity itself users a powerful one called mocker, via the plone.mocktestcase integration package. You are encouraged to read the documentation for those two packages to better understand how mock testing works, and what options are available.

Take a look at the tests in plone.dexterity if you're looking for more examples of mock tests using plone.mocktestcase.

To use the mock testing framework, we first need to depend on plone.mocktestcase. As usual, we add it to setup.py and re-run buildout.

      install_requires=[
          ...
          'plone.mocktestcase',
      ],

As an example test case, consider the following class in test_presenter.py:

import unittest

...

from plone.mocktestcase import MockTestCase
from zope.app.container.contained import ObjectAddedEvent
from example.conference.presenter import notifyUser

class TestPresenterUnit(MockTestCase):
    
    def test_notify_user(self):
        
        # dummy presenter
        presenter = self.create_dummy(
                __parent__=None,
                __name__=None,
                title="Jim",
                absolute_url = lambda: 'http://example.org/presenter',
            )
        
        # dummy event
        event = ObjectAddedEvent(presenter)
        
        # search result for acl_users
        user_info = [{'email': 'jim@example.org', 'id': 'jim'}]
        
        # email data
        message = "A presenter called Jim was added here http://example.org/presenter"
        email = "jim@example.org"
        sender = "test@example.org"
        subject = "Is this you?"
        
        # mock tools/portal
        
        portal_mock = self.mocker.mock()
        self.expect(portal_mock.getProperty('email_from_address')).result('test@example.org')
        
        portal_url_mock = self.mocker.mock()
        self.mock_tool(portal_url_mock, 'portal_url')
        self.expect(portal_url_mock.getPortalObject()).result(portal_mock)
        
        acl_users_mock = self.mocker.mock()
        self.mock_tool(acl_users_mock, 'acl_users')
        self.expect(acl_users_mock.searchUsers(fullname='Jim')).result(user_info)
        
        mail_host_mock = self.mocker.mock()
        self.mock_tool(mail_host_mock, 'MailHost')
        self.expect(mail_host_mock.secureSend(message, email, sender, subject))
        
        
        # put mock framework into replay mode
        self.replay()
        
        # call the method under test
        notifyUser(presenter, event)

        # we could make additional assertions here, e.g. if the function
        # returned something. The mock framework will verify the assertions
        # about expected call sequences.

...

def test_suite():
    return unittest.defaultTestLoader.loadTestsFromName(__name__)

Note that the other tests in this module have been removed for the sake of brevity.

If you are not familiar with mock testing, it may take a bit of time to get your head around what's going on here. Let's run though the test:

• First, we create a dummy presenter object. This is not a mock object, it's just a class with the required minimum set of attributes, created using the create_dummy() helper method from the MockTestCase base class. We use this type of dummy because we are not interested in making any assertions on the presenter object: it is used as an "input" only.
• Next, we create a dummy event. Here we have opted to use a standard implementation from zope.app.container.
• We then define a few variables that we will use in the various assertions and mock return values: the user data that will form our dummy user search results, and the email data passed to the mail host.
• Next, we create mocks for each of the tools that our code needs to look up. For each, we use the expect() method from MockTestCase to make some assertions. For example, we expect that getPortalObject() will be called (once) on the portal_url tool, and it should return another mock object, the portal_mock. On this, we expect that getProperty() is called with an argument equal to "email_from_address". The mock will then return "test@example.org". Take a look at the mocker and plone.mocktestcase documentation to see the various other types of assertions you can make.
• The most important mock assertion is the line self.expect(mail_host_mock.secureSend(message, email, sender, subject)). This asserts that the secureSend() method gets called with the required message, recipient address, sender address and subject, exactly once.
• We then put the mock into replay mode, using self.replay(). Up until this point, any calls on our mock objects have been to record expectations and specify behaviour. From now on, any call will count towards verifying those expectations.
• Finally, we call the code under test with our dummy presenter and event.
• In this case, we don't have any "normal" assertions, although the usual unit test assertion methods are all available if you need them, e.g. to test the return value of the method under test. The assertions in this case are all coming from the mock objects. The tearDown() method of the MockTestCase class will in fact check that all the various methods were called exactly as expected.

To run these tests, use the normal test runner, e.g.:

$ ./bin/test example.conference -t TestPresenterMock

Note that mock tests are typically as fast as unit tests, so there is typically no need for something like roadrunner.

Mock testing caveats

Mock testing is a somewhat controversial topic. On the one hand, it allows you to write tests for things that are often difficult to test, and a mock framework can - once you are familiar with it - make child's play out of the often laborious task of creating reliable test doubles. On the other hand, mock based tests are inevitably tied to the implementation of the code under test, and sometimes this coupling can be too tight for the test to be meaningful. Using mock objects normally also means that you need a very good understanding of the external APIs you are mocking. Otherwise, your mock may not be a good representation of how these systems would behave in the real world. Much has been written on this, for example by Martin Fowler.

As always, it pays to be pragmatic. If you find that you can't write a mock based test without reading every line of code in the method under test and reverse engineering it for the mocks, then an integration test may be more appropriate. In fact, it is prudent to have at least some integration tests in any case, since you can never be 100% sure your mocks are valid representations of the real objects they are mocking.

On the other hand, if the code you are testing is using well-defined APIs in a relatively predictable manner, mock objects can be a valuable way to test the "side effects" of your code, and a helpful tool to simulate things like exceptions and input values that may be difficult to produce otherwise.

Remember also that mock objects are not necessarily an "all or nothing" proposition. You can use simple dummy objects or "real" instances in most cases, and augment them with a few mock objects for those difficult-to-replicate test cases.