"I know I should write tests, but ...

• ... they take time to write
• ... I’m a good developer
• ... my customer / the community does the testing"

Sound familiar? No matter how good you think you are, you will make mistakes. Your code will contain bugs and someone will come after you demanding an explanation. Without some methodical way of testing, you are guaranteeing your code with nothing more than guesswork and arrogance. Clicking around the Plone interface for a few minutes before you ship your code off to the customer or user is simply not enough.

Testing is an art, it needs to be built into your development cycle from the very beginning - it is not something you do only after all the other work is finished, it is something you do continuously. Unfortunately, testing often evokes emotions of dread in developers. It's slow, it's boring, it's not what they signed up to do. But the art of testing has evolved beyond that - there is considerable elegance and fun to be found in well-conceived test strategies.

This tutorial aims to give you the tools you need to write tests and testable software in Plone. If you are writing software for Plone core itself, don't even think about commiting any bug fix or feature without test coverage. If you are writing an add-on product or doing a customisation, holding yourself to the same high standards that the Plone core team do will give you better confidence in your software and will likely save you considerable pain down the road.

Examples

This tutorial contains several examples of the various types of tests. They are available in the example.tests package, which you can install as a develop egg in a Plone 3 buildout. The examples of running tests use the standard commands for buildouts, since this is the only way that works reliably on Windows (that is, plain zopectl test will not work on Windows).

Take a look at the buildout tutorial for more information.

Try to find the bug in the following piece of code:

class Employee(object): 
    def __init__(self, name, position, employee_no=None): 
        self.name = name 
        self.position = position 
        self.employee_no = employee_no 

salaries = {0: 12000, 
            1: 4000, 
            2: 8000, 
            3: 4000}

def print_salary(employee): 
    if employee.employee_no: 
        salary = salaries.get(employee.employee_no, 0) 
        print "You make EUR %s." % salary 
    else: 
        print "You're not an employee currently."

Found it yet? Did you have to spend more than a few seconds thinking about it? Any developer could have written that code and not seen the problem. Furthermore, the bug is an edge case that you may not have tested using manual/through-the-web testing.

Let us write a test (actually, a doc/unit test) for this code. Don't worry too much about how this is set up and executed just yet.

Employee w/o an employee number is ignored: 

  >>> print_salary(Employee('Adam', 'Developer')) 
  You're not an employee currently 

Employee w/o a known employee number earns nothing: 

  >>> print_salary(Employee('Berta', 'Designer', 100)) 
  You make EUR 0. 

Employee w/ a valid employee number is found properly: 

  >>> print_salary(Employee('Chris', 'CTO', 2)) 
  You make EUR 8000.
 
Zero is a valid employee number: 

  >>> print_salary(Employee('Devon', 'CEO', 0)) 
  You make EUR 12000

As it happens, the last test would fail. It would print You are not an employee currently., unless we fixed the code:

class Employee(object): 
    def __init__(self, name, position, employee_no=None): 
        self.name = name 
        self.position = position 
        self.employee_no = employee_no 

salaries = {0: 12000, 
            1: 4000, 
            2: 8000, 
            3: 4000} 

def print_salary(employee): 
    if employee.employee_no is not None: 
        salary = salaries.get(employee.employee_no, 0) 
        print "You make EUR %s." % salary 
    else: 
        print "You're not an employee currently."

The moral of the story?

• you rarely catch problems like these with manual testing
• put the time you waste catching silly bugs and typos into writing tests
• with decent test coverage, you end up saving lots of time when you refactor

Broadly speaking, there are four main types of tests:

Unit tests

These are written from the programmer's perspective. A unit test should test a single method or function in isolation, to ensure that it behaves correctly. For example, testing that a given calculation is performed correctly given a variety of input is a good unit test for that one method.

Integration tests

Whereas unit tests try to remove or abstract away as many dependencies as possible to ensure that they are truly only concerned with the method under test, integration tests exercise the integration points between a method or component and the other components it relies on. For example, testing that a method performs some calculation and then correctly stores the result in the ZODB is an integration test in that it tests the integration between that component and the ZODB.

Functional tests

A functional test is typically demonstrating a use case, exercising a "vertical" of functionality. For example, testing that filling in a form and clicking "Save" then makes the resulting object available for future use, is a functional test for the use case of using that form to create content objects. 

System tests

These are written from the user's perspective, and treat the system as a black box. A system test may be simulating a user interacting with the system according to expected usage patterns. By their nature, they are typically less systematic than the other types of tests. 

Furthermore, functional tests may be white box, in which case they can make assertions about things like the underlying data storage (but only if this is specified clearly; implementation details should never affect functional tests). Such tests are also called functional integration tests (you can see where the lines start to blur, but don't worry too much about the naming). Alternatively, functional tests can be black box in which case they only perceive the system from the point of view of an actor (usually the end user) and make assertions only on what is presented in the (user) interface to that actor. Such tests, also known as acceptance tests would not make assumptions about the underlying architecture at all.

Tests and documentation

In a post to the Zope 3 mailing list, Jim Fulton explains the importance of tests and documentation, and how they go hand-in-hand:

   One of the important things about this is that most doctests
   should be written as documentation.  When you write new software
   components and you need to write tests for the main functionality
   of your software you need to:

   - Get your head into the mode of writing documentation.
     This is very very very important.

   - You need to document how to use the software.  Include examples,
     which are tests

We will learn more about doctests, and how they are used for unit testing and functional testing later. The important thing to note is that good tests often serve as documentation describing how your component is supposed to be used. Thinking about the story they tell is just as important as thinking about the number of input and output states they cover.

By their nature, tests should exercise an API and demonstrate how it is used. Thus, for other developers trying to understand how a module or library should be used, tests can be the best form of documentation. Python supports the notion of doctests, otherwise known as executable documentation.

Doctests look like Python interpreter sessions. They contain plain text (normally in reStructedText, which can be rendered to HTML or PDF easily) as well as examples. The idea is to show something that could have been typed in an interpreter session and what the expected outcome should be. In the Zope 3 world, doctests are extremely prevalent and are used for most unit and integration testing.

Doctests come in two main flavours: You can write a simple text file, such as a README.txt, that explains your code along with verifiable examples, or you can add doctests for a given method or class into the docstring of that method or class.

The full-file approach - sometimes known as documentation-driven development - is the most common. This type of test is very well suited for explaining how an API should be used and ensuring that it works as expected at the same time. However, note that these are not technically proper unit tests, because there is no guarantee of isolation between the steps of the "script" that the doctest describes. The docstring version uses the same basic syntax, but each docstring is executed as its own test fixture, guaranteeing full isolation between tests.

Here is a trivial example of a doctest. We will learn how to set up such a test shortly.

Interfaces are defined using Python class statements:: 

  >>> import zope.interface 
  >>> class IFoo(zope.interface.Interface): 
  ...    """Foo blah blah""" 
  ... 
  ...    x = zope.interface.Attribute("""X blah blah""") 
  ... 
  ...    def bar(q, r=None): 
  ...        """bar blah blah""" 

In the example above, we've created an interface:: 

  >>> type(IFoo) 
  <class 'zope.interface.interface.InterfaceClass'> 

We can ask for the interface's documentation:: 

  >>> IFoo.__doc__ 
  'Foo blah blah'

We could create an arbitrary object - this will of course not provide 
the interface.

  >>> o = object()
  >>> o # doctest: +ELLIPSIS
  <object at ....>
  >>> IFoo.providedBy(o)
  False
  >>> o.bar() # doctest: +ELLIPSIS
  Traceback (most recent call last):
  ...
  AttributeError: 'object' object has no attribute 'bar'

Each time the doctest runner encounters a line starting with >>>, the prompt of the Python interpreter (i.e. what you get by running python without any arguments in a terminal), it will execute that line of code. If that statement is then immediately followed by a line with the same level of indentation as the >>> that is not a blank line and does not start with >>>, this is taken to be the expected output of the statement. The test runner will compare the output it got by executing the Python statement with the output specified in the doctest, and flag up an error if they don't match.

Note that not writing an output value is equivalent to stating that the method has no output. Thus, this is a failure:

    >>> foo = 'hello'
    >>> foo
    >>> # do something else

The reference to foo on its own will print the value of foo. The correct DocTest would read:

    >>> foo = 'hello'
    >>> foo
    'hello'
    >>> # do something else

Notice also the ... (ellipsis) element in the expected otuput. These mean "any number of characters" (anologus to a .* statement in a regular expression, if you are familiar with those). They are usually convenient shorthand, but they can sometimes be necessary. For example:

  >>> class Foo:
  ...     pass
  >>> Foo()
  <__main__.Foo instance at ...>

Here, the ... in the expected output replaces a hexadecimal memory address (0x0x4523a0 on the author's computer at the time of writing), which cannot be predicted in advance. When writing doctests in particular (but also when writing regular unit tests), you need to be careful about values you cannot predict, such as auto-generated ids based on the current time or a random number. The ellipsis operator can help you work around those.

Do not confuse the ellipsis operator in the expected output with the syntax of using ... underneath a >>> line. This is the standard Python interpreter syntax used to designate statments that run over multiple lines, normally as the result of indentation. You can, for example, write:

  >>> if a == b:
  ...     foo = bar

 if that is necessary in your test.

Doctest tips and tricks

As with all testing, you will get better at doctests over time. Below are a few tips that may help you get started.

Read the documentation

doctests have been in Python for a long time. The doctest module comes with more documentation on how they work.

A test is just a bunch of python statements!

Never forget this. You can, for example, reference helper methods in your own product, for example, imagine you have a method in Products.MyProduct.tests.utils that has a method setUpSite() to pre-populate your site with a few directories and users. Your doctest could contain:

  >>> from Products.MyProduct.tests.utils import setUpSite
  >>> setUpSite()

The test suite can perform additional initialisation

A test suite can have setUp() and/or tearDown() handlers that perform additional set-up or clean-up. We will see further examples of this later. 

PDB is still your friend

You can put the standard import pdb; pdb.set_trace() on a line in doctest. Unfortunately, you can't step through a doctest line by line, but you can print variables and examine the state of the test fixture.

You can catch exceptions

If you need to debug a doctest that is throwing an exception, this statement is often useful:

  >>> try:
  ...     someOperation()
  ... except:
  ...     import pdb; pdb.set_trace()
  >>> # continue as normal

  ./bin/zopectl test -s Products.RichDocument

This would run all tests in the Products.RichDocument module. If you are using a buildout with an instance control script called instance, this would be:

  ./bin/instance test -s Products.RichDocument

Using buildout is probably a good idea - see the buildout tutorial - not at least because this is the only way that works reliably on Windows. We will use this syntax from now on.

To execute a single test or a set of tests matched by regular expression, you can use:

  ./bin/instance test -s Products.RichDocument -t setup

This would run tests in files like test_setup.py. To run all doctests in README.txt (presuming there was a test suite for this file) you would write:

  ./bin/instance test -s Products.RichDocument -t README.txt

The new test runner also includes a few debugging options. For example:

  ./bin/instance test -m Products.RichDocument -D

This will stop execution at the first failing test and drop into a PDB post-mortem.

To see the other options that are available, run:

  ./bin/instance test --help

When the tests you think are relevant all pass, it's time to run all tests and make sure nothing else broke. (No, we don't care that you are writing your code in a totally different python module than what those other tests are supposed to test, and that they were all fine and good and all you changed was a docstring. Run the tests when you think you're done.)

When tests finish running, you will see a report like:

    ...
    Ran 18 tests in 6.463s

    OK

(it may look slightly different, depending on which test runner you are using)

Rehearse a satisfied sigh as you read the line "OK", as opposed to seeing a count of failed tests. With time, this will be the little notifier that lets you go to bed, see your friends again or generally get back to real life with an svn commit.

If you're not so lucky, you may see:

    Ran 18 tests in 7.009s

    FAILED (failures=1, errors=1)

(again, the output may look slightly different depending on your test runner, but the same information should always be there)

This means that there were 1 python error and 1 failed test during test execution.

A python error means that some of your test code, or some code that was called by a test, raised an exception. This is bad, and you should fix it right away.

A failed test means that your test was trying to assert something that turned out not to be true. This could be OK. It could mean you haven't written the code the test is testing yet (well done, you wrote the test first!), or that you don't yet know why it's failing. Sometimes you may be radically refactoring or rewriting parts of your code, and the tests will keep on failing until you're done. Incidentally, this is part of the reason why unit tests are so good - you can do that kind of stuff.

It's sometimes (not always - don't try this on Plone core unless you've been told it's OK by the release manager) acceptable to go to bed and check in a failing test if you are not in a position to know how to fix it. At least other developers will be aware of the problem and may be able to fix it.

We will start by showing how to create a simple unit test with doctest syntax. There is nothing Zope- or Plone-specific about this test. This type of test is ideal for methods and classes that perform some kind of well-defined operation on primitives or simple objects. The doctest syntax is well-suited for explaining the inputs and outputs. Since the tests are relatively few and/or descriptive, keeping the tests, documentation and code close together makes sense.

Tests are usually found in a tests/ sub-package. In the example.tests package, we have created a file called tests/test_simple_doctest.py. This sets up a test suite to run doctests in the doc strings in the module example.tests.context. Let's look at the test setup first:

"""This is the setup for a doctest where the actual test examples are held in 
docstrings in a module.

Here, we are not using anything Zope-specific at all. We could of course 
use the Zope 3 Component Architecture in the setup if we wanted. For that,
take a look at test_zope3_doctest.py.

However, we *do* use the zope.testing package, which provides improved
version of Python's standard DocTestSuite, DocFileSuite and so on. If you
don't want this dependency, just use doctest.DocTestSuite.
"""

import unittest
import zope.testing

import example.tests.context

def setUp(test):
    """We can use this to set up anything that needs to be available for
    each test. It is run before each test, i.e. for each docstring that
    contains doctests.
    
    Look at the Python unittest and doctest module documentation to learn 
    more about how to prepare state and pass it into various tests.
    """
    
def tearDown(test):
    """This is the companion to setUp - it can be used to clean up the 
    test environment after each test.
    """
    
def test_suite():
    return unittest.TestSuite((
    
        # Here, we tell the test runner to execute the tests in the given
        # module. The setUp and tearDown methods can be used to perform
        # test-specific setup and tear-down.
    
        zope.testing.doctest.DocTestSuite(example.tests.context,
                     setUp=setUp,          # setUp and tearDown are optional!
                     tearDown=tearDown),
        ))

There are a lot of comments here, and we show how to use setUp() and tearDown() methods for additional initialisation and clean-up, if necessary. The test runner will call the test_suite() method and expect a TestSuite object back. If desired, we could have put multiple test suites referring to multiple modules into the TestSuite that is being returned.

Here is the actual code under test, in context.py:

from zope.interface import implements
from example.tests.interfaces import IContext

class Context(object):
    """An object used for testing. We will register an adapter from this
    interface to IUpperCaser in the test setup.
    
    Here's how you use it. First, import the class.
    
        >>> from example.tests.context import Context
        
    Then in-stan-ti-ate it (with me so far?):
    
        >>> my_context = Context()

    Okay, here's the tricky bit ... now we need to set the title:
    
        >>> my_context.title = u"Some string!"
        
    Phew ... did that work?
    
        >>> my_context.title
        u'Some string!'
        
    Yeah!
    """
    
    implements(IContext)
    
    def __init__(self, title=u""):
        self.title = title

Here is how we may run the tests from a buildout:

./bin/instance test -s example.tests -t context
Running unit tests:
  Running:
....
  Ran 4 tests with 0 failures and 0 errors in 0.071 seconds.

In the previous example, we wrote a doctest in a docstring. As tests become more complex or require more involved configuration, it is usually better to separate the actual test into a text file. Sometimes, this can be the README.txt file of a package. This is the approach favoured by Zope 3 components.

In this example, we will register an adapter that is used in a doctest. This doctest also serves to illustrate how this particular adapter should be used.  This style of test is great when the emphasis is on the documentation as well as the test. Note that we do not load the package's ZCML in its entirely. Instead, we register the required components explicitly. This means that we retain control over what is executed in the test. We use the zope.component.testing.tearDown method to ensure that our test environment is properly cleaned up.

In the example.tests package, we have the following test setup in tests/test_zope3_doctest.py:

"""This is the setup for a doctest that tests a Zope 3 component.

There is really nothing too different from a "plain Python" test. We are not
parsing ZCML, for example. However, we use some of the helpers from Zope 3
to ensure that the Component Architecture is properly set up and torn down.
"""

import unittest

import zope.testing
import zope.component

def setUp(test):
    """This method is used to set up the test environment. We pass it to the
    DocFileSuite initialiser. We also pass a tear-down, but in this case,
    we use the tear-down from zope.component.testing, which takes care of
    cleaning up Component Architecture registrations.
    """
    
    # Register the adapter. See zope.component.interfaces for more

    from example.tests.context import UpperCaser
    zope.component.provideAdapter(UpperCaser)

def test_suite():
    return unittest.TestSuite((
    
        # Here, we tell the test runner to execute the tests in the given
        # file. The setUp and tearDown methods employed make use of the Zope 3
        # Component Architecture, but really there is nothing Zope-specific
        # about this. If you want to test "plain-Python" this way, the setup
        # is the same.
    
        zope.testing.doctest.DocFileSuite('tests/zope3.txt',
                     package='example.tests',
                     setUp=setUp,
                     tearDown=zope.component.testing.tearDown),
        ))

Notice how we use a custom setUp() method to register the custom adapter, and then reference zope.component.testing.tearDown for the tear-down method.

This refers to the file zope3.txt, which looks like this:

==========================
A Zope 3 component doctest
==========================

This is the type of test found most commonly in Zope 3. We have a custom
setup method (in test_zope3_doctest.py) which registers the components we
need for the test. We can then use those here. ZCML is not processed directly,
nor do we have a full Zope 2/Plone environment available. This makes the test
more isolated (and faster!). Often, we may choose to use mock implementations
of certain components in order to make the test properly isolated.

Of course, we should still tell a story with this documentation.

Let's say we had one of our really exciting context objects:

    >>> from example.tests.context import Context
    >>> context = Context()
    >>> context.title = u"Some puny title"

Of course, that's nice, but what if we wanted to make a bit more of an impact?
We can use our handy upper-caser adapter!

    >>> from example.tests.interfaces import IUpperCaser
    >>> shout = IUpperCaser(context)
    >>> shout.title
    u'SOME PUNY TITLE'
    
Wow!

To run just this test, we may do:

  ./bin/instance test -s example.tests -t zope3.txt
  Running unit tests:
    Running:
  ..
    Ran 2 tests with 0 failures and 0 errors in 0.010 seconds.

PloneTestCase, which in turn uses ZopeTestCase, is used to set up a full Zope environment, including a Plone instance, for testing. This type of test is very convenient and often necessary because content types, tools and other parts of Plone have hard dependencies on various underlying Zope, CMF and Plone components. It is generally better to write simpler tests, however, both because they provide better isolation (thus testing the component more directly and under better controlled circumstances) and because they execute faster.

PloneTestCase-tests are often referred to as "unit tests", but in truth they are integration tests, since they depend on a "live" Zope instance and thus test the integration between your code and the underlying framework. We can use the PloneTestCase setup to run doctests, as we will see in the next section.

Here, however, we will demonstrate how to use unittest.TestCase classes, where each test is a method on a class (with a name beginning with test) This type of test is not as good for documentation, but can be very useful for systematically executing many variations on the same test. Some developers also find this type of test easier to debug, since it is plain Python code which can be stepped through using the debugger.

In the example.tests package, we have tests/base.py. This does not contain any tests, but performs the necessary configuration to set up the test fixture:

"""Test setup for integration and functional tests.

When we import PloneTestCase and then call setupPloneSite(), all of Plone's
products are loaded, and a Plone site will be created. This happens at module
level, which makes it faster to run each test, but slows down test runner
startup.
"""

from Products.Five import zcml
from Products.Five import fiveconfigure

from Testing import ZopeTestCase as ztc

from Products.PloneTestCase import PloneTestCase as ptc
from Products.PloneTestCase.layer import onsetup

#
# When ZopeTestCase configures Zope, it will *not* auto-load products in 
# Products/. Instead, we have to use a statement such as:
# 
#   ztc.installProduct('SimpleAttachment')
# 
# This does *not* apply to products in eggs and Python packages (i.e. not in
# the Products.*) namespace. For that, see below.
# 
# All of Plone's products are already set up by PloneTestCase.
# 

@onsetup
def setup_product():
    """Set up the package and its dependencies.
    
    The @onsetup decorator causes the execution of this body to be deferred
    until the setup of the Plone site testing layer. We could have created our
    own layer, but this is the easiest way for Plone integration tests.
    """
    
    # Load the ZCML configuration for the example.tests package.
    # This can of course use <include /> to include other packages.
    
    fiveconfigure.debug_mode = True
    import example.tests
    zcml.load_config('configure.zcml', example.tests)
    fiveconfigure.debug_mode = False
    
    # We need to tell the testing framework that these products
    # should be available. This can't happen until after we have loaded
    # the ZCML. Thus, we do it here. Note the use of installPackage() instead
    # of installProduct().
    # 
    # This is *only* necessary for packages outside the Products.* namespace
    # which are also declared as Zope 2 products, using 
    # <five:registerPackage /> in ZCML.
    
    # We may also need to load dependencies, e.g.:
    # 
    #   ztc.installPackage('borg.localrole')
    # 
    
    ztc.installPackage('example.tests')
    
# The order here is important: We first call the (deferred) function which
# installs the products we need for this product. Then, we let PloneTestCase 
# set up this product on installation.

setup_product()
ptc.setupPloneSite(products=['example.tests'])

class ExampleTestCase(ptc.PloneTestCase):
    """We use this base class for all the tests in this package. If necessary,
    we can put common utility or setup code in here. This applies to unit 
    test cases.
    """

class ExampleFunctionalTestCase(ptc.FunctionalTestCase):
    """We use this class for functional integration tests that use doctest
    syntax. Again, we can put basic common utility or setup code in here.
    """

Notice how we can explicitly install third party products (and egg-based packages which use product semantics) and then tell PloneTestCase to quick-install these into the test fixture site. The test runner will not automatically load all products in the Products.* namespace, nor will it execute ZCML for packages outside Products.* automatically.

The test class which uses this environment is found in tests/test_integration_unit.py:

"""This is an integration "unit" test. It uses PloneTestCase, but does not
use doctest syntax.

You will find lots of examples of this type of test in CMFPlone/tests, for 
example.
"""

import unittest
from example.tests.tests.base import ExampleTestCase

from Products.CMFCore.utils import getToolByName

class TestSetup(ExampleTestCase):
    """The name of the class should be meaningful. This may be a class that
    tests the installation of a particular product.
    """
    
    def afterSetUp(self):
        """This method is called before each single test. It can be used to
        set up common state. Setup that is specific to a particular test 
        should be done in that test method.
        """
        self.workflow = getToolByName(self.portal, 'portal_workflow')
        
    def beforeTearDown(self):
        """This method is called after each single test. It can be used for
        cleanup, if you need it. Note that the test framework will roll back
        the Zope transaction at the end of each test, so tests are generally
        independent of one another. However, if you are modifying external
        resources (say a database) or globals (such as registering a new
        adapter in the Component Architecture during a test), you may want to
        tear things down here.
        """
    
    def test_portal_title(self):
        
        # This is a simple test. The method needs to start with the name
        # 'test'. 

        # Look at the Python unittest documentation to learn more about hte
        # kinds of assertion methods which are available.

        # PloneTestCase has some methods and attributes to help with Plone.
        # Look at the PloneTestCase documentation, but briefly:
        # 
        #   - self.portal is the portal root
        #   - self.folder is the current user's folder
        #   - self.logout() "logs out" so that the user is Anonymous
        #   - self.setRoles(['Manager', 'Member']) adjusts the roles of the current user
        
        self.assertEquals("Plone site", self.portal.getProperty('title'))

    def test_able_to_add_document(self):
        new_id = self.folder.invokeFactory('Document', 'my-page')
        self.assertEquals('my-page', new_id)
        
    # Keep adding methods here, or break it into multiple classes or
    # multiple files as appropriate. Having tests in multiple files makes
    # it possible to run tests from just one package:
    #
    #   ./bin/instance test -s example.tests -t test_integration_unit


def test_suite():
    """This sets up a test suite that actually runs the tests in the class
    above
    """
    suite = unittest.TestSuite()
    suite.addTest(unittest.makeSuite(TestSetup))
    return suite

Here, we have a test suite with one test class - we could have added more classes if necessary. The afterSetUp() and beforeTearDown() methods - if present - are called immediately before and after each test. After a test is run, the transaction is rolled back, causing tests to run in isolation. You only really need explicit teardown if your tests make permantent changes that are not covered by the ZODB transaction machinery.

You are free to add whatever helper methods you wish to your unit test class, but any method with a name starting with test will be executed as a test. Tests are usually written to be as concise (not to be confused with "obfuscated") as possible.

Notice the calls to methods like self.assertEqual() or self.failUnless(). These are the assertion methods that do the actual testing. If any of these fail, that test is counted as a failure and you'll get an ugly F in your test output.

To run the test, we would do:

  ./bin/instance test -s example.tests -t test_integration_unit
    Running:
  ..
    Ran 2 tests with 0 failures and 0 errors in 0.178 seconds.

There is actually more output than this, as PloneTestCase installs a number of products and processes ZCML.

Rules of thumb

There are some basic rules of thumb for writing unit tests with PloneTestCase you should be aware of:

• Write test first, don't put it off, and don't be lazy (did we say this enough already?)
• Write one test (i.e. one method) for each thing you want to test
• Keep related tests together (i.e. in the same test case class)
• Be pragmatic. If you want to test every combination of inputs and outputs you will probably go blue in the face, and the additional tests are unlikely to be of much value. Similarly, if a method is complicated, don't just test the basic case. This comes with experience, but in general, you should test common cases, edge cases and preferably cases in which the method or component is expected to fail (i.e. test that it fails as expected - you still shouldn't get any F's in your test output!).
• Keep tests simple. Don't try to be clever, don't over-generalise. When a test fails, you need to easily determine whether it is because the test itself is wrong, or the thing it is testing has a bug.
  

Assertion and utility methods in the unit testing framework

There are quite a few assertion methods, most of which do basically the same thing - check if something is True or False. Having a variety of names allows you to make your tests read the way you want. The list of assertion methods can be found in the Python documentation for unittest.TestCase. The most common ones are:

failUnless(expr)

Ensure expr is true 

assertEqual(expr1, expr2) 

Ensure expr1 is equal to expr2 

assertRaises(exception, callable, ...) 

Make sure exception is raised by the callable. Note that callable here should be the name of a method or callable object, not an actual call, so you write e.g. self.assertRaises(AttributeError, myObject.myMethod, someParameter). Note lack of () after myMethod. If you included it, you'd get the exception raised in your test method, which is probably not what you want. Instead, the statement above will cause the unit testing framework to call myMethod(someParameter) (you can pass along any parameters you want after the calalble) and check for an AttributeError.

fail() 

Simply fail. This is useful if a test has not yet been completed, or in an if statement inside a test where you know the test has failed. 

In addition to the unit testing framework assertion methods, ZopeTestCase and PloneTestCase include some helper methods and variables to help you interact with Zope. It's instructive to read the source code for these two products, but briefly, the key variables you can use in unit tests are:

self.portal 

The Plone portal the test is executing in 

self.folder 

The member folder of the member you are executing as 

And the key methods are:

self.logout() 

Log out, i.e. become anonymous 

self.login() 

Log in again. Pass a username to log in as a different user. 

self.setRoles(roles) 

Pass in a list of roles you want to have. For example, self.setRoles(('Manager',)) lets you be manager for a while. How nice.

self.setPermissions(permissions)

Similarly, grant a number of permissions to the current user in self.folder. 

self.setGroups(groups) 

Set which groups the test user is in. 

Tips & Tricks

Good unit testing comes with experience. It's always useful to read the unit tests of code with which you are fairly familiar, to see how other people unit test. We'll cover a few hints here to get you thinking about how you approach your own tests:

• Don't be timid! Python, being a dynamic scripting language, lets you do all kinds of crazy things. You can rip a function right out from the Plone core and replace it with your own implementation in afterSetUp() or a test if that serves your testing purposes.
• Similarly, replacing things like the MailHost with dummy implementations may be the only way to test certain features. Look at CMFPlone/tests/dummy.py for some examples of dummy objects.
• Use tests to try things out. They are a safe environment. If you need to try something a bit out of the ordinary, writing them in a test is often the easiest way of seeing how something works.
• During debugging, you can insert print statements in tests to get traces in your terminal when you execute the tests. Don't check in code with printing tests, though. :)
• Similarly, the python debugger is very valuable inside tests. Putting import pdb; pdb.set_trace() inside your test methods lets you step through testing code and step into the code it calls. If you're not familiar with the python debugger, your life is incomplete. More about using pdb with Plone.
  

The choice of test case classes over doctest is purely one of syntactic preference. We can use the test setup from the previous section (in base.py) in a doctest as well. This type of test is more useful for documenting the integration of your code with Zope/Plone in a narrative fashion.

There is no change to tests/base.py for this type of setup. However, we must be careful to use a test class that derives from FunctionalTestCase, since this performs the initialisation necessary for doctests. The test setup is found in tests/test_integration_doctest.py:

"""This is an integration doctest test. It uses PloneTestCase and doctest
syntax.
"""

import unittest
import doctest

from zope.testing import doctestunit
from Testing import ZopeTestCase as ztc

from example.tests.tests import base

def test_suite():
    """This sets up a test suite that actually runs the tests in the class
    above
    """
    return unittest.TestSuite([

        # Here, we create a test suite passing the name of a file relative 
        # to the package home, the name of the package, and the test base 
        # class to use. Here, the base class is a full PloneTestCase, which
        # means that we get a full Plone site set up.

        # The actual test is in integration.txt

        ztc.ZopeDocFileSuite(
            'tests/integration.txt', package='example.tests',
            test_class=base.ExampleFunctionalTestCase,
            optionflags=doctest.REPORT_ONLY_FIRST_FAILURE | 
                        doctest.NORMALIZE_WHITESPACE | doctest.ELLIPSIS),
            
        # We could add more doctest files here as well, by copying the file
        # block above.

        ])

Here, we set ExampleFunctionalTestCase from base.py as the test_class, which means that self in the doctest will be the same as self in the test class we saw in the previous section. In particular, we can access variables such as self.portal and self.folder. We also set some common doctest option flags - reporting only the first failure (to avoid overly long error output when an example early on in the doctest fails), normalising whitespace (so that we can use newlines freely) and allowing the ellipsis operator everywhere (as opposed to having to turn it on each time we want to use it). Look at the doctest module documentation for more information.

The test itself, in tests/integration.txt, is written much like the other doctests we have seen:

======================
An integration doctest
======================

This test is an integration test that uses PloneTestCase. Here, 'self' is
the test class, so we can use 'self.folder', 'self.portal' and so on. The
setup is done in teststest_integration_doctest.py

Being a doctest, we can tell a story here. 

For example, let's say a user had a dying wish: to add a news item. We'll do
that using the standard Plone API.

    >>> self.folder.invokeFactory('News Item', 'news-item')
    'news-item'
    
That's great, but really, he wanted to add it to the portal root:
    
    >>> self.portal.invokeFactory('News Item', 'news-item')
    Traceback (most recent call last):
    ...
    Unauthorized: Cannot create News Item

Whoops! Too bad! 

At least we got to demonstrate the ellipsis operator, which
matches arbitrary text. We enabled this in test_integration_doctest.py. It
is also possible to enable (or disable) this flag on a single statement.
See the Python doctest documentation for more information.

To run this test on its own, we would do:

  ./bin/instance test -s example.tests -t integration.txt
    Running:
  ..
    Ran 2 tests with 0 failures and 0 errors in 0.384 seconds.

Again, we have cut out some of the output from PloneTestCase.

No developer likes to click around the browser to check if that button that was only supposed to show up in some cases really did show up. Unfortunately, these are also the types of problems that most often suffer from regressions, because templates are difficult (and slow) to test.

Zope 3 has an elegant library called zope.testbrowser which lets you write doctests that behave like a real web browser (almost... it cannot yet handle JavaScript, which means that testing dynamic UIs that depend on JavaScript is not possible, although Selenium may be a viable alternative here). You can open URLs, click links, fill in form fields and test the HTTP headers, URLs and page contents that are returned from Plone. In fact, you could test any website, not just Zope or Plone ones.

Functional tests are no replacement for unit tests. They test a slice of functionality, typically as the user sees it. Thus, they may not systematically include every aspect of the application. For example, a functional test may check whether a "Delete" button is present, and even that it works as expected, but should not be used to exhaustively test whether the delete operation works in every possible edge case. Where they excel, however, is in testing things like which options appear to which users depending on roles and permissions, or simply to exercise all the various templates used in a given product to make sure they don't break.

Here is an example from the example.tests package. The test setup is in tests/test_functional_doctest.py:

"""This is a a functional doctest test. It uses PloneTestCase and doctest
syntax. In the test itself, we use zope.testbrowser to test end-to-end
functionality, including the UI.

One important thing to note: zope.testbrowser is not JavaScript aware! For
that, you need a real browser. Look at zope.testbrowser.real and Selenium
if you require "real" browser testing.
"""

import unittest
import doctest


from Testing import ZopeTestCase as ztc

from example.tests.tests import base

def test_suite():
    """This sets up a test suite that actually runs the tests in the class
    above
    """
    return unittest.TestSuite([

        # Here, we create a test suite passing the name of a file relative 
        # to the package home, the name of the package, and the test base 
        # class to use. Here, the base class is a full PloneTestCase, which
        # means that we get a full Plone site set up.

        # The actual test is in functional.txt

        ztc.ZopeDocFileSuite(
            'tests/functional.txt', package='example.tests',
            test_class=base.ExampleFunctionalTestCase,
            optionflags=doctest.REPORT_ONLY_FIRST_FAILURE | 
                        doctest.NORMALIZE_WHITESPACE | doctest.ELLIPSIS),
            
        # We could add more doctest files here as well, by copying the file
        # block above.

        ])

This code is actually identical to the test setup for the integration doctest in the previous section. The differences are found in the actual test itself, which uses Products.Five.testbrowser.Browser, a Zope 2 compatability wrapper around zope.testbrowser.Browser:

====================
A functional doctest
====================

This is a full-blown functional test. The emphasis here is on testing what
the user may input and see, and the system is largely tested as a black box.
We use PloneTestCase to set up this test as well, so we have a full Plone site
to play with. We *can* inspect the state of the portal, e.g. using 
self.portal and self.folder, but it is often frowned upon since you are not
treating the system as a black box. Also, if you, for example, log in or set
roles using calls like self.setRoles(), these are not reflected in the test
browser, which runs as a separate session.

Being a doctest, we can tell a story here. 

First, we must perform some setup. We use the testbrowser that is shipped
with Five, as this provides proper Zope 2 integration. Most of the 
documentation, though, is in the underlying zope.testbrower package.

    >>> from Products.Five.testbrowser import Browser
    >>> browser = Browser()
    >>> portal_url = self.portal.absolute_url()

The following is useful when writing and debugging testbrowser tests. It lets
us see all error messages in the error_log.

    >>> self.portal.error_log._ignored_exceptions = ()

With that in place, we can go to the portal front page and log in. We will
do this using the default user from PloneTestCase:

    >>> from Products.PloneTestCase.setup import portal_owner, default_password

    >>> browser.open(portal_url)

We have the login portlet, so let's use that.

    >>> browser.getControl(name='__ac_name').value = portal_owner
    >>> browser.getControl(name='__ac_password').value = default_password
    >>> browser.getControl(name='submit').click()

Here, we set the value of the fields on the login form and then simulate a
submit click.

We then test that we are still on the portal front page:

    >>> browser.url == portal_url
    True
    
And we ensure that we get the friendly logged-in message:

    >>> "You are now logged in" in browser.contents
    True

To learn more, look at the zope.testbrowser documentation and interfaces.
There are also a few examples of testbrowser tests in Plone itself.

All the action happens with the browser object. This simulates a web browser (though as stated above, one that does not support JavaScript), and has a pleasant API for finding form controls and links and clicking on them. The variables browser.url and browser.contents represent what would've been in the URL bar and the rendered view of the page, respectively, and can be examined like any other variable.

zope.testbrowser has pretty comprehensive documentation in its README.txt file - which is, of course, a runnable doctest. In brief, the most important methods of the IBrowser interface (and thus the Browser class) are:

open(url)

Open a given URL.

reload()

Reload the current page, much as the Refresh button in your browser would do.

goBack(count=1)

Simulate pressing the Back button count times.

getLink(text=None, url=None, id=None)

Get an ILink (which you can then call click() on), either by the text inside the <a> tags, by the URL in the href attribute, or the id of the link.

getControl(label=None, name=None, index=None)

Get an IControl, representing a form control, by label (either the value of a submit button or the contents of an associated <label> tag) or form name. The index argument is used to disambiguate if there is more than one control (e.g. index=0 gets the first one). Again, you can call click() on the control object to simulate clicking on it.

The IBrowser interface also provides some properties that can be used to examine the state of the current page. The most important ones are:

url

The full URL to the current page.

contents

The full contents of the current page, as a string (usually containing HTML tags)

headers

A dict of HTTP headers

Please refer to the interfaces and the README file for details on the other methods and attributes, the interfaces for various types of links and controls, and further examples.

Debugging functional tests

Sometimes you will get errors from Zope resulting from some command executed using the testbrowser. In this case, it can sometimes be difficult to know what the underlying cause is. Two debugging aids exist to make this a bit easier.

First of all, make sure you see all errors in full by setting:

    >>> browser.handleErrors = False

If handleErrors is True (the default) you will get errors like HTTPError: HTTP Error 404: Not Found or HTTPError: HTTP Error 500: Internal Server Error. Those are probably not very useful to you. Setting handleErrors to False will show the full exceptions Zope (or possibly the HTML rendering of the error page, depending on the type of error).

Secondly, if you are using PloneTestCase, you can use Plone's error log. At the top of the example, we do:

    >>> self.portal.error_log._ignored_exceptions = ()

This means that errors such as NotFound and Unauthorized will be shown in the error log. It may also be useful to enable Verbose Security in zope.conf (see the comments in that file for details). Now, when a line appears that is throwing an error you can't debug, you can do:

    >>> try:
    ...     browser.getControl('Save').click()
    ... except:
    ...     print self.portal.error_log.getLogEntries()[0]['tb_text']
    ...     import pdb; pdb.set_trace()
    >>> # continue as normal

This will print the most recent entry in the error log, and set a PDB break point.

Using a real browser to render the results of your tests

Sometimes you would like to see the output of browser.contents in a browser to easily debug what's happening in your functional tests. To do so, place a PDB break point in your tests as described above (import pdb; pdb.set_trace()) and type the following when you get to the PDB prompt while running the tests:

>>> from Testing.ZopeTestCase.utils import startZServer
>>> startZServer()

This will print a tuple like

('127.0.0.1', 55143)

containing an IP address and port where you can access the same test site that the testbrowser is working with, in a real browser.

Functional tests vs. system tests

A system test is one which treats the entire system as a black box, interacting with it as a user would. A functional test is more focused on a single "vertical" of functionality, typically linked to a particular use case.

For a functional test, it may be acceptable to examine the internal state of the portal (using self.portal and the PloneTestCase.FunctionalTestCase class to build a test suite) to provide assertions. A system test, by contrast, makes no such assumptions. Ideally, you should be able to point a zope.testbrowser test at a remote site running a fresh installation of your system, and have the tests pass.

Beyond that, the tools used to write a system test are the same. It is only the approach to testing that changes. Whether you need one, or the other, or both, will depend on the level of rigour you need in your tests, and how your system is constructed. In general, though, true system tests are more rare than functional (integration) tests and unit tests.

Functional tests using zope.testbrowser save us from clicking around the browser to regression test UI, but writing them could still be easier. With complex templates, it can sometimes be difficult to find out what actual links and form fields the testbrowser test should be looking for, and what text to use in assertions.

This is where zope.testrecorder comes in. The theory is that you click around the UI only once, and then render the history of what you did to a runnable testbrowser test. zope.testrecorder can even create Selenium tests - an alternative form of functional tests which runs in the browser (i.e. it automates your browser right before your eyes) and thus supports JavaScript, but which cannot be run as part of an automated test run without a browser.

Installing zope.testrecorder is simple. First, check it out from Zope's subversion repository:

    svn co svn://svn.zope.org/repos/main/zope.testrecorder/trunk zope.testrecorder

See INSTALL.txt for further instructions, but the easiest way to install it in a Zope 2 instance is just to put it in your Products directory: Copy zope.testrecorder/src/zope/testrecorder as a product into Products/testrecorder and restart Zope. Then, go to the ZMI and add a Test Recorder object in the root of your Zope instance. Call it e.g. test-recorder.

Presuming you run Zope on localhost:8080, you should now be able to go to http://localhost:8080/test-recorder/index.html. You should see a page something like this:

NOTE: Like most things, zope.testrecorder seems to work better in Firefox than in other browsers.

Now, enter the address of your Plone site (or indeed any web site), e.g. http://localhost:8080/Plone and click Go. You can perform any number of operations, e.g. logging in and clicking around the UI. If you wish to add a comment to your test run, as you would add free text inside a doctest, click the Add comment button. If you wish to verify that some text appears on the page, highlight that text, shift-click on it, and select "Check text appears on page":

When you are done, click Stop recording. You can then choose to render the test as a Python doctest and you will get something like:

  Create the browser object we'll be using.

      >>> from zope.testbrowser import Browser
      >>> browser = Browser()
      >>> browser.open('http://localhost/test')

  A test comment.

      >>> 'start writing' in browser.contents
      True

You can then paste this into a doctest file, and perform any post-processing or make any changes that may be necessary to make the test more generally valid.

Tips for using zope.testrecorder

Plan, plan, plan

It's best if you have a rough script in front of you before you start recording tests, or you may get lost afterwards. Make good use of the Add comment button to state what you are testing before you test it, so that the final doctest will make sense.

Careful where you click

Some parts of the Plone UI are more ephemeral than others. It may not be a good idea to rely on links in the Recent portlet, for example. Think about what operations will provide the most general and valid test. It will save you time in the long run.

Set up your site beforehand

Recall from the section on zope.testbrowser that we set up users and basic site structure with calls to the Python APIs instead of using testbrowser to manipulate the "site setup" screents. When using zope.testrecorder you may want to set up the same users with the same user names and passwords, and the same site structure before you start recording to test. Otherwise, you may need to change some of the values of the test.

Check the doctest

zope.testrecorder is a time-saving tool. Sometimes, it may end up referring to parts of the page that can't be guaranteed to be consistent (such as randomly generated ids of content objects), and sometimes you may have gone on a detour and ended up with a test that contains irrelevant or duplicate sections. Always fix up your test (and run it!) afterwards, to make sure that the test remains valid for the future - otherwise, you will end up clicking around the UI in anger again before you know it.

The better your test suite's coverage, the lower the likelihood that some modification to your code will break another piece of functionality in some unanticipated way.  But, how do you know the quality of your test coverage?  Zope's test runner comes with several features to help you do just that.

But first, let's say you've written some code with a Python conditional like the following:

if value % 2 == 0:
    print "This is an even number"
else:
    # we need to do some more complex
    # computation to handle odd numbers
    _someComplexCodeDealingWithOddNumbers(value)

The comments and function call in the else clause are symbolic of some advanced coding that's required to handle all odd numbers. 

Now, as you've no doubt learned while reading this tutorial, testing is important.  But what if for one reason or another, all the test cases you've come up with during testing amount to even numbers when you get to the aforementioned block of code. If this were the case, you'd have a big risk of unanticipated code breakage to the way that you handle odd numbers.  This is something that you'd ideally cover in your test suite.

Discovering the untested sections of your code

You've learned how to run your test suite in this tutorial.  Zope's test runner accepts an optional parameter called --coverage.  When passed a path to a directory, Zope will generate some high-level output and produce a coverage file for each of the Python modules in your product or package. 

In full, running your test suite with the coverage option enabled looks like:

./bin/instance test -s Products.productname --coverage=$HOME/coverage

Note: Running your tests with the coverage option enabled takes significantly longer (as in ~10 times or more) than without, so this is something to be done occasionally to gauge your work, rather than each time you run your tests.

At the end of running your test suite, you'll get some immediate output like the following, which includes lines of code and your coverage percentage:

lines   cov%   module   (path)
  104   100%   $INSTANCE_HOME.parts.salesforce-integration-products.salesforcepfgadapter.Extensions.Install
               ($INSTANCE_HOME/parts/salesforce-integration-products/salesforcepfgadapter/Extensions/Install.py)
   39    41%   $INSTANCE_HOME.parts.salesforce-integration-products.salesforcepfgadapter.__init__   
               ($INSTANCE_HOME/parts/salesforce-integration-products/salesforcepfgadapter/__init__.py)
    2   100%   $INSTANCE_HOME.parts.salesforce-integration-products.salesforcepfgadapter.content.__init__   
               ($INSTANCE_HOME/parts/salesforce-integration-products/salesforcepfgadapter/content/__init__.py)
  168    91%   $INSTANCE_HOME.parts.salesforce-integration-products.salesforcepfgadapter.content.salesforcepfgadapter   
               ($INSTANCE_HOME/parts/salesforce-integration-products/salesforcepfgadapter/content/salesforcepfgadapter.py)
   21   100%   $INSTANCE_HOME.parts.salesforce-integration-products.salesforcepfgadapter.migrations.migrateUpTo10rc1   
               ($INSTANCE_HOME/parts/salesforce-integration-products/salesforcepfgadapter/migrations/migrateUpTo10rc1.py)

 If all you're looking for is a quick status report, this should suffice.

However, if you want to dig deeper, head to the directory you listed in the --coverage option.  Note: The files may be preceded with dots, thus requiring an ls -a in order to reach the coverage files.

A sample file may look like the following:

    1:     def initializeArchetype(self, **kwargs):
               """Initialize Private instance variables
               """
   15:         FormActionAdapter.initializeArchetype(self, **kwargs)
              
   15:         self._fieldsForSFObjectType = {}
          
      
    1:     security.declareProtected(View, 'onSuccess')
    1:     def onSuccess(self, fields, REQUEST=None):
               """ The essential method of a PloneFormGen Adapter 
"""
>>>>>>         logger.debug('Calling onSuccess()')
>>>>>>         sObject = self._buildSObjectFromForm(fields, REQUEST)
>>>>>>         if len(sObject.keys()) > 1:

It's really just your file with some meaningful data proceeding each line.  Anything with a 1: signifies that your code was at least touched during the running of the test suite.  The higher the number, the more often your code was touched.  Perhaps this is intentional and signifies really good coverage in other cases, it's may be either unavoidable or could even signify that the high level of coverage wouldn't actually be required.  The >>>>>> means that you've missed a line and you should consider coming up with a test scenario or more that will touch the line of code in question.  The number of untested lines divided by total lines gives you your coverage percentage.

If what you really want is eye-candy

svn co  svn://svn.zope.org/repos/main/z3c.coverage/trunk z3c.coverage

Create a directory within your previously created coverage directory.  We call it reports.  Run the coveragereport.py module with the source being you coverage output and the destination, your newly created reports directory.  See the following:

mkdir $HOME/coverage/reports
python z3c.coverage/src/z3c/coverage/coveragereport.py $HOME/coverage $HOME/coverage/reports

You should now be able to open $HOME/coverage/reports/all.html within your browser for a pretty output like the one below.

With this information available, you can start to make conclusions about how you may work your way towards better coverage of your product.

Testing is best learned by example. It can be very instructive to read through the tests written by other developers and learn what they test, what they don't test and how they write their tests.

• example.tests, which we have already mentioned, contains an example of each of the different types of tests covered in this tutorial. The test setup code is well-commented, with the intention that this package should provide good boilerplate for developers setting up a new project.
• Plone itself has more than 1,600 tests at the time of writing. Most of these are integration tests using unit-test syntax with PloneTestCase.
• RichDocument has a basic test_setup.py integration test. This is a good example of the kind of testing you may want to do to ensure that your package installs cleanly.
• borg.project contains a README.txt file with an integration doctest demonstrating how it is used. It has only a single test module, tests.py, which performs the same setup as base.py and test_integration_doctest.py from example.tests.
• Many of the tests in the plone.app.controlpanel package use basic test-browser functional tests to verify that the Plone control panels work as expected.
  

Feel free edit or comment on this page if you have more examples to add!