Unit testing

Unit tests are not strictly a Plone 2.1 feature, but they are a very important "best practice", and are definitely something you should be familiar with. Plone uses unit testing extensively (don't try to check in code to Plone itself without unit tests, you will be burned at the stake), and we will introduce them briefly here. It's time to get religious.

The idea of unit testing is that as the complexity of a piece of software increases, it becomes harder to test. It is difficult to know if you cover all the cases when you simply test your product in the Plone UI, and as you continue working on your product, you will invariably break something you thought were working before.

The golden rules of unit testing are:

• Write at least one test for every feature
• Write the interface and/or stub methods first (if applicable), then write the test, make sure the test fails (because the code isn't written yet!)
• Only when you have a failing test are you allowed to implement your feature. The goal of every line of code you write should be to make a failing test pass.
• When you find a bug, don't fix it...
• ...instead, write a failing test to demonstrate that the bug is there...
• ...and then you're allowed to fix the bug

No, we're not kidding. This may all seem cumbersome and unintuitive to you. You're wrong. Unit tests are:

1. The only way of even remotely convincing your customers and friends your code doesn't completely suck.
2. The only way of making sure (or at least being more confident) you don't break things without realising it.
3. The only way of making sure (or at least being more confident) you don't re-introduce bugs you thought you'd fixed.
4. Usually a way of saving time in the long run, because you know immediately when you break something, and you spend less time chasing down obscure bugs in code you wrote six months ago.
5. A useful way of writing and testing code in the same environment - you don't have to switch context to a browser and click around to test your newest feature - just run the tests!

Much like a good Scotch Whiskey, making unit tests pass is a good feeling, and it only gets better as time goes by and your test coverage increases. Plone has, at the time of writing, over 1500 passing tests. In fact, we need perhaps three times that, even for the existing code base.

Zope/Plone unit tests

Unit testing works by setting up a sandbox (also called a test fixture) in which the test is run. Using PloneTestCase, this is basically an empty Zope instance with a single Plone site, containing a single member, with a default member folder. All tests are run in the same environment - that is, each time a test method is finished, the Zope transaction is aborted so that no matter how that test changed the state of the portal, the next test will be completely unaffected. You are free to do obscene things in tests - look at testMigrations.py in Plone. We delete things like portal_types without so much as a twitch. Tests are executed in arbitrary (actually, alphabetical) order, and you should not rely on any interaction between tests at all.

Some basic rules of thumb for writing unit tests with PloneTestCase should be aware of:

• Write test first, don't put it off, and don't be lazy :)
• Write one test (i.e. one method) for each thing you want to test
• Keep related tests together (i.e. in the same class)
• Be pragmatic. If you want to test every combination of inputs and outputs you will probably go blue in the face, and the additional tests are unlikely to be of much value. Similarly, if a method is complicated, don't just test the basic case. This comes with experience, but in general, you should test common cases, edge cases and preferably cases in which the method or component is expected to fail (i.e. test that it fails as expected).
• Keep them simple. Don't try to be clever, don't over-generalise. When a test fails, you need to easily determine whether it is because the test itself is wrong, or the thing it is testing has a bug.
• Always run all tests before you check in your code (especially if you're not the original author/maintainer) and make sure you didn't break anything. Not doing this is a cardinal sin for the Plone core, and should be in your own environment, too.

Setting up your test environment

Adding unit tests to your product is fairly simple. You must:

• Install PloneTestCase and its dependencies (see its INSTALL.txt. Note that ZopeTestCase, upon which PloneTestCase depends, ships with Zope 2.8 and later)
• Add a tests/ directory and copy the files runalltests.py and framework.py into it. You can find these in the folder RichDocument/tests, any many other packages.
• Add some test case classes, with test methods. To make this easier, you will normally define a test case base class to set up your test environment in a consistent manner. For RichDocument, you can find this in rdtc.py. This is then used by testSetup.py, which contains the actual test methods.

We're going to re-visit the RichDocument tests in a moment, but first let's see how to run unit tests.

Running tests

There are three different ways in which you can run unit tests. The easiest one is to use zopectl from your Zope instance root, with a command like:

  ./bin/zopectl test --libdir Products/RichDocument

Alternatively, you can go to the tests directory (e.g. Products/RichDocument/tests) and run a test or all tests directly with:

    python testSetup.py

or:

    python runalltests.py

In both cases, you may have to set the environment variables INSTANCE_HOME and SOFTWARE_HOME. The former should point to your Zope instance (the parent of your Products folder), the latter should point to the python library directory where Zope is installed, e.g. /usr/local/zope-2.8.4/lib/python.

The second of the two commands above, runalltests.py, is equivalent to the zopectl method above, but may be more convenient. The first, calling a test file directly, is quite useful - it allows you to run only a subset of tests. Since test execution can take quite a long time on larger projects, this is often a good way of testing only what you think is relevant. When the tests you think are relevant all pass, it's time to run all tests and make sure nothing else broke. (No, we don't care that you are writing your code in a totally different python module than what those other tests are supposed to test, and that they were all fine and good and all you changed was a docstring. Run the tests when you think you're done.)

When tests finish running, you will see a report like:

    ...
    Ran 18 tests in 6.463s

    OK

Rehearse a satisfied sigh as you read the line "OK", as opposed to seeing a count of failed tests. With time, this will be the little notifier that lets you go to bed, see your friends again or generally get back to real life with an svn commit.

If you're not so lucky, you may see:

    Ran 18 tests in 7.009s

    FAILED (failures=1, errors=1)

This means that there were 1 python error and 1 failed test during test execution.

A python error means that some of your test code, or some code that was called by a test, raised an exception. This is bad, and you should fix it right away.

A failed test means that your test was trying to assert something that turned out not to be true. This could be OK. It could mean you haven't written the code the test is testing yet (well done, you wrote the test first!), or that you don't yet know why it's failing. Sometimes you may be radically refactoring or rewriting parts of your code, and the tests will keep on failing until you're done. Incidentally, this is part of the reason why unit tests are so good - you can do that kind of stuff.

It's sometimes (not always - don't try this on Plone core unless you've been told it's OK by the release manager) acceptable to go to bed and check in a failing test if you are not in a position to know how to fix it. At least other developers will be aware of the problem and may be able to fix it.

Writing your own unit tests

Now that you know how to set up and run your tests, let's get back to how you actually write some. Unit testing is an art form that is best learnt by example. You are encouraged to look at the RichDocument unit tests, the Plone unit tests and any other unit tests you think may be relevant to see how they work and what sort of patterns they use.

Unit testing in Zope/Plone relies on a few naming introspections to reduce the burden on you the programmer. Basically:

• All test .py files must begin with the word test, as in testSetup.py
• Inside the test files, you define test case classes. In each class, you can have any number of testing methods, which must also begin with the word test, as in testSkinLayersInstalled().

Let us look at an example. In rdtc.py, the base test case for RichDocument, we have:

  # These install (or fail) quietly
  ZopeTestCase.installProduct('CMFCore', quiet=1)

  ...

  ZopeTestCase.installProduct('kupu', quiet=1)

  # These must install cleanly
  ZopeTestCase.installProduct('RichDocument')

  PRODUCTS = ['RichDocument']
  PloneTestCase.setupPloneSite(products=PRODUCTS)

  class RichDocumentTestCase(PloneTestCase.PloneTestCase):

    ....

These lines register the standard products with ZopeTestCase, and sets up a Plone site with RichDocument installed. Then, the base class for all other test case classes is defined. In this case, it does nothing but some boilerplate. In other cases, you may find it useful to put utility methods here that may be called by the tests.

In testSetup.py, the first (and to date only) unit test file for RichDocument, we then have:

  import os, sys
  if __name__ == '__main__':
      execfile(os.path.join(sys.path[0], 'framework.py'))

  from Products.RichDocument.tests import rdtc

  class TestInstallation(rdtc.RichDocumentTestCase):
      """Ensure product is properly installed"""

      def afterSetUp(self):
          self.css        = self.portal.portal_css
          self.kupu       = self.portal.kupu_library_tool
          self.skins      = self.portal.portal_skins
          self.types      = self.portal.portal_types
          self.factory    = self.portal.portal_factory
          self.workflow   = self.portal.portal_workflow
          self.properties = self.portal.portal_properties

          self.metaTypes = ('RichDocument', 'ImageAttachment', 'FileAttachment')

      def testSkinLayersInstalled(self):
          self.failUnless('RichDocument' in self.skins.objectIds())
          self.failUnless('attachment_widgets' in self.skins.objectIds())

      ...

  def test_suite():
      from unittest import TestSuite, makeSuite
      suite = TestSuite()
      suite.addTest(makeSuite(TestInstallation))
      ...
      return suite

  if  __name__ = '__main__':
      framework()

At the top, some python magic is included to make sure you can run this test in isolation by running python testSetup.py. Then, we define the test class. Finally, the test suite is defined. You can add several test classes to each test suite. When you run a test suite (e.g. with python testSetup.py) all test methods in all test classes in the test suite in this file will be included.

Notice the line from unittest import .... This is importing test machinery from the standard Python unit-testing module, which ZopeTestCase uses. You can read more about the unittest module in the Python documentation.

Inside the TestInstallation test class, the afterSetUp() method is called immediately before each test, and can be used to set up some test data. This is frequently used to fetch tools, create dummy content etc. After a test is executed, the transaction will be rolled back, so it's normally not necessary to do much cleanup, but if you are interacting with something outside of Zope and need to clean up after each test, you can do this in a method called beforeTearDown().

You are free to add whatever helper methods you wish to your unit test class, but any method with a name starting with test... will be executed as a test. Tests are usually written to be as concise (not to be confused with "obfuscated") as possible. You'll see several calls to methods like self.assertEqual() or self.failUnless(). These are the assertion methods that do the actual testing. If any of these fail, that test is counted as a failure and you'll get an ugly F in your test output.

Assertion and utility methods in the unit testing framework

There are quite a few assertion methods, most of which do basically the same thing - check if something is True or False. Having a variety of names allows you to make your tests read the way you want. In fact, tests should be viewed as stories of how your code is expected to behave. It's often useful for other developers to read your tests to see what your actual code is capable of. The list of assertion methods can be found in the Python documentation for unittest.TestCase. The most common ones are:

failUnless(expr)

Ensure expr is true

assertEqual(expr1, expr2)

Ensure expr1 is equal to expr2

assertRaises(exception, callable, ...)

Make sure exception is raised by the callable. Note that callable here should be the name of a method or callable object, not an actual call, so you write e.g. self.assertRaises(AttributeError, myMethod, someParameter). Note lack of () after myMethod. If you included it, you'd get the exception raised in your test method, which is probably not what you want. Instead, the statement above will cause the unit testing framework to call myMethod(someParameter) (you can pass along any parameters you want after the calalble) and check for an AttributeError.

fail()

Simply fail. This is useful if a test has not yet been completed, or in an if statement inside a test where you know the test has failed.

In addition to the unit testing framework assertion methods, ZopeTestCase and PloneTestCase include some helper methods and variables to help you interact with Zope. It's instructive to read the source code for these two products, but briefly, the key variables you can use in unit tests are:

self.portal

The Plone portal the test is executing in

self.folder

The member folder of the member you are executing as

And the key methods are:

self.logout()

Log out, i.e. become anonymous

self.login()

Log in again. Pass a username to log in as a different user.

self.setRoles(roles)

Pass in a list of roles you want to have. For example, self.setRoles((Manager,)) lets you be manager for a while. How nice.

self.setPermissions(permissions)

Similarly, set a number of permissions for the test user in self.folder

self.setGroups(groups)

Set a list of groups for the current user.

Tips & Tricks

Good unit testing comes with experience. It's always useful to read the unit tests of code with which you are fairly familiar, to see how other people unit test. We'll cover a few hints here to get you thinking about how you approach your own tests:

• Don't be timid! Python, being a dynamic, weakly typed scripting language, lets you do all kinds of crazy things. You can rip a function right out from the Plone core and replace it with your own implementation in afterSetUp() or a test if that serves your testing purposes.
• Similarly, replacing things like the MailHost with dummy implementations may be the only way to test certain features. Look at CMFPlone/tests/dummy.py for some examples of dummy objects.
• Use tests to try things out. They are a safe environment. If you need to try something a bit out of the ordinary, writing them in a test is often the easiest way of seeing how something works.
• During debugging, you can insert print statements in tests to get traces in your terminal when you execute the tests. Don't check in code with printing tests, though. :)
• Similarly, the python debugger is very valuable inside tests. Putting import pdb; pdb.set_trace() inside your test methods lets you step through testing code and step into the code it calls. If you're not familiar with the python debugger, your life is incomplete. Read more in the python documentation.