This tutorial shows how to install Plone 3 into a buildout, and how to use that buildout when working on a software project that extends Plone. A buildout is a self-contained environment where you can manage the dependencies (including Zope and Plone and any third-party products or libraries you need) and custom code for your project. Even if you are not planning on writing any custom code, the buildout approach is an easy way to install Plone in a robust, well-tested manner.  As of Plone 3.2, all of the installers are now buildout based.

Prior to Plone 3.0, most developers and users who did not use a GUI installer, would set up a Zope instance, drop in a few products into the Products folder, and be done with it. Unfortunately, this approach has a few problems:

• Plain old Zope instances are not very well equipped to deal with packages distributed as python eggs or using setuptools namespace packages. Many new packages in Plone 3 are made in this way, and more and more third party modules will be as well.
• Without access to the metadata that is held in eggs, developers may find it too time-consuming or confusing to factor their work into multiple packages that are more re-usable, preferring monolithic products that are impossible to re-use outside Zope.
  
• Without any further tools, it is cumbersome to repeat a setup across different environments.

As eggs become more important, developers should look to employ more appropriate tools for managing their code. zc.buildout, hereafter referred to only as "buildout" is one such tool. This tutorial shows how to use buildout for day-to-day development as well as deployment.

More buildout documentation and background

Buildout was created by Jim Fulton of Zope Corporation, and is documented in depth at: http://buildout.org/

Terminology

Before we begin, you should familiarize yourself with these terms:

• Software home
  
• Zope instance
• Python path
• Python package
• Zope product
• Python egg
• The Python Package Index
  
• easy_install
• Namespace package

The magic Products namespace 

When Zope finds a "product", it will create an entry in Control_Panel/Products in the root of the ZMI, and run the initialize() method, found in the product's root __init__.py file, each time Zope starts up. Not every package used in a Plone context needs to be a product, but "productness" is required for:

• GenericSetup profiles
• Skin directories being installed as layers in the portal_skins tool (but not for Zope 3-style browser views)

The easiest way to create a product is to use Paster/ZopeSkel to create an egg-ready package in the Products.* namespace using the basic_namespace template:

$ paster create -t basic_namespace Products.myproduct
Selected and implied templates:
  ZopeSkel#basic_namespace  A project with a namespace package

Variables:
  egg:      Products.myproduct
  package:  productsmyproduct
  project:  Products.myproduct
Enter namespace_package (Namespace package (like plone)) ['plone']: Products
Enter package (The package contained namespace package (like example)) ['example']: myproduct
... accept defaults to end

If you're using buildout, create your package in the src directory, and add references to it in the develop and instance/eggs sections of buildout.cfg:

develop =
    src/Products.myproduct
...
[instance]
...
eggs =
    ${buildout:eggs}
    ${plone:eggs}
    Products.myproduct

Run bin/buildout and you'll be set up to develop your egg-ready product in the src directory. Turn it into a distribution egg when complete.

It is possible to use packages (including egg-distributed ones) outside the Products namespace/directory as Zope 2 products. Many developers prefer this approach, feeling it unnatural to keep everything in a single, "flat" namespace.

Extra steps are required for this. Prior to Zope 2.10.4, this is also required for products in the Products namespace

We must add a line like the following to the package's configure.zcml:

<configure 
    xmlns="http://namespaces.zope.org/zope"
    xmlns:five="http://namespaces.zope.org/five">

  <five:registerPackage package="." initialize=".initialize" />

</configure>

Secondly, it is important to realize that packages outside the Products namespace are not automatically detected when Zope starts up. If they contain configure.zcml files (as most packages will do), this must be explicitly included from somewhere. This may be:

• Another package's configure.zcml file.
• Zope's site.zcml, the root of all ZCML files, which is found in the etc directory in the instance home.
• A ZCML slug, a one-liner created in the zope instance's etc/package-includes directory, with a name like my.package-configure.zcml.

In all cases, the syntax is the same:

<include package="my.package" file="configure.zcml" />

If you have meta.zcml or overrides.zcml files, you can add <include /> directives for these as well. If you are using slugs, it must be named accordingly, e.g. my.package-meta.zcml or my.package-overrides.zcml. A slug can not contain more than one line.

Later in this tutorial, we will show how buildout can manage slugs for us automatically.

Before we can create a buildout to manage Zope and Plone, there are a few prerequisites to take care of.

As of Plone 3.2, all of the Plone installers are buildout based.  You can get the latest installer and run it to have a working buildout without having to follow these steps. However, these steps are still valid if you want to create the buildout manually with ZopeSkel.

First, you will need an appropriate Python interpreter, if you do not have one already:

• Install Python 2.4 for your platform, and add it to your system PATH. It is easiest if Python 2.4 is what you get when you type python -V on a command line. Make sure you're using Python 2.4 and not 2.5, since Plone 3.x doesn't support Python 2.5 or later. You might need to type python2.4 instead of just python when running some of the following commands.
  
• If you installed Python using an operating system package (e.g. an RPM), make sure you get the development package (e.g. python-devel) as well. This includes Python header files that we will use later to compile Zope. If you installed from source, or used the Python Windows installer, you should already have these.
  
• Install PIL, the Python Imaging Library into this Python interpreter.
• Install setuptools. If you're using Linux and your distributioni doesn't provide a package for setuptools, download ez_setup.py and run it with:

  $ python ez_setup.py
  

  This will download and install setuptools and the easy_install script. Watch the console output to understand where easy_install is installed. If this is not in your system PATH, you should add this directory to the path as well.

Finally, use easy_install to get ZopeSkel, a collection of skeleton templates for Zope and Plone development:

$ easy_install -U ZopeSkel

This will get Paste Script and various other dependencies.

Linux note: If you're installing setuptools and ZopeSkel system-wide, you will probably need to become superuser or use sudo, if you're not using virtualenv or similar. But please note that bin/buildout (introduced later) should never be run as root. If you really can't avoid running this script as root, don't forget to change the owner of created files (chown -R) so the unprivileged user that runs the zope instance will be able to read those files.

If you added the Python console scripts directory (where easy_install was placed) to your system path, you should now be able to run the paster command. You can test it with:

$ paster create --list-templates
Available templates:
  basic_namespace:          A project with a namespace package
  basic_package:            A basic setuptools-enabled package
  basic_zope:               A Zope project
  nested_namespace:         A project with two nested namespaces.
  plone:                    A Plone project
  plone2.5_theme:           A Theme for Plone 2.5
  plone2_theme:             A Theme for Plone 2.1 & Plone 2.5
  plone3_buildout:          A buildout for Plone 3 projects
  plone3_theme:             A Theme for Plone 3.0
  plone_app:                A Plone App project

Your output may differ slightly, but make sure you have the plone3_buildout and plone templates at least.

Additional installation steps for Windows

If you are using Windows, there are a few more things you need to do.

First, get and install the Python Win32 extensions for Python 2.4.

If you intend to compile Zope yourself, rather than using a binary installer, or if you ever need to compile an egg with C extensions, you will need the mingw32 compiler. Make sure you choose the "base" and "make" modules at a minimum when the installer asks. By default, this installs into C:\MingW32. Inside the installation directory, there will be a bin directory, e.g. C:\MingW32\bin. Add this to your system PATH.

Finally, you need to configure Python's distutils package to use the mingw32 compiler. Create a file called distutils.cfg in the directory C:\Python24\Lib\distutils (presuming Python was installed in C:\Python24, as is the default). Edit this with Notepad, and add the following:

[build]
compiler=mingw32

We are now ready to create a new buildout. The "buildout" is a directory containing all the parts that make up a project, including a Zope instance, the Plone sources, custom configuration options, and your our project's source code.

As of Plone 3.2, all of the Plone installers are buildout based.  You can get the latest installer and run it to have a working buildout without having to follow these steps. However, these steps are still valid if you want to create the buildout manually with ZopeSkel.

Create one like this:

$ paster create -t plone3_buildout myproject

This will ask a series of questions. If you want to use an existing installation of Zope rather than have buildout download and compile one for you, specify an absolute path as the zope2_install. Similarly, if you do not want buildout to download the core Plone products, you can point it to an existing directory containing all the products (it will still download Plone 3's eggs, but as we will see later, it is possible to share an eggs directory among multiple buildouts). You will need to enter a Zope administrator username and password, and you may want to turn debug mode and verbose security on during development.

Now, enter the newly created myproject directory, and run the buildout bootstrap script.  NOTE: Python 2.4 is currently required to Plone 3.x:

$ cd myproject
$ python2.4 bootstrap.py

This will create a number of directories and scripts and dowload the latest version of the zc.buildout egg. This step should be needed only once.

To get started straight away, run:

$ ./bin/buildout

This reads the generated buildout.cfg file and executes its various "parts", setting up Zope, creating a Zope instance, downloading and installing Plone. We will explain this file in more detail shortly.

You will need to run ./bin/buildout again each time you change buildout.cfg. If you do not want buildout to go online and look for updated versions of eggs or download other archives, you can run it in non-updating, offline mode, with;

$ ./bin/buildout -No

To start Zope in foreground and debug mode, run:

$ ./bin/instance fg

The instance script is analogous to zopectl as found in a standard Zope instance. You can use ./bin/instance start to run Zope in daemon mode. It can also be used to run tests:

$ ./bin/instance test -s plone.portlets

Running:

bin/instance console

is equivalent to bin/instance fg, but does not implicitly turn on debug mode but respects the debug-mode setting in buildout.cfg. This can be useful to run Zope in non-development mode with daemon-control programs like supervisord.

Once your buildout installation is up and running, you will still need to install a Plone site.  Log in to the Zope Management Interface (ZMI) and from "select type to add..." choose Plone Site.  Fill in the required details and submit.  Now you have a Plone site at the ID that you specified.

Directories in the buildout

Before we dive into buildout.cfg, let us take a quick look at the directories that buildout has created for us:

bin/

Contains various executables, including the buildout command, and the instance Zope control script.

eggs/

Contains eggs that buildout has downloaded. These will be explicitly activated by the control scripts in the bin/ directory.

downloads/

Contains non-egg downloads, such as the Zope source code archive.

var/

Contains the log files (in var/log/) and the file storage ZODB data (in var/filestorage/Data.fs). Buildout will never overwrite these.
If you want to import a .zexp file, place it in the var/instance/imports folder.

Previously one had to put that file into parts/instance/import, but this folder gets wiped and regenerated when running bin/buildout, so the import location was changed.

src/

Initially empty. You can place your own development eggs here and reference them in buildout.cfg. More on that later.

products/

This is analogous to a Zope instance's Products/ directory (note the difference in capitalisation). If you are developing any old-style Zope 2 products, place them here. We will see how buildout can automatically download and manage archives of products, but if you want to extract a product dependency manually, or check one out from Subversion, this is the place to do so.

parts/

Contains code and data managed by buildout. In our case, it will include the local Zope installation, a buildout-managed Zope instance, and Plone's source code. In general, you should not modify anything in this directory, as buildout may overwrite your changes.

You can check in a buildout directory to a source code repository to share it among developers. In this case, you should ignore the directories bin/, eggs/, downloads/, var/, and parts/. Each developer can run bootstrap.py to get these back, and will normally need local copies anyway. All your configuration should be in the buildout.cfg file, and all custom code in src/ or products/.

Important note:This document applies to Plone 3.2 onwards. In Plone versions prior to 3.2 the vanilla buildout.cfg file was significatively different because Plone wasn't fully eggified.

buildout.cfg is the most important file in your new buildout environment. Here is how it looks:

[buildout]
parts =
    zope2
    productdistros
    instance
    zopepy

# Change the number here, and in find-links below, to change the version of
# Plone being used
extends = http://dist.plone.org/release/3.3/versions.cfg
versions = versions

# Add additional egg download sources here. dist.plone.org contains archives
# of Plone packages.
find-links =
    http://dist.plone.org/release/3.3
    http://download.zope.org/ppix/
    http://download.zope.org/distribution/
    http://effbot.org/downloads

# Add additional eggs here
eggs =
    
# Reference any eggs you are developing here, one per line
# e.g.: develop = src/my.package
develop =

[zope2]
recipe = plone.recipe.zope2install
url = ${versions:zope2-url}

# Use this section to download additional old-style products.
# List any number of URLs for product tarballs under URLs (separate
# with whitespace, or break over several lines, with subsequent lines
# indented). If any archives contain several products inside a top-level
# directory, list the archive file name (i.e. the last part of the URL, 
# normally with a .tar.gz suffix or similar) under 'nested-packages'.
# If any archives extract to a product directory with a version suffix, list
# the archive name under 'version-suffix-packages'.
[productdistros]
recipe = plone.recipe.distros
urls =
nested-packages =
version-suffix-packages = 

[instance]
recipe = plone.recipe.zope2instance
zope2-location = ${zope2:location}
user = admin:admin
http-address = 8080
# comment the following two options in production sites
debug-mode = on
verbose-security = on

# If you want Zope to know about any additional eggs, list them here.
# This should include any development eggs you listed in develop-eggs above,
# e.g. eggs = Plone my.package
eggs =
    Plone
    ${buildout:eggs}

# If you want to register ZCML slugs for any packages, list them here.
# e.g. zcml = my.package my.other.package
zcml = 

products =
    ${buildout:directory}/products
    ${productdistros:location}

[zopepy]
recipe = zc.recipe.egg
eggs = ${instance:eggs}
interpreter = zopepy
extra-paths = ${zope2:location}/lib/python
scripts = zopepy

Let us walk through this file step-by-step:

The main [buildout] section

The [buildout] section is the starting point for the file. It lists a number of "parts", which are configured in separate sections later in the file. Each part has an associated recipe, which is the name of an egg that knows how to perform a particular task, e.g. build Zope or create a Zope instance. A recipe typically takes a few configuration options.

Our global settings are as follows:

[buildout]
parts =
    zope2
    productdistros
    instance
    zopepy

find-links =
    http://dist.plone.org/release/3.3
    http://download.zope.org/ppix/
    http://download.zope.org/distribution/
    http://effbot.org/downloads

eggs =
    
develop =

This specifies that the parts zope2, productdistros, instance and zopepy will be run, in that order. Then, we tell buildout that it can search one of a number of URLs when it is looking for eggs to download. In addition, it will always search the Cheese Shop.

Note that configuration entries are commonly split into multiple lines.  For this to work, all lines after the first must begin with at least 4 spaces.

Next, we can list any eggs that buildout should download and install for us. This may include version specifications. For example, if you want sqlalchemy 0.3,  but not 0.4, you could list;

eggs = 
    sqlalchemy>=0.3,<0.4dev

Finally, we can list development eggs, by specifying a directory where the egg is extracted in source format. For example:

eggs =
    my.package

develop = 
    src/my.package

This presumes that there is an egg called my.package in the src/ directory. We will learn how to create such eggs a little later in this tutorial. Notice how we must also list my.package as an actual egg dependency: development eggs are not automatically added to the "working set" of eggs that are installed for Zope.

The extends and versions lines

This part was introduced with Plone 3.2. It references a remote file where the version of each needed package is specified. Check that remote file to see yourself how these dependencies are specified.

# Change the number here, and in find-links below, to change the version of
# Plone being used
extends = http://dist.plone.org/release/3.3/versions.cfg
versions = versions

If you want to use a local file instead of a remote one to be able to work offline, download it to your buildout directory and reference it like this:

extends = versions.cfg

The [zope2] section

This part builds Zope 2, using plone.recipe.zope2install. If you specified an existing Zope installation, you will not have this part. Otherwise, it looks like this:

[zope2]
recipe = plone.recipe.zope2install
url = ${versions:zope2-url}

Here, we reference the download location for Zope as present in the versions file. This ensures that we always get the recommended version of Zope. You could specify a download URL manually instead, if you wanted to use a different version of Zope.

When the recipe is run, Zope 2 is installed in parts/zope2. The Zope software home becomes parts/zope2/lib/python.

The [productdistros] section

This uses the plone.recipe.distros recipe, which is able to download distributions (archives) of Zope 2 style products and make them available to Zope. It is empty to begin with:

[productdistros]
recipe = plone.recipe.distros
urls =
nested-packages =
version-suffix-packages =

However, you can list any number of downloads. The recipe is also able to deal with archives that contain a single top-level directory that contains a bundle of actual product directories (nested-packages), or packages that have a version number in the directory name and thus need to be renamed to get the actual product directory (version-suffix-packages).

Consider the following distributions:

# A typical distribution 
ExampleProduct-1.0.tgz
 |
 |- ExampleProduct
     |
     |- __init__.py
     |- (product code)

# A version suffix distribution
AnotherExampleProduct-2.0.tgz
 |
 |- AnotherExampleProduct-2.0
     |
     |- __init__.py
     |- (product code)

# A nested package distribution
ExampleProductBundle-1.0.tgz
 |
 |- ExampleProductBundle
     |
     |- ProductOne
     |   |- __init__.py
     |   |- (product code)
     | 
     |- ProductTwo
         |- __init__.py
         |- (product code)

Here is what the part would look like if we try to install the three distributions above:

[productdistros]
recipe = plone.recipe.distros
urls =
    http://example.com/dist/ExampleProduct-1.0.tgz
    http://example.com/dist/AnotherExampleProduct-2.0.tgz
    http://example.com/dist/ExampleProductBundle-1.0.tgz
nested-packages = ExampleProductBundle-1.0.tgz
version-suffix-packages = AnotherExampleProduct-2.0.tgz

You can specify multiple downloads on separate lines. When the recipe is run, the product directories for downloaded products are found in parts/productdistros.

The [instance] section

The instance section pulls it all together: It configures a Zope instance using the plone.recipe.zope2instance script. Here is how it looks:

[instance]
recipe = plone.recipe.zope2instance
zope2-location = ${zope2:location}
user = admin:admin
http-address = 8080
# comment the following two options in production sites
debug-mode = on
verbose-security = on

eggs =
    Plone
    ${buildout:eggs}

zcml = 

products =
    ${buildout:directory}/products
    ${productdistros:location}

Here, we reference the Zope 2 installation from the [zope2] part - if you specified a location yourself when creating the buildout, you would see that one here. Then, we specify the initial admin user and password used only when creating the initial database, and the port that Zope will be bound to. We also turn on debug mode and verbose security. They are useful for development, but remember to turn them off in production sites since they can compromise the security of your site. These options are used to generate an appropriate zope.conf file for this instance. See the recipe page in the Cheese Shop for more details on the options available.

Next, we specify which eggs that will be made available to Zope. This references the "global" eggs from the [buildout] section, as well as Plone itself. You could add additional eggs here, though it is generally easier to specify these at the top of the file, so that they get included in the ${buildout:eggs} working set.

Zope 3 configure.zcml files are not automatically loaded for eggs or packages that lack z3c.autoinclude support and are not in the Products namespace. To load ZCML files for a regular package, we can make buildout create a ZCML slug by listing the package under the zcml option:

zcml =
    my.package
    my.package-overrides

This assumes that my.package was previously referenced in the buildout. This would load both the main configure.zcml and the overrides.zcml file from this package.  Over time, the need for these entries should diminish, as z3c.autoinclude support becomes widespread.

Finally, we list the various directories that contain Zope 2 style products - akin to the Products/ directory in a traditional instance. Notice how the products/ directory in the main buildout directory comes first, followed by the products downloaded with the [productdistros] part.

When the recipe is run, the Zope instance home will be parts/instance, and a control script is created in ./bin/instance.

The [zopepy] section

This final section creates a Python interpreter that has all the eggs and packages (but not Zope 2 style products) that Zope would have during startup. This can be useful for testing purposes.

[zopepy]
recipe = zc.recipe.egg
eggs = ${instance:eggs}
interpreter = zopepy
extra-paths = ${zope2:location}/lib/python
scripts = zopepy

Here, we copy the eggs from the [instance] section, and include in the pythonpath the Zope instance home.

When the recipe is run, the script will be created in ./bin/zopepy.

To set "global" options affecting all buildouts, create a directory .buildout (note leading dot) in your home directory, and add a file there called default.cfg. Any option set here will be applied to the corresponding section in any buildout.cfg that you run, unless it is overridden by a more specific option in the buildout.cfg file itself.

The most common options are:

executable

Specify a python interpreter other than the system default. This is useful if you have Python 2.5 installed, say, but you want your buildouts to use another installation of Python 2.4.

eggs-directory

Specify a directory where eggs will be downloaded. This allows multiple buildouts to share the same eggs, saving disk space and download time. Note that only those eggs explicitly required by a particular buildout will be activated. The eggs directory may contain many more eggs (or many different versions of the same package) than what is used at any one time.

download-cache

Specify a shared directory for downloaded archives. Again, this can save disk space and download time. NOTE: before zc.buildout 1.0, this was called download-directory

extends-cache

Specify a shared directory for extended buildout configurations that are downloaded from a URL. As of Plone 3.2 this is how Plone pins the versions of its eggs. This option was added in zc.buildout 1.4.1, prior to that the offline mode in combination with a extends URL would not work.

Here is an example ~/.buildout/default.cfg setting all three:

[buildout]
executable = /opt/python24/bin/python
eggs-directory = /home/username/.buildout/eggs
download-cache = /home/username/.buildout/downloads
extends-cache = /home/username/.buildout/extends

This assumes Python 2.4 is installed in /opt/python2.4. For the last two options to work, you would need to create the directories eggs and downloads inside the ~/.buildout directory.

How to install a new third-party products will depend on whether it is packaged as an egg, or a traditional Zope 2 product.

Installing eggs

So long as an egg has a release in the PyPi or elsewhere, buildout can download and install it, including any explicitly specified dependencies. Simply list the egg, and optionally a version (otherwise, you get the latest available), in the eggs option.

[buildout]
...
eggs = 
    elementtree
    borg.project>=1.0b1,<2.0dev

If you want buildout to search an index other than PyPi's, you can add a URL to find-links that contains download links for the eggs. In fact, we have already seen an example of this: elementtree is found at http://effbot.org/downloads, not in PyPi directly. Thus, we have:

[buildout]
...

find-links =
    http://dist.plone.org
    http://download.zope.org/ppix/
    http://download.zope.org/distribution/
    http://effbot.org/downloads

eggs =
    elementtree

We have also listed some of the download locations for Zope and Plone eggs.

Again - re-run buildout for the changes to take effect:

$ ./bin/buildout

Development eggs

If there is not a release for your egg, or you want to track an egg in Subversion, check it out to the src/ directory. Make sure you get the full egg, including the top-level setup.py file. For example, to get the plone.portlets trunk development, egg do:

$ cd src
$ svn co https://svn.plone.org/svn/plone/plone.portlets/trunk plone.portlets

Then, add the following to buildout.cfg:

[buildout]
...
eggs =
    ...
    plone.portlets

develop =
    src/plone.portlets

Note that:

• The develop option contains a relative path to where the source egg is installed. Buildout will expect to find a suitable setup.py in this directory.
  
• Development eggs always take precedence over regular eggs.
• You still need to list the egg name  in the eggs option for it to be installed.
• If you are overriding an egg that ships with Plone, you may need to list it in the eggs section of the [plone] part instead:

[buildout]
...
develop =
    src/plone.portlets

...

[plone]
recipe = plone.recipe.plone
eggs = 
    plone.portlets

This is because plone.recipe.plone is very expilcit about which versions of its various eggs to use, to ensure Plone keeps running as it was released.

Buildout recipes (such as plone.recipe.plone) are distributed as eggs. You can use a development egg of a recipe by listing it under the develop option. There is no need to explicitly list it under the eggs option, since it is referenced by the recipe option of the relevant part.

Installing a traditional Zope 2 product

The easiest way to try out a traditional Zope 2 product is to extract it into the products/ folder inside the buildout. If you see documentation referring to the Products/ folder in a Zope instance, this is the same thing.

However, this approach makes it harder to redistribute your project and share it with other developers. It is often more predictable to let buildout download and install the package for you. You can do this with the [productdistros] section of buildout.cfg. For example, here is how you might install a product named ExampleProduct and a set of products named ExampleProductBundle:

[productdistros]
recipe = plone.recipe.distros
urls =
    http://example.com/dist/ExampleProduct-1.0.tgz
    http://example.com/dist/ExampleProductBundle-1.0.tgz
nested-packages =
    ExampleProductBundle-1.0.tgz
version-suffix-packages =

Note that our fictional ExampleProductBundle is distributed as a single directory containing a number of products in sub-directories, so we list it under nested-packages.

As always, if you change buildout.cfg, you must re-run buildout:

$ ./bin/buildout

Managing ZCML files

It is important to realize that Zope will not load configure.zcml files automatically for packages that are not in the Products.* namespace and lack support for z3c.autoinclude (see next page for more on using z3c.autoinclude). Instead, you must explicitly reference the package. Buildout can create such a reference (known as a ZCML slug) with the zcml option under the [instance] part. Here is how to ensure that borg.project is available to Zope:

[buildout]
...
eggs =
    elementtree
    borg.project

...

[instance]
...
zcml = 
    borg.project

Should you need to load an overrides.zcml or a meta.zcml, you can use a syntax like:

zcml =
    some.package
    some.package-overrides
    some.package-meta

Policy products

Many developers prefer to create a single "policy product" (also known as a "deployment product") that orchestrates various dependencies. If you have such a product, you may want to include various dependencies directly from the policy product's configure.zcml file, with lines such as:

<configure xmlns="http://namespace.zope.org/zope">

    <include package="borg.project" />

</configure>

In this case, you may still need one slug (using the zcml option as above) for the policy product.

Creating a traditional Zope 2 product

To create a traditional Zope 2 product, put it in the top-level products/ directory and re-start Zope. Nothing more should be required. As explained previously, products placed here will be found automatically at start-up, and their configure.zcml files will be executed automatically.

Creating an egg

Of course, if you are using products, you cannot benefit from the additional features of eggs, including automatic dependency
management, distribution via the Cheese Shop and nested namespaces.

The easiest way to create a new egg is to use the paster command, which we already used to create the buildout. To create a new basic package, with a top-level namespace (e.g. your company name) and a specific name, go to the src/ directory and run:

$ cd src
$ paster create -t plone myorg.mypackage

You will be asked a series of questions. Make sure that the namespace package and package name correspond to the name of the egg. In this case, the namespace package is myorg and the package name is mypackage. In general, answer False to the question on whether your package if "zip safe". Enter other metadata as requested.

You will now have:

• A setup.py which contains the metadata you entered
• A package in myorg.mypackage/myorg/mypackage. Your source code goes here.
• A skeleton configure.zcml, tests.py and a few other useful starting points.
• Some generic documentation in myorg.mypackage/docs.

Of course, you must also add this package to the buildout. In buildout.cfg, you might have:

[buildout]
...
eggs =
    ...
    myorg.mypackage

develop =
    src/myorg.mypackage

Unless you plan to include this package from another one (or use automatic ZCML loading, explained below), you probably also need a ZCML slug:

[instance]
...
zcml =
    myorg.mypackage

Do not forget to re-run buildout after making the change:

$ ./bin/buildout

Automate ZCML loading for your package

If you're not including your package from another one, you can still avoid having to include a ZCML slug in buildout.cfg for it. This is particulary useful to avoid unneccessary repetition of package names in buildout.cfg, which begginer integrators might easily overlook. From Plone 3.3 on, you can make your packages signal that their ZCML should be included by adding:

setup(...
entry_points="""
...
[z3c.autoinclude.plugin]
target = plone
...)
"""

to their setup.py file. For further information, see the setuptools documentation about dynamic discovery of services and plugins.

Specifying dependencies

If your new package has explicit dependencies, you can list them in setup.py. That way, buildout will be able to download and install these as well. Dependencies are listed in the install_requires argument to the setup() method, By default, setuptools is listed here, since we need this to support namespace packages. To add sqlalchemy 0.3 (but not 0.4), and the MySQL-Python driver, you could amend this to read:

install_requires=[
          'setuptools',
          'sqlalchemy>=0.3,<0.4dev',
          'MySQL-Python',
      ],

Uploading your egg to the Cheese Shop

If you want to share your packge with the rest of the Python community and make it easy to install using tools like buildout and easy_install, you can upload the package to the Cheese Shop.

Before doing so, you should:

• Commit your latest changes and tag the release in Subversion, if applicable.
  
• Remove (temporarily) the setup.cfg file: this makes your package a development release.
  
• Make sure the version number in setup.py is correct. This should use common conventions such as "1.0b2" for the second beta of version 1.0, or "2.1.3rc1" for the first release candidate of version 2.1.3.
• If you are using Mac OS X, run export COPY_EXTENDED_ATTRIBUTES_DISABLE=true on the shell first - otherwise, the egg will contain Mac OS X resource forks which cause problems if your egg is used on Windows.

When you are ready, run the following command from your package's directory (e.g. src/myorg.mypackage):

$ python setup.py egg_info -RDb "" sdist register upload

This will ask you to create a Cheese Shop account if you do not have one already. You can run this command as often as you'd like to release a new version (probably with a new version number).

Finally, let's take a look at a more advanced configuration, better suited for deployment. Save this file as deployment.cfg, at the root of the buildout next to the main buildout.cfg file:

[buildout]
extends =
    buildout.cfg

parts +=
    debug-instance
    zeoserver
    varnish-build
    varnish-instance

[zeoserver]
recipe = plone.recipe.zope2zeoserver
zope2-location = ${instance:zope2-location}
zeo-address = ${instance:zeo-address}

[instance]
recipe = plone.recipe.zope2instance
zope2-location = ${zope2:location}
zeo-client = true
zeo-address = 8100
zodb-cache-size = 5000
zeo-client-cache-size = 300MB
debug-mode = off
verbose-security = off
eggs += Products.CacheSetup

[debug-instance]
recipe = collective.recipe.zope2cluster
instance-clone = instance
http-address = 8081
debug-mode = on
verbose-security = on

[varnish-build]
recipe = zc.recipe.cmmi
url = http://downloads.sourceforge.net/varnish/varnish-2.0.2.tar.gz

[varnish-instance]
recipe = plone.recipe.varnish
daemon = ${buildout:parts-directory}/varnish-build/sbin/varnishd
bind = 127.0.0.1:8082
backends = 127.0.0.1:8080
cache-size = 1G

Here, we are:

• Referencing the main buildout.cfg file, extending and overriding it with configuration more appropriate for deployment.
  
• Setting up a ZEO server with two client instances, instanceand debug-instance (see plone.recipe.zope2zeoserver and plone.recipe.zope2instance for more details)
• Compiling the Varnish cache server (see plone.recipe.varnish for more details).

By combining buildout configuration files like this, you can create tailor-made configurations for different deployment scenarios. To learn more about the advanced features of buildout, see its documentation.

To build this environment, you must explicitly specify a configuration file:

$ ./bin/buildout -c deployment.cfg

To start Zope and Plone, you will need to start the ZEO server, the instance and the Varnish server:

$ ./bin/zeoserver start
$ ./bin/instance start
$ ./bin/varnish-instance

If you need to bring up an instance for debugging then you can start up the debug-instance in foreground mode.

$ ./bin/debug-instance fg

The recipes will also create scripts to back up the ZODB filestorage (in ./bin/repozo) and to pack the database (in ./bin/zeopack).

Further options

zc.buildout is a very flexible system. It is relatively easy to create new recipes, and you can combine existing recipes in powerful ways. Search the Cheese Shop for "buildout" to find more recipes, or take a look at the source code for some of Plone's own recipes to understand how recipes are created.

The list is more or less sorted by topic. Check all available recipes at PyPI.

• zc.recipe.egg - Installs eggs into a buildout eggs directory. It also generates scripts in a buildout bin directory with egg paths baked into them.
• infrae.subversion - This zc.buildout recipe will check out a number of URLs into its parts directory. It won't remove its parts directory if there are any changes in the checkout, so it's safe to work with that checkout for development.
• plone.recipe.zope2install - Installs Zope 2, i.e. its Python libraries and scripts, but doesn't create any instance.
  
• plone.recipe.zope2instance - Creates and configures a Zope 2 instance in parts. It also installs a control script, which is like zopectl, in the bin/ directory.
• plone.recipe.zope2zeoserver - This recipe creates and configures a Zope 2 ZEO server in parts. It also installs a control script, which is like zeoctl, in the bin/ directory.
• plone.recipe.distros -  Installs distributions, i.e. Zope products not packaged as eggs.
• plone.recipe.apache - Builds and configures the Apache web server.
• gocept.nginx - zc.buildout recipe for configuring an nginx server
• plone.recipe.varnish - Installs the Varnish reverse-cache proxy. It works for non-Zope sites as well.
• plone.recipe.squid - Installs the Squid proxy. It works for non-Zope sites as well.
• collective.recipe.omelette - Creates a unified directory structure of all namespace packages, symlinking to the actual contents, in order to ease navigation.
• collective.recipe.i18noverrides - Creates an i18n directory within one or more Zope 2 instances in your buildout. It copies some .po files to those directories. The translations in those .po files will override any other translations.
• zc.recipe.cmmi - The Configure-Make-Make-Install recipe automates installation of configure-based source distribution into buildouts.
• plone.recipe.command - Execute arbitrary commands in buildout through os.system.