robust (adjective)

1 a : having or exhibiting strength or vigorous health b : having or showing vigor, strength, or firmness c : strongly formed or constructed : STURDY

If you follow this tutorial, you'll end up with a robust Zope and Plone installation, including

• A ZEO storage server and multiple clients
• Two virtualhosts -- one heavily cached for anonymous visitors, and one secured via https for authenticated users, with redirection between the two as needed [1]
• Scripts for packing the ZODB, backing up your site, starting up on boot, and information about other maintenance "best practices" [1]

as well as the reasoning behind these configuration choices.

This tutorial assumes that you will be deploying Plone for a site that has a large ratio of anonymous viewers to logged-in Members. This is far from the only use-case for Plone - if your situation differs you may want to consider modifications to the steps outlined in the following pages, particularly when it comes to the caching virtualhost configurations.

This tutorial also assumes that you will deploy Zope and Plone on a Unix-based system. I do not have enough experience with Windows to include specific information on how to adapt this installation method for that platform.

Finally, the configuration choices in the tutorial are made with production deployment in mind. Plone ships configured for development, and some of the changes that follow reduce the ability to easily develop Plone products and skins. In any case, best practices suggest that you should create a separate Zope instance and Plone sites for development and do as much as possible in filesystem-based products that can be used by both production and development instances.

Prerequisites:

• The latest available versions of Python 2.3.x and Apache 1.3.x installed
• Two users - one normal user (can be your regular login account) and one unprivileged user to run Zope and ZEO. The normal user should be a member of the same group as the unprivileged user. For example, create a zope user with a zope group and add the normal user to the zope group.
• Root access/ability to restart Apache and edit Apache configuration files
• Intermediate Unix knowledge
• Intermediate Plone knowledge

Versions used at the time of writing:

• Python 2.3.4
• Zope 2.7.3
• Plone 2.0.5
• Apache 1.3.33 (including mod_proxy, mod_rewrite, mod_expires, mod_headers)

[1] When I started this tutorial I had grand plans...unfortunately life, in the form of a new baby and a new full-time job after several years of consulting, has intruded. Stay tuned for a part 2 to this tutorial that will cover the above topics

Why install Zope and Plone from source?

With a Windows and Mac OS X installer available from plone.org, as well as the inclusion of packages in most Linux distributions and BSD ports systems, you may be wondering why I elect to install Zope and Plone from source. The answer is simple and can be expressed in one word: Control. By using a package or port you are giving up some control to the maintainer over directory layout, dependencies, and versions. Some distributions only include outdated versions of Zope and Plone, while others include the latest released versions immediately after they have been released, often without adequate testing. This can be a problem if you need to upgrade a dependency and your package manager insists on upgrading Zope or Plone to a version that you have not tested with your application.

My philosophy regarding OS packaging systems is that I use them to install all of the utilities and applications that make my life easier, but I prefer to install many of the "mission-critical" services from source to have more control over the upgrade cycle. In the world of Zope this is especially pleasant, as it's a trouble-free and quick compile if you have the right version of python.

Should I also install Python and Apache from source?

Depending on your needs and the version that ships with your OS you may also want to install Python or Apache from source. For example, Mac OS X 10.3 ships with Python 2.3, which is a bit out of date. I install 2.3.4 via the excellent DarwinPorts system.

Apache is a special case. I prefer Apache 1.3.x due to its proven reliability. Changes to Apache 1.3 are limited to bug and security fixes, so I am comfortable using the latest version supplied with the OS or packaging system. Of course, if your Apache system depends on add-on modules that may be newer than you are willing to deploy, you should also consider maintaining a source install outside of your packaging system.

Why use ZEO?

ZEO is most known for its clustering abilities - allowing you to distribute your site's load over multiple CPUs. However, there are several reasons to use ZEO even if you are only using one single-processor machine.

• You can dedicate one client to the public site and have a second client that handles batch jobs like packing the ZODB and re-indexing the catalog
• You can debug the live site via zopectl debug - some problems may only show up under a production-level load
• You're future-proofing your site against the day that you do need a cluster of machines

How many ZEO clients?

The ZEO server is I/O bound, while ZEO clients are CPU and RAM bound. I would recommend creating a publicly accessible ZEO client for each CPU you have available, if you have at least 256MB of RAM (preferable 512MB+) for each.

In the interest of simplicity, most of this tutorial will assume that you are working with one single-processor machine and will have two zeo clients -- one for the public site proxied by Apache and one for maintenance/batch purposes. A final section covering performance (TBD) will discuss how to migrate this configuration to a cluster of machines.

Choosing a base directory for the installation and directory layout

Now that we have decided to install Zope and Plone from source, we need to decide where to do so. You should choose an area on the filesystem that is not used by the OS or its packaging system. Your choice may also be influenced by the number of disks in your system(s) and their partitioning and filesystem settings. Common choices might include /opt/zope, /usr/local/zope, or /home/zope/zope (where /home/zope is the home directory of the zope user). The Filesystem Hierarchy Standard (FHS) may be worth consulting.

For simplicity's sake, this tutorial will install Zope and create Zope and ZEO instances and Product directories under one base directory, hereafter referred to as $BASE. For ease of management, I encourage you to replicate the following directory structure via symlinks if you decide to separate components onto different filesystems/disks:

    $BASE/
         client0/
         client1/
         # etc...
         products/
         z2.7.3-p2.3.4/
         zeo/ 
         zope # symlink to currently used version of zope (here z2.7.3-p2.3.4)

I'll explain more about the reasoning for this layout as the tutorial progresses.

Step-by-step Installation

The following steps should be performed with your regular user account, not the account that will eventually run Zope. For security reasons we want to limit what files and directories the zope user can access, so that if there is ever any security problem with Zope or Plone the damage will be as limited as possible.

Get Zope

Download and unpack the Zope sources from http://www.zope.org/Products/ (The recommended version for Plone at the time of this writing is 2.7.3)

Configure, compile and install Zope

Go to the directory that contains the sources and execute the following commands (assuming your zope version is 2.7.3 and your python version is 2.3.4):

      ./configure --prefix=$BASE/z2.7.3-p2.3.4 \
          --with-python=/path/to/python

      make

      make install

      cd $BASE

      ln -s z2.7.3-p2.3.4 zope

Create a ZEO server and two client instances

Once zope is installed, make sure you are in the $BASE directory and run the following commands:

      /path/to/python zope/bin/mkzeoinstance.py zeo 8100

      /path/to/python zope/bin/mkzopeinstance.py --dir=client0
      # Follow the prompts to enter the name and password for 
      # the admin user

      /path/to/python zope/bin/mkzopeinstance.py --dir=client1
      # Follow the prompts to enter the name and password for 
      # the admin user

Configure the clients and server

At the bottom of the zope.conf files found in client[0|1]/etc, comment out the section for zodb_main and uncomment the example for zeo client storage. After you have made your changes, the section should look like this[1]:

      #<zodb_db main>
      #    # Main FileStorage database
      #    <filestorage>
      #      path $INSTANCE/var/Data.fs
      #    </filestorage>
      #    mount-point /
      #</zodb_db>

      <zodb_db temporary>
          # Temporary storage database (for sessions)
          <temporarystorage>
            name temporary storage for sessioning
          </temporarystorage>
          mount-point /temp_folder
          container-class Products.TemporaryFolder.TemporaryContainer
      </zodb_db>

      # Other storage examples
      #
      # ZEO client storage:
      #
      <zodb_db main>
        mount-point /
        <zeoclient>
          server localhost:8100
          storage 1
          name zeostorage
          var $INSTANCE/var
        </zeoclient>
      </zodb_db>

Change the address directive in the http-server and ftp-server sections of one of the client zope.conf files so that the clients can run on separate ports. Set the effective-user for both clients to zope. In zeo/etc/zeo.conf uncomment the user zope line.

Set permissions

Modify filesystem permissions so that the zope user can write to log/ and var/ directories. From the $BASE directory, execute the following commands:

      chgrp zope zeo/var zeo/log client0/var client0/log \
          client1/var client1/log

      chmod g+w zeo/var zeo/log client0/var client0/log \
          client1/var client1/log

Test your installation

Start things up and test with the following commands. You should do this as either the root user (or via sudo) or the zope user:

      zeo/bin/runzeo &

      client0/bin/zopectl fg

When you see Zope ready to handle requests, visit http://example.org:8080/ to make sure everything is running and that client0 can talk to the zeo server. If you see the Zope Quick Start page, you're ready to move on to the next section.

[1] Note that if you are running multiple clients for logged-in users you will also need to share the temporary storage via ZEO so that sessions can be visible to all clients. The configuration above is sufficient to follow along with the rest of this tutorial.

Where to keep Products

ZEO clients that use the same server for storage need to have the exact same products installed. It's easier to just maintain one directory of products and make sure that all clients can see it. Accordingly, I create a $BASE/products directory and create the Products directory within each client as a symlink to the directory[1]:

    $BASE/
        products/
            CMFCore/
            CMFPlone/
            # etc.
        client0/
            Products@ -> ../products
            # etc.
        client1/
            Products@ -> ../products
            # etc.
        # etc.

This directory is where the contents of the Plone-x.y.z tarball and all other add-on products should be placed. To create this structure, download the Plone Core source tarball and place it in /tmp. Then execute the following commands:

    cd /tmp
    tar zxf Plone-x.y.z.tar.gz
    cd $BASE
    mkdir products
    mv /tmp/Plone-x.y.z/* products
    rm -r /tmp/Plone-x.y.z*
    cd client0
    rm -r Products
    ln -s ../products Products
    # repeat the last three steps for all Zeo clients/Zope instances

How to organize and maintain Products

I use subversion to keep this directory of products under version control for the following reasons:

• You can start with a base set of products that are common to several customers or sites and create branches where customer or site-specific products are added or included as shown below.
• You can use subversion's externals feature to include other products that are maintained in separate subversion trees. This will be even more useful as the Collective is moved to subversion. [2]
• You can tag the directory each time you deploy the products to the production server so that you have a repeatable deployment. This makes trouble-shooting a production installation locally as easy as making a new Zope instance, checking out the products directory with the appropriate tag, and copying over the production Data.fs. It also simplifies cluster installations.
• If you need to make changes to products that you did not write but are unable or unwilling to contribute them back to the authors, you can keep them under version control and use svn merge to merge them into newer versions as they are released [3]

I generally check out the products directory into $BASE as a working copy on my development machines, but use svn export to get just the files without the svn metadata on the production and staging machines. This also helps to enforce the rule that no changes should be made in production/staging without being tested in a development environment, as you are unable to check in new files or changes to existing files from an exported copy. (Of course it doesn't stop someone from changing a file and not recording that fact in version control, but the thought of doing that just makes me sick, so let's move on.) On your development machines you will want to configure subversion to ignore .pyc files.

Permissions and ownership

It is important that the products directory and files and directories contained within it have the proper permissions and ownership. The user that Zope runs under needs to be able to write to the directories in order to create the byte-code (*.pyc) files when the products are first installed. However, the source files should not be writable by the Zope user. The solution is to have the products directory and everything under it owned by another user, to change the group ownership of the directories to the zope group, and to turn on the group write bit:

    # The first command is not necessary if you have checked out 
    # or otherwise populated the products directory yourself
    sudo chown -R $youruser:$yourgroup products
    find products -type f|xargs chmod g+r
    find products -type d|xargs chgrp zope
    find products -type d|xargs chmod g+rxw

You will have to run the last two chgrp and chmod commands as root or via sudo if you are not a member of the Zope user's group.

Test your installation

Once your products are in place and the permissions are set, test your installation by starting up (or restarting) Zope and adding a new Plone site from the ZMI. Check to make sure that the *.pyc files are written in the product directories. If everything looks good you can move on to configuring Apache as a front-end.

[1] You can also specify the Products directory explicitly in the zope.conf file for each client/instance.

[2] Make sure to use an URL for the external Product that represents a tag rather than a branch or the trunk, otherwise you may get updates that you haven't tested

[3] See the discussion of vendor branches in the svn book for more information

Resources/Prior Art

• Zope + Apache2
• Caching w/Apache
• and many more that will get added here some day ;)

Credits

• Thanks to Tiran for pointing out an issue with ZEO and sessions

Thanks

• Learning Lab Denmark
• University of Hawaii College of Engineering