Build products for each major feature

For every feature of the site that might be re-used in another site, build a product for this. For example, I recently wanted a Plone site that allowed Plone to use the "sent to friend" feature for any page, based on URL, not just based on Plone content. This only required changing 3 skins from CMFPlone, and could have been customized for this site. Instead, I chose to separate it out to the product SendToURL, allowing me to add this to any site instantly. And, if I want to customize these skins further for another site, I can do so.

Build a site product for site customization

For the site itself, build a "site product" to hold, at the very least, the setup scripts, graphics, skins, etc. Keep this information, which is very site-specific, from the features.

Avoid disconnected External Methods, scripts, etc.

External Methods or other kinds of page templates or scripts that are hanging out in your ZODB end up being disconnected and hard to maintain. Make them part of your feature products, or if they're only for the site itself, part of your site product.

Use your favorite editing environment

Don't underestimate the value of this. I use vim, which launches quickly (so it can be used easily with ExternalEditor TTW), and which practically every server already has installed (so I can use through ssh). However, even considering that, I find myself almost twice as productive being able to work in my exact editing environment, with my macros, scripts, paths, etc., already set.

One word: grep!

grep and find are the godsends of working on the filesystem. Once you've reacquainted yourself with them after years of ignoring them for the ZMI, you'll wonder what you were thinking.

Source code management

You can use your usual source code management tools.

Staging/replication

It is easy to replicate your setup across multiple instances.

Goal: no skins or scripts in ZODB. Only config results and content.

Our end goal is that the only thing in the ZODB will be our actual content objects and configuration settings made by scripts. All of our skins and all of our scripts will be on the filesystem.

Sometimes, it's very helpful to have scripts or skins be placeful. For example, you might have a different logo in different parts of your site, and a common Zope practice for this would be to have logo.jpg in the root of your site, and a different one in /about. However, this leaves a piece of skin in the site, and ruins our ability to maintain it well. Better is to have the root folder and /about folder have a property--say, logo_name, which tells us what logo to use in this area. Then, we can keep all of these logos on the filesystem, and have achieved our goal of just keeping the configuration part in the ZODB.

Adding New FS-Stored Content

Follow the examples of content types in CMFPlone.

In CMFPlone, there are examples of all of the types that can be stored on the filesystem including feature like proxy settings, titles, security settings. grep is your friend here: grep -r --include="*.metadata" proxy * Will give you an example of using a proxy setting in .metadata files.

There's no example of a working ZSQLMethod shipped with CMFPlone, CMFDefault, or CMFPlone. For a discussion of ZSQLMethods and an example of a filesystem-stored ZSQLMethod, see http://joelburton.com/resources/reldb.pdf.

.metadata files for title, security, proxy roles

These are the accompanying files that hold all the "other stuff" for objects, titles, security settings, proxy roles for PythonScripts, and more. These replace the .properties files in earlier versions of the CMF.

One common mistake is to customize a skin object file (say, foo.py) by copying it to a new directory, but not copying any foo.py.metadata. Note that the .metadata file must be in the same directory, so in this case, the customized foo object is used, and it does not get the settings in its metadata file. If this contained important things, like security or proxy settings, this could be disastrous.

Debug mode and FS-stored content

In debug mode, changes to filesystem-stored content are seen immediately. If you're not in debug mode, you'll have to either restart the Zope server or use product refresh.

Note that if you're using SpeedPack under debug mode, you'll see changes to skin objects, as long as they don't get copied from one directory to another. However, if you customize a skin object to a new directory and are running SpeedPack under debug mode, Zope has already cached the old location of the skin object, and won't use the new location version. Restart or use product refresh to have Zope notice this.

Metadata File Example

From register.cpy.metadata :

  [default]
  title=Register a User
  proxy=Manager,Anonymous

  [validators]
  validators = validate_registration

  [actions]
  action.failure=traverse_to:string:join_form
  action.success=traverse_to:string:registered
  action.prefs=traverse_to:string:prefs_users_overview

New FS-Stored Types

It is possible to create your own FS-stored types. If there are non-contentish objects you use often, it's worthwhile creating the small script that will allow it to be stored on the filesystem. You can look at the code in CMFCore for storing the existing types (PageTemplates, PythonScripts, Files, etc.)

An example of this is FSExternalMethod. It is taking an existing Zope object type and creating a FS-stored capable version of it. Found in the product "FileSystemSite".

Atonement for Your Sins - FSDump

Sometimes, it's not possible to work on the filesystem. You may only have access to a web browser or have created many existing skin objects in the ZODB. FSDump will take existing skin objects and dump them out to the filesystem.

FSDump:

• Dumps existing ZODB stuff to FS
• You need to write dumpers for your FS-Stored types.

If you've written custom FS-stored types, you'll have to write your own Dump plug-in for this. This is quite easy to do--see the code in FSDump for examples of how it dumps the basic types.

At the very least — use FSDump for backups

Critical for teams

If your team is larger than one, source code management is essential. It will allow your team to work together more quickly, with much less "stay out of this; I'm working on it", and much less "I'm not sure what this person was doing here". Hands-down, the successful implementation of SCM will be the biggest win for your team's coordination and performance.

Helpful for code archaeology

When faced with old code of your own, or someone else's code, seeing the log messages and way it was built is often incredibly helpful in finding bugs and maintenance.

Branches

Often, you'll work for a day or two on a new idea, only to figure out that it isn't working out, you've screwed up, and you can't remember all the billions of things you changed while on a tear to try out this new idea. This is exactly what branches in a version control system are meant to manage.

Learn how to use branches for your VC system. They're easier than you think, especially in Subversion.

Subversion

• Similar to CVS, but redesigned from ground-up
• Commits are for an entire changeset, not just files
• Easier to understand relationship of commits
• Sane Python API for utilities

Subversion in 1 Minute

• svnadmin create /var/lib/svn
• Create your product structure and files
• svn import *url to repository*
• Excellent book on Subversion published by O'Reilly - based on the online book.

SCM Not Just for Coders

• Graphic/shell front ends for Windows, Linux, and OS X
• Simple enough for designers and scripters to use
• Easy to sell — "You don't have to worry"

At first, the task of having non-developers use your version control system seems daunting. I've found, however, the non-developers can love it when they realize they can easily work off files on their computer and sync it with the server, and they understand that they don't have to "worry" about mistakes. It's all in how you sell this idea.

Best Practices for SCM

• Log messages are your friend
• So don't treat them like your enemy:

Refer to collector items in messages

It's good to pick a simple, standard syntax for this. I frequently use the phrase "Collector #123", and can build web tools that allow you to jump right to that bug to see the details of what you were tryin to fix. This helps close the loop on why you were making these changes in the first place.

Focus on why, not what

Of course you were editing login_form. We can see that. Why, though, did you make those changes? What was broken? What client request does this address? This is the information you'll want later. Explanations of what you're doing should be in the code as comments, anyway.

Check in "pristine" copy as first copy

If you want to customize Plone's login_form, for example, don't copy it to your site product directory and immediately start hacking on it. Instead, copy it to your directory, check it in right then, in its pristine form, then start hacking away. Now, you've solved two problems: (a) it's trivial for you to diff revision 1 and revision 2 of this file to find out why you were customizing it in the first place, and (b) when Plone is upgraded and there are changes to the shipped login_form, it's much easier to incorporate those, since you know exactly how login_form looked when you started, without having to dig around and find that version.

These things take only a tiny bit of discipline — and they really pay off later. It makes it much easier to understand why you customized a Plone skin two months back.

Throw out your database. It's liberating. —Kapil Thangavelu

Everyone that's worked with Zope for more than a few months has encountered "ZODB Dread": that awful, sinking feeling that you've sunk a chunk of your very life into a single, binary-format object database, with no hope you'll ever be able to remember all the scripts, skins, properties, and settings you've put into it. You konw that if this puppy ever gets badly corrupted, you're going to be in a world of hurt. This is what we want to avoid, and that's why we create setup code on the file system.

Writing Setup Code

General advice:

• Each setup function is method that you can call independently.
• Registry of functions.
• Can either prevent calling twice, or delete and re-create, as appropriate.

How I Learned To Stop Worrying and Love the API

DocFinderTab

DocFinderTab gives you instant, through-the-web access to the API for any object you can get to in the ZMI. In many ways, it's nicer than looking through the source code, since you see all the methods of the base classes, and nicer that looking in the debugger, because you get things arranged by base class. This product is dead simple to install and use. There's no excuse for not trying it out today. This really should get shipped as part of Zope.

Epydoc

Epydoc is a smarter and more featureful version of the pydoc module that ships with Zope. It builds handsome, indexed API documentation for your product, or even for Zope and Plone itself. It can even generate this as a PDF, which impresses clients and saves you time in creating this kind of documentation. Plus, actually seeing your docstrings typeset is a good incentive to write better ones.

Other products' Install.py files

A great way to see how to configure things is to see how other products do it, of course. Look at the Install.py for your favorite product. For example, to learn how to install new workflows from disk, see how we do this in PloneHelpCenter (in the collective.)

CMF 1.5

Includes a XML dumper for many (but not yet all) CMFCore/Default tools

So, you can make the changes in the ZMI, quickly and intuitively, then get a snapshot of these chages. These snapshots can be checked into your version control system, diff'ed, etc. And you can restore from a snapshot, makig it easier to step back to a different setup.

Won't need to make API calls

For most things, at least.

CMF 1.5-compatibility and dumpers for our tools might land for Plone 2.1.

Developer Documentation

• Use Python interfaces
• Definitely use your docstrings
• Alpha-test your documentation on other developers
• Unit tests rule!

Useful Documentation Products

• DCWorkflowGraph
• DCWorkflowDump
• EpyDoc
• ArchGenXML

Debugging

(For a more thorough example of working in the debugger, see Using PDB)

• Life without debuggers isn't worth living
  • Simple problems get solved in 2 minutes
  • Complicated problems are possible to solve

The Debugger is Your Friend

• Using debugger with ZEO in Zope 2.7

  zopectl debug will enter the debugger under ZEO.

• Can examine anything
• Can execute arbitrary code, change variables, change ZODB

Executing Requests

• Zope.debug(...)
  • Read documentation on debug with Zope
  • , u="user:password"
  • , pm=1 for postmortem

Playing in the ZODB

Often, even more helpful than debugging is the ability to simple examine your ZODB directly outside of any request or debug-stepping. Once you're used to this, you'll probably find you keep the debugger open all the time while you're developing and debugging, just to quickly see how things are actually created and working in Zope.

• Most useful feature
• Can interactively examine and change ZODB outside of debugging
• The root of ZODB is app

ZODB Changes

• To commit your transaction: get_transaction().commit()
• To sync yourself to the current ZODB: app._p_jar.sync()

Graphical Debuggers

• BoaConstructor
  • Can debug ZODB Python Scripts
  • Can build Zope objects
• WingIDE
  • Powerful remote debugging
  • Can debug FS-stored Python Scripts
• Komodo
  • Regex debugging
  • Excellent IDE

Developing, Staging, Syncing

• Working off laptop and moving to server
• Moving from staging server to production server
• Working in teams

You Have a Laptop: Use It

• It's not a $3000 ssh client
• Faster edit/test/debug cycle.
• No wires, no wireless
• Easier to work privately

Subversion for Sharing

• Work on your laptop
• When at the "right point", check-in your code
• rsync to your sandbox on server
• Emergency changes on server can be handled, too

Simple Sandboxes

• Each developer gets a skin folder in your product
• rsync your skin changes to your skin folder
• Each developer has their own skinpath, with their skinfolder as most customized
• Developers can switch from their sandbox skinpath to common skinpath.

Synchronizing of Content

• Create starter content in setup scripts
• Create starter content in .zexp files
  • Can't change, though
• Use ZSyncer to sync from one server to another
• Use AT XMLTool or Marshall to export/import XML of content

Testing

• PloneTestCase makes it easy
• Trade off a bit of time for a lot of worry
  • Think broadly about the benefits

    For example, good tests will let you re-use more of your code, because you'll have the confidence to do so.

• Look at a product with good existing tests — such as ATContentTypes
• Mechanize-driven testing for interface elements
• Consider billing this separately to your customers

Sample Test Setup

• Sample test setup for PloneHelpCenter:

    class PHCSiteTestCase(ArchetypesTestCase.ArcheSiteTestCase):
      """Test structure for PHC"""
  
      def afterSetUp(self):
          ArchetypesTestCase.ArcheSiteTestCase.afterSetUp(self)
  
          self._portal = self.app.portal
          # login as manager
          user = self.getManagerUser()
          newSecurityManager(None, user)
  
          self._portal.invokeFactory(type_name='HelpCenter',
            id='helpctr')
          self._hc=getattr(self._portal, 'helpctr')
  
      def beforeTearDown(self):
          ArchetypesTestCase.ArcheSiteTestCase.beforeTearDown(self)
          noSecurityManager()
          del self._hc
  

Sample Test

• Sample test for PloneHelpCenter:

    class TestFAQ(PHCTestCase):
      """Test those parts of FAQ and FAQ Folder that don't 
      require a real site framework. Allows tests to run faster
      """
  
      def afterSetUp(self):
          PHCTestCase.afterSetUp(self)
          self._dummy = FAQ.HelpCenterFAQ(oid='dummy')
          self._dummy.initializeArchetype()
  
      def test_answerSTX(self):
          dummy = self._dummy
  
          dummy.setAnswer(example_stx, mimetype='text/structured')
          answer=no_whitespace.sub('',dummy.getAnswer())
          self.failUnless(answer==example_html, 'Value is %s' % answer)
  

More information on unit testing

• Lots of material and tutorials available on the web
• Get Stefan Holek's "Unit Testing Zope for Fun and Profit" presentation from EuroPython
• A very good book is Kent Beck's Unit Testing book

Joel Burton

Original author

Kapil Thangavelu

Inspiration to throw out my database and much more

Calvin Hendryx-Parker and David "Whit" Morriss

Ideas and feedback

Alan Runyan

For releasing several products teaching the "right path"

Alexander Limi

PloneHelpCenter conversion, insistence ;)