How do I put parts of a distributed ZEO-based Plone into read-only mode?

This guide is like http://plone.org/documentation/faq/plone-read-only-mode, only for distributed ZEO based Plone sites. It is assumed that as with most large Plone sites you have broken up your data storage into multiple ZODB FS files as http://plone.org/documentation/kb/mount-portal_catalog-in-separate-zodb describes and that you have a standard buildout based ZEO Plone installation in /usr/local/Plone/zeocluster. It is also assumed that you have already done an initial rsync of the ZODB FS files in order to transfer the bulk of the data - this is safe so long as you repeat the rsync after making the FS files read-only.

Unfortunately it isn't as easy as simply writing 'read-only true' into the top stanza of zeo.conf! Zope clients will try to repeatedly renegotiate the connection if they want write access and ZEO won't let them which appears to you as your site hanging itself - I personally would call this a bug in Zope actually, as at the very least it should timeout or do anything other than silently hang the site. But no matter, and I should add that current versions of Plone (v3.3) seem to need to write data to the root ZODB on startup anyway - otherwise it refuses to start properly.

Hence what we need to do is to make each of the mounted ZODBs read-only, but NOT the root ZODB (usually Data.fs). And because of that constant retry bug in Zope, you need to do the following for not just the ZEO server but also each and every Zope client too in order to prevent the clients retrying. In short, it's fairly painful, but currently there is no alternative (oh for proper buildout support!).

To make the ZEO server read-only, do cd /usr/local/Plone/zeocluster then do nano parts/zeoserver/etc/zeo.conf. For each <filestorage> OTHER than for path / (the root ZODB) do as follows:

<filestorage deepereconomics.org>
  path /usr/local/Plone/zeocluster/var/filestorage/deepereconomics.org.fs
  read-only true
</filestorage>

Now for each client zope N, do nano parts/clientN/etc/zope.conf. For each <zeoclient> in its <zodb_db>, do:

<zodb_db deepereconomics.org>
  <zeoclient>
    server 127.0.0.1:8100
    storage deepereconomics.org
    name deepereconomics.org
    var /usr/local/Plone/zeocluster/var/client1
    read-only true
  </zeoclient>
  mount-point /deepereconomics.org
  container-class OFS.Folder.Folder
</zodb_db>

If you're like me then you have a python script which munges buildout.cfg to automatically append the slightly customised zodb_db stanzas as zope-conf-additional's for each of the client definitions, so if you do then adding in the read-only to all clients at once is actually not too bad. Personally I'd love if the default Plone buildout did all this stuff for you, or at worst it defined variables which meant that I could avoid a specialised customisation of the var variable per client, but no matter.

Finally put your site into maintenance mode and take down ALL client zopes and the ZEO server - I run a varnish reverse proxy cache on the front that will continue serving cached data even when I take down the entire Zope cluster at once. Because of that constant retry bug one can't separately modify ZEO from the client zopes, so you have no choice: you must ensure the whole lot is down at once before restarting everything.

If you're really paranoid, you can also make the filestorage directory ZODB FS files read-only before restart - this snapshots the ZODBs at that point, but interestingly it does not on its own prevent writes as Zope initially writes into a temporary file and then merges the changes later (this is done to aid concurrency, as writing to any object relational database is inherently exclusive of all other operations as a consequence of OODB design). If you set unix permissions to read-only then ZEO simply doesn't merge the changes which may be useful or not to you.

Personally I find it aids my personal sense of security during the subsequent rsync as you can be absolutely guaranteed that the .fs file is coherent. Obviously I don't transfer the .fs.index or .fs.lock files, so the subsequent rsync should be very fast and deliver a coherent transfer.

I hope that this is useful to people - I hadn't found it anywhere else.