Debugging ZODB Bloat

About

Having spent a lot of time tracking down the cause of ZODB bloat in an Archetypes application I thought I'd share my experience in case it was useful to anyone else.

Step 1: Analysis

First step was to analyse the extent of the bloat. The analyze.py script in the Zope bin directory allows you to see what sort of objects are using up space in your database and how many are current or old revisions. If you pack the db then add an object and analyze.py you can see how much bloat is being caused by the number of old object revisions around. add objects and analyze a few times and you can see what sort of objects are the cause of the bloat.

From this I could see that BTrees.IOBtree.IOBucket objects were the culprit. I know they're used in the catalog, but where?

Step 2: Manually look at the contents of the ZODB

analyze.py shows how to open a file storage and inspect the contents. So I iterated to the transaction in question and listed the records:

    fs = FileStorage(path_to_Data_fs, read_only=1)
    fsi = fs.iterator()
    TCOUNT = 2000 # or whatever
    for n in xrange(TCOUNT):
        fsi.next()
    txn = fsi.next()
    records = list(txn)

You can get the size of each record and the oid using [(len(rec.data),rec.oid) for rec in records]

From a Zope debug console you can get the object with ob = app._p_jar[oid]

For some objects though (like IOBuckets) all you get is the c data structure back, which is not all that helpful for our purposes.

Step 3: Reconstruct the object path

I needed to get the path of the object represented. Fortunately ZODB gives you the tools to make a good guess at it. Using the attached utility methods you can build first a map of object references and then try to reconstruct the object path:

    from inspectZodbUtils import buildRefmap, doSearch
    target = rec.oid # assuming rec is the record your interested in
    refmap = buildRefmap(fs)
    path, additionals = doSearch(target,refmap)
    print path

use app._p_jar[oid] from a zope debug console to see what sort of object it is.

inspectZodbUtils.py

Laurence Rowe