Zero-in on the Code

The zeoserver.log file

After trying (unsuccessfully) to debug the ZEO server using a pdb.set_trace(), I discovered the zeoserver.log file.  (I learn something new every day.)  Here's the relevant snippet:

(18529/127.0.0.1:51659) pack(time=1231187683.527384) started...
(18529) Error raised in delayed method
Traceback (most recent call last):
  File "/var/Plone/eggs/ZODB3-3.8.1b7-py2.4-linux-x86_64.egg/ZEO/StorageServer.py", line 1247, in run
    result = self._method(*self._args)
  File "/var/Plone/eggs/ZODB3-3.8.1b7-py2.4-linux-x86_64.egg/ZEO/StorageServer.py", line 416, in _pack_impl
    self.storage.pack(time, referencesf)
  File "/var/Plone/eggs/ZODB3-3.8.1b7-py2.4-linux-x86_64.egg/ZODB/blob.py", line 555, in pack
    self._packUndoing(packtime, referencesf)
  File "/var/Plone/eggs/ZODB3-3.8.1b7-py2.4-linux-x86_64.egg/ZODB/blob.py", line 497, in _packUndoing
    for oid, oid_path in self.fshelper.listOIDs():
  File "/var/Plone/eggs/ZODB3-3.8.1b7-py2.4-linux-x86_64.egg/ZODB/blob.py", line 391, in listOIDs
    oid = utils.repr_to_oid(candidate)
  File "/var/Plone/eggs/ZODB3-3.8.1b7-py2.4-linux-x86_64.egg/ZODB/utils.py", line 142, in repr_to_oid
    as_bin = unhexlify(repr)
TypeError: Odd-length string

Obviously, there's some object (repr) that's causing unhexlify() to bomb in line 142 of /var/Plone/eggs/ZODB3-3.8.1b7-py2.4-linux-x86_64.egg/ZODB/utils.py.  But if you look a few lines prior to that, there are some promising references to OIDs.  (OID means Object ID.)  Perhaps we can find out which object is the problem, after all!

Use a Try-Except clause

Here we've again got to be careful.  We don't want to drop into a pdb prompt everytime this is called--it's got to be thousands of times.  Rather, we just want to find out what's up the one value that's causing an error.  In this case, let's wrap just that one line with try and except:

def repr_to_oid(repr):
    if repr.startswith("0x"):
        repr = repr[2:]
    try:
        as_bin = unhexlify(repr)
    except TypeError:
        import pdb; pdb.set_trace()
    as_bin = "\x00"*(8-len(as_bin)) + as_bin
    return as_bin

Honest Mistake #3: I made this change before running buildout (in the next step).  I'll explain more later...

Why did grep not find this error?

When we ran grep, back in the beginning, looking for 'Odd-length string', it took a long time, so I halted the process.  That was "Honest Mistake #2."  Had I let it run, it would have found:

var/log/zeoserver.log:TypeError: Odd-length string