Migrate Custom AT Types with Products.contentmigration

Prerequisites

• Plone 3.0

Step by step

Buildout

A new buildout for your migration will keep everything organized.  Be sure to include both your old and new types. I put the migration in it's own package, you can put it with your type if you like.

[buildout]
extensions = gp.svndevelop
develop-dir = src
svn-develop =
    oldpackage#egg=newpackagename ##If it's not an egg
    newpackage#egg=newpackagename ##If it's not an egg
    migration#egg=migrationame    ##If it's not an egg
    https://svn.plone.org/svn/collective/Products.contentmigration/trunk/#egg=Products.contentmigration

develop =
    src/oldpackage ##If it's not an egg
    src/newpackage ##If it's not an egg
    src/migration  ##If it's not an egg
    src/Products.contentmigration

eggs=
    oldtype
    newtype
    migration
    Products.contentmigration

zcml=
    oldtype
    newtype
    migration
    Products.contentmigration

Run buildout to get all of your packages.

GenericSetup

Use GenericSetup to register your upgrade steps.

<configure
    xmlns="http://namespaces.zope.org/zope"
    xmlns:gs="http://namespaces.zope.org/genericsetup">

    <gs:registerProfile
        name="migrationname"
        title="Title of the migration that will appear in the ZMI"
        for="Products.CMFPlone.interfaces.IMigratingPloneSiteRoot"
        provides="Products.GenericSetup.interfaces.EXTENSION" />

    <gs:upgradeSteps
        source="oldversionnumber"
        destination="newversionnumber"
        profile="GenericSetup profile to use - version:default or something">

        <gs:upgradeStep
            title="Title of the upgrade step"
            description="Description of the upgrade step"
            handler=".migrate.migratetype" />
        <!-- The above handler should be the path and function name of the migration -->

    </gs:upgradeSteps>

</configure>

The Test

This process is easier for me as a test-as you go, than as a test afterwards.  Tests are very important in migrations, you want to make sure that the objects are doing what you expect them to do.

I always start with a base test file, and subclass from that.  In my tests directory, there is a file called base.py.

from Products.Five import zcml
from Products.Five import fiveconfigure

from Products.PloneTestCase import PloneTestCase as ptc
from Products.PloneTestCase.layer import onsetup

from Testing import ZopeTestCase as ztc

import oldpackage
import newpackage

PRODUCTS = ['oldpackage',
            'newpackage']

#Set up the Plone site used for the test fixture.
@onsetup
def setup_migration():
    fiveconfigure.debug_mode = True

    #Load the zcml for the packages you are working with.
    #If you are not able to create an instance of your type, you may have skipped this step.
    zcml.load_config('configure.zcml', oldpackage)
    zcml.load_config('configure.zcml', newpackage)
    zcml.load_site()
    fiveconfigure.debug_mode = False

    ztc.installPackage('oldpackage')
    ztc.installPackage('newpackage')

#Run the setup function
setup_migration()
ptc.setupPloneSite(products=PRODUCTS)

class MigrationTestCase(ptc.PloneTestCase):
    """Base class for your migration tests"""
    pass

 Ok, now time to get started with our test.

In your tests folder, make a test file for your migration.  I'll call this test_migration.py.

import transaction
from mymigration.migrations import migrate #We'll add this in a minute
from mymigration.tests.base import MigrationTestCase

class TestThisMigration(MigrationTestCase):
    """Test the migration from oldtype to newtype
    """
    def afterSetUp(self):
        #Create an object to migrate
        self.folder.invokeFactory('oldtype', 'test_oldtype')

        #A direct assignment doesn't work here, you have to use the setter or
        #the value will come out blank.
        #The field migrator uses obj.getRawFieldName()
        self.folder.test_oldtype.setField1('Field 1 value')

        #Need this to avoid copy errors
        transaction.savepoint(optimistic=True)


    def testItemMigrated(self):
        #portal_type and meta_type should be of type newtype
        oldItem = self.folder.test_oldtype
        
        ###Before the migration
        self.assertEqual(oldItem.portal_type, "The old package's portal type",
                         "oldItem.portal_type was %s, but should have been blah" %
                         oldItem.portal_type)
        self.assertEqual(oldItem.meta_type, "The old package's meta type",
                         "oldItem.meta_type was %s, but should have been blah" %
                         oldItem.meta_type)

        ###Run the migration
        old = migrate(self.portal)

        ###After the migration
        #Recopy the item, the old object held on to that old type info
        newItem = self.folder.test_oldtype

        #The item should now have the new types
        self.assertEqual(newItem.portal_type, "The new package's portal type",
                         "newItem.portal_type was %s, but should have been blah" %
                         newItem.portal_type)
        self.assertEqual(newItem.meta_type, "The new package's meta type",
                         "newItem.meta_type was %s, but should have been blah" %
                         newItem.meta_type)

        #Make sure the field values transferred to the new fields
        self.assertEqual(newItem.newfield1, 'Field 1 value',
                         'field1 field did not transfer')


def test_suite():
    from unittest import TestSuite
    from unittest import makeSuite

    suite = TestSuite()
    suite.addTest(makeSuite(TestThisMigration)) #Class from above

    return suite

Running the test now gets an import failure, because we haven't created the migration yet.  Go ahead and run it if you want to double check your syntax.

The Migration

Now, for the migration itself.  The test file imported the migration from a file called migrations.py.  In this file goes a migrate function.

from StringIO import StringIO #That's a big o

from Products.contentmigration.walker import CustomQueryWalker
from Products.contentmigration.archetypes import ATItemMigrator

class OldItemMigrator(object, ATItemMigrator):
    """Migrate the old item type to the new item type
    """
    walker = CustomQueryWalker
    src_meta_type = "old_meta_type"
    src_portal_type = "old_portal_type"
    dst_meta_type = "new_meta_type"
    dst_portal_type = "new_portal_type"

    def __init__(self, *args, **kwargs):
        self.old = args[0] #The original object
        self.orig_id = self.old.id #The original object id
        self.old_id = 'old_%s' % self.orig_id #Change the old id to this after copying
        self.new_id = self.old.id #Make the new object id this
        self.parent = self.old.getParentNode() #Grap the object's parent

        self.fields_map = dict(oldFieldName='newFieldName',
                               field1='newField1')

        super(OldItemMigrator, self).__init__(*args, **kwargs)

def migrate(portal):
    """The migrate function that runs everything.  This is what gets imported
    for your tests.
    """
    out = StringIO()  #Big o
    migrators = (OldItemMigrator,)

    #Run the migrations
    for migrator in migrators:
        print >>out, "--Migrating %ss \n\n" % migrator.src_meta_type
        walker = migrator.walker(portal, migrator)
        walker.go(out=out)
        print >>out, walker.getOutput()

    return out.getvalue()

Save the file, and run your test.

./bin/instance test -m mymigration

Running the Migration

Take down your site and back up the database before you run this!  

Make sure all of your types are installed in the database, new and old.

In the ZMI for your site, go to portal_migration and make sure your site is up to date.  If it isn't, run the migration to get it there.

Back at the site root in the ZMI, go to portal_setup.  Go to the Upgrades tab at the top of the screen.  In the picklist at the left, you will see the profile name that you put above in the configure.zcml file.  Choose it and click Choose Profile.  You will see the list of your upgrade steps.  Check the steps that you want to run, and click the Upgrade button.

Further information

For older versions of Plone, you can use the CMFItemMigrator for migrations.  These instructions are based on Martin Aspelli's instructions on that kind of migration, found here.