Importing your product into the Collective

« Return to page index

There are two strategies for introducing a new product in the Plone Collective when you already have it in your own repository: a simple svn export or a dump file that is loaded by the plone.org admins. This tutorial explains the differences and shows how to do both.

Introduction

Description of the problem this tutorial handles

You are a developer at Spacely Sprockets. You have developed a Plone product called plockets. Your company is using it internally for its sprocket inventory. You are still sane, so apparently you have used a code repository. We assume that it is subversion. You have told some others about your product. You even gave them read access to your repository. Some are now making patches and are asking for write access. You decide that you do not want to deal with that, so you want to move your code to the Plone Collective. Resistance is after all still futile. (This tutorial assumes that you have write access to the Collective, if not, read the instructions for how to get access to the Collective) The Plone Collective is a unique, open code repository that contains hundreds of add-on Products for Plone from over 250 developers. Some of the coolest, most exciting stuff in the world of Plone is happening the Collective, and it's the lab for most of what eventually makes it into the Plone Core. Access to the Plone Collective is (almost) wide-open.

Simple export

The simplest way to import your product is by making a clean svn export.

Your company repository lives on a server and your product can be found at http://spacely.space/svn/plockets. One option is to simply make a subversion export of your product, and importing it in the collective:

svn export --ignore-externals http://spacely.space/svn/plockets plockets
svn import plockets https://svn.plone.org/svn/collective/plockets

One major downside when using a simple export is that you lose all subversion history information. If you are the only developer who worked on plockets or there are not so many files and revisions, this may not matter. But the history can give important information about why a change was made, especially in the revision log.

The good thing about an export is of course that it is by far the simplest method. Two simple commands and you are ready. And this is something that you can do without help from the administrators at plone.org.

If the history of your product is important, then you have the option of making a dump of the part of the subversion repository that has your plockets product in it.

Code dumping

How to make a dumpfile of your complete repository

First you make a dump file of the repository. You could play around on your company server, but if something goes wrong, Mr Spacely will be angry with you and you do not want that. So to be on the safe side, you replicate the server repository on your own computer.

You might be able to use svk for this, if you are used to it. I will not cover that here. If the name does not ring a bell, never mind.

Instead I assume that you actually have access to that server and are allowed to use the svnadmin command. So you login to that server and (quietly if you want to) dump the complete repository to a dump file. That would look something like this:

svnadmin dump --quiet /var/lib/svn/spacely > spacely.dump

Then you copy that file to your own computer and you can now safely play with it without having to say sorry to Mr Spacely when you make a mistake that kills the complete repository. Just a precaution.

Filtering the dump file

Getting only your project in a dumpfile

You now have a spacely.dump file on your computer. But that contains the complete repository. Now you want to create a new dump file that only contains the plockets product. You do that with the svndumpfilter command:

cat spacely.dump | svndumpfilter --drop-empty-revs --renumber-revs --quiet include plockets > plockets.dump

This does the following:

you pipe the complete dump file to svndumpfilter,
you only include revisions that contain changes to plockets,
you drop (ignore) any revisions that have nothing to do with plockets,
you renumber the revisions (this is recommended when you use the previous step),
you do this quietly, though when you do this the first time you probably want to see what happens, so remove the --quiet part,
you put the result in a new dump file plockets.dump.

Several things could go wrong here, but we assume for now that this command succeeds without warnings or errors. If you run into problems, then consult the section called Problems with svndumpfilter.

Testing your dump file

How to make sure that your dumpfile is correct

During testing it may be handy to have the complete Spacely repository available on your own machine instead of just via the dump file. If you want to recreate the complete Spacely repository, you can do that like this:

svnadmin create /var/lib/svn/spacely
svnadmin load --quiet /var/lib/svn/spacely < spacely.dump

You can use these same commands for testing your new plockets dump file. Create a temporary subversion repository on your machine and load the sprockets product:

svnadmin create /var/lib/svn/temp
svnadmin load --quiet /var/lib/svn/temp < spacely.dump

This should just work. But you probably want to be certain that the resulting repository has exactly the same information as the original sprockets part of the spacely repository.

One simple way is to create svn exports of both repositories and compare them. If you have svn:externals in your repository, then it is best to ignore those. Create and compare two exports like this:

# Make an export from the Spacely repository:
svn export --ignore-externals --quiet http://spacely.space/svn/plockets plockets.ori
# Alternatively, make an export from your local Spacely repository:
svn export --ignore-externals --quiet file://var/lib/svn/spacely/plockets plockets.ori
# Make an export from your local temporary repository:
svn export --ignore-externals --quiet file://var/lib/svn/temp/plockets plockets.new
# See the differences between those two exports:
diff -r plockets.ori plockets.new

This should not give a lot of lines of output. You may see some output like this:

3c3
< # $Id: config.py 551 2006-09-19 13:04:49Z sashav $
---
> # $Id: config.py 507 2006-09-19 13:04:49Z sashav $

That is okay. That is just a part of a file that is automatically adjusted by subversion based on the version number in the repository. If you see other differences then that is an indication that something went wrong. You need to investigate then.

Loading in the collective

Get your dumpfile noticed by the plone.org admins

You can now send the plockets dumpfile to the administrators at plone.org. The best way to do that is to file a ticket in the plone.org tracker. See the example for Importing RealEstateBroker into the collective.

You can upload your dump file to the ticket by using the Attach File button. Note that this button only becomes visible after you submit your ticket.

Problems with svndumpfilter

When you run into problems when using svndumpfilter don't panic.

Unsupported dumpfile version

Some versions of svnadmin and svndumpfilter do not work nicely together. When you pipe a dump file created by svn through svndumpfilter you will then see this error:

svn: Unsupported dumpfile version: 3

This means that svndumpfilter expects a dumpfile of version 2 and does not know how to handle version 3. Then it may help to first load the server dump file in your local repository like mentioned above. If you do not get it to work, try upgrading to a newer version of your subversion packets, or consult the internet. For example, svndumpfilter vs svnadmin and version inconsistency.

Invalid copy source path

Another problem can occur when you include or exclude a path and one revision has changes on files that are both inside and outside the path that you want to keep. For instance, I tried excluding the integration directory when making a dump for the eXtremeManagement product as it did not make sense to include that in the dump file. Then I saw this:

Revision 369 committed as 355.
Revision 370 skipped.
svndumpfilter: Invalid copy source path '/eXtremeManagement/integration'

This means that the following is the case:

revision 369 is fine;
revision 370 is skipped because the excluded path was referenced there;
revision 371 causes problems, as it changes something both inside and outside the excluded path: probably you copied something from an excluded path to an included path.

In my case, this was the problem:

$ svn log -r 371 file:///var/lib/svn/temp/ -v
------------------------------------------------------------------------
r371 | maurits | 2006-02-23 15:21:18 +0100 (do, 23 feb 2006) | 1 line
Changed paths:
   D /eXtremeManagement/integration
   A /eXtremeManagement/temp (from /eXtremeManagement/integration:370)

Moved integration revision 4068 to temp.

So when you use svndumpfilter and run into this problem, you need to manually work around that. I wrote a script that does that. It also does some checking. Perhaps it is of use to others. You cannot use it off the shelf. You need to adapt it to your own needs. If you use it unthinkingly on your server, Mr Spacely could get very upset.