#191: Unify folder implementations

Contents
  1. Motivation
  2. Proposal
  3. Implementation
  4. Deliverables
  5. Risks
  6. Progress log
  7. Participants
by Martin Aspeli last modified Jan 21, 2010 07:27 AM

We currently have "Folder" and "Large Plone Folder" implementations. There should be only one.

Proposed by
Martin Aspeli
Seconded by
Andreas Zeidler
Proposal type
Architecture
Assigned to release
Repository branch
plone.folder
State
draft

Motivation

Shipping with two folder types is unnecessary for several reasons:

  • It forces the user to make an a-priori choice about the number of objects they plan to put into a folder
  • We ship with the "Large" type disabled by default to avoid UI confusion
  • We don't have a proper search-based UI for large folders anyway

Also the standard Folder type stores attributes, and has a single list _objects tuple which keeps the list of objects and order. This is prone to ConflictErrors and is slower. In simple benchmarks, a BTree-based folder performs orders of magnitude faster than a basic folder.

Proposal

Have a single folder implementation.

  • The internal storage is BTrees
  • It still supports ordering, by storing a separate sort order list/tree
  • It has at least two views - one search-based for "large" folders, one batch-based for "small" folders. This is either just a "display" menu choice, or a choice in the object's schema. Explicit sorting may be turned off for "large" folders.

Implementation

The package plone.folder in the Plone SVN provides a base implementation of a folder base class, which is not Archetypes specific, based on BTreeFolder2, but adding ordering. The exact ordering implementation is left up to an adapter, with a default providing explicit ordering. This allows other implementations, such as auto-sorting based on some key.

The diagram below shows the folderish base- and mix-in classes used in OFS, CMF, Archetypes and Plone. Count 'em:

Folder inheritance

An initial investigation suggests that this could be made a base class for BaseBTreeFolder in Archetypes. This is backwards compatible, since the new type simply replaces the rather simple CMFBTreeFolder.

We'd then need to make this folder type the default, and ensure that there's adequate migration (either in that existing instances continue to work but new folders have this feature, maybe with an optional "migrate" action, or through some mass migration).

The Container type in plone.app.content does not use BTrees, nor does it support ordering. Rather than changing this class and providing migration, we could add a new type, e.g. called OrderedContainer or just Folder (to signify closer resemblance to the standard folder behaviour) that mixes in the new class instead of PortalFolder.

Deliverables

  • Improved Products.ATContentTypes.content.ATBTreeFolder that uses BTrees and supports ordering.
  • New class plone.app.content.container.Folder (?) that uses BTrees and supports ordering.
  • An out-of-the-box Folder type that is BTree-based and supports large content sets as well as ordering.
  • The ability to switch to a folder view/behaviour that's optimised for "large" content sets.
  • Migration/graceful fallback for all existing sites.

Risks

  • There may be non-trivial migration involved if existing instances should be changed.
  • This vision makes the existing BaseBTreeFolder in Archetypes orderable (though in some UI configurations you may not see the ordering, to prevent the user invoking slow operations), which may be unexpected

Progress log

Complete:

  • Create a base folder implementation - OrderedBTreeFolderBase
  • Hook this into BaseBTreeFolder in Archetypes
  • Fix CMFPlone so that it detects the order support based on a Zope 3 interface rather than a Zope 2 one
  • Test that order support works on the new folder
  • Switch FTIs and portal_type's around so that the default "Folder" portal_type is a btree-based folder
  • Enable next/previous navigation for this folder type
  • Enable photo-album support for this folder type
  • Enable archiving support for this folder type
  • Enable ordering policy to be specified by adapter

To-do - ATContentTypes:

  • Fix remaining ATCT and CMFPlone test failures (mainly around WebDAV)
  • Write migrations for FTI and portal_type changes
  • Provide optional migration to change types in-place

To-do - plone.app.content:

  • Create new Folder base class, using the new mixin, with tests

Benchmark results

The following two graphs show some simple benchmarks using the various types of folders (regular = ATFolder, large = ATBTreeFolder, ordered = ATBTreeFolder w/ order support) with 50 and 500 items, respectively. The tests conducted where creating content, retrieving a batch of items from the folder, getting all items from the folder as well as random accessing the items. The benchmark tests used can be found in the tests/ folder.

Benchmarks with 50 itemsBenchmarks with 500 items

A more realistic scenario was provided by some benchmarks based on JMeter (available in jmeter/):

JMeter Benchmark tests

The graphs show the time it took to create a folder with 500 news items both using a standard folder and the new btree-based folder with order support. Also, and that's probably the most interesting figure so far, the time for repeatedly browsing the "folder contents" view is decreased to slightly less than 75% by using the btree folder.

Another interesting thing to note is that the "content creation" tests show that btree-based folders are faster than the currently used implementation (from ATFolder) even for very small numbers of contained objects. The tests still run faster when adding 500, 50, 10, 5, 2 or even just a single object to the folder.

Participants

Martin Aspeli (IRC nickname: <optilude>)
Andreas Zeidler (IRC nickname: <witsch>)
Martijn Pieters (IRC nickname: <MJ>)
Alec Mitchell (IRC nickname: <alecm>)