#191: Unify folder implementations
We currently have "Folder" and "Large Plone Folder" implementations. There should be only one.
- Proposed by
- Martin Aspeli
- Seconded by
- Andreas Zeidler
- Proposal type
- Assigned to release
- Repository branch
Shipping with two folder types is unnecessary for several reasons:
- It forces the user to make an a-priori choice about the number of objects they plan to put into a folder
- We ship with the "Large" type disabled by default to avoid UI confusion
- We don't have a proper search-based UI for large folders anyway
Also the standard Folder type stores attributes, and has a single list _objects tuple which keeps the list of objects and order. This is prone to ConflictErrors and is slower. In simple benchmarks, a BTree-based folder performs orders of magnitude faster than a basic folder.
Have a single folder implementation.
- The internal storage is BTrees
- It still supports ordering, by storing a separate sort order list/tree
- It has at least two views - one search-based for "large" folders, one batch-based for "small" folders. This is either just a "display" menu choice, or a choice in the object's schema. Explicit sorting may be turned off for "large" folders.
The package plone.folder in the Plone SVN provides a base implementation of a folder base class, which is not Archetypes specific, based on BTreeFolder2, but adding ordering. The exact ordering implementation is left up to an adapter, with a default providing explicit ordering. This allows other implementations, such as auto-sorting based on some key.
The diagram below shows the folderish base- and mix-in classes used in OFS, CMF, Archetypes and Plone. Count 'em:
An initial investigation suggests that this could be made a base class for BaseBTreeFolder in Archetypes. This is backwards compatible, since the new type simply replaces the rather simple CMFBTreeFolder.
We'd then need to make this folder type the default, and ensure that there's adequate migration (either in that existing instances continue to work but new folders have this feature, maybe with an optional "migrate" action, or through some mass migration).
The Container type in plone.app.content does not use BTrees, nor does it support ordering. Rather than changing this class and providing migration, we could add a new type, e.g. called OrderedContainer or just Folder (to signify closer resemblance to the standard folder behaviour) that mixes in the new class instead of PortalFolder.
- Improved Products.ATContentTypes.content.ATBTreeFolder that uses BTrees and supports ordering.
- New class plone.app.content.container.Folder (?) that uses BTrees and supports ordering.
- An out-of-the-box Folder type that is BTree-based and supports large content sets as well as ordering.
- The ability to switch to a folder view/behaviour that's optimised for "large" content sets.
- Migration/graceful fallback for all existing sites.
- There may be non-trivial migration involved if existing instances should be changed.
- This vision makes the existing BaseBTreeFolder in Archetypes orderable (though in some UI configurations you may not see the ordering, to prevent the user invoking slow operations), which may be unexpected
- Create a base folder implementation - OrderedBTreeFolderBase
- Hook this into BaseBTreeFolder in Archetypes
- Fix CMFPlone so that it detects the order support based on a Zope 3 interface rather than a Zope 2 one
- Test that order support works on the new folder
- Switch FTIs and portal_type's around so that the default "Folder" portal_type is a btree-based folder
- Enable next/previous navigation for this folder type
- Enable photo-album support for this folder type
- Enable archiving support for this folder type
- Enable ordering policy to be specified by adapter
To-do - ATContentTypes:
- Fix remaining ATCT and CMFPlone test failures (mainly around WebDAV)
- Write migrations for FTI and portal_type changes
- Provide optional migration to change types in-place
To-do - plone.app.content:
- Create new Folder base class, using the new mixin, with tests
The following two graphs show some simple benchmarks using the various types of folders (regular = ATFolder, large = ATBTreeFolder, ordered = ATBTreeFolder w/ order support) with 50 and 500 items, respectively. The tests conducted where creating content, retrieving a batch of items from the folder, getting all items from the folder as well as random accessing the items. The benchmark tests used can be found in the tests/ folder.
The graphs show the time it took to create a folder with 500 news items both using a standard folder and the new btree-based folder with order support. Also, and that's probably the most interesting figure so far, the time for repeatedly browsing the "folder contents" view is decreased to slightly less than 75% by using the btree folder.
Another interesting thing to note is that the "content creation" tests show that btree-based folders are faster than the currently used implementation (from ATFolder) even for very small numbers of contained objects. The tests still run faster when adding 500, 50, 10, 5, 2 or even just a single object to the folder.
Andreas Zeidler (IRC nickname: <witsch>)
Martijn Pieters (IRC nickname: <MJ>)
Alec Mitchell (IRC nickname: <alecm>)