Large file handling

by Nate Aune — last modified Jan 23, 2006 08:15 PM
As we all know, Zope doesn't handle large files very efficiently. Since multimedia often involves large files, we need to try to find some clever ways to handle them in Plone.

Archetypes storage layers

There are 3 external storage layers for Archetypes:

The merits of these various storage layers have been discussed, but there has not been a push to consolidate these products into a definitive file system storage layer for Archetypes. Hopefully we can make some progress on this during the sprint.

The PloneMultimedia project team is considering switching ATMediaFile's storage layer to use FileSystemStorage (issue #98) pending further investigation.

BLOB support

Chris McDonough has developed blob, which provides an alternative to storing large binary chunks of arbitrary data in ZODB, allowing this kind of data to live instead in files on a filesystem. Unlike a "roll your own" solution, it provides transactional integrity while doing so. It also provides a simple UNIX-like API for dealing with file objects. Note: This blob support may have already been implemented in recent versions of Zope.


Caching of files on external servers

Archive.org, CoralCache and FreebandwidthProject all offer to cache large files to reduce strain and bandwidth costs on your server. There is even a desktop program called CCPublisher which will let you choose a Creative Commons license for your media file and assist in uploading it to archive.org.

It is relatively simple to rewrite the URL to retrieve the large media file off of these servers instead of the Plone site. This could perhaps be implemented using a portal tool with a configlet to configure which service to use. The templates would then ask the portal tool to return the URL rewritten.

Salim Fadhley's Plodcasting product includes a modified portal_url tool that generates Coral-Cache compatible URLs. This feature makes it possible to offload much of the bandwidth requirments onto a global network of free cache servers, and as a result achieve a potential saving in hosting costs.

Comments (0)