PloneLocalFolderNG

PloneLocalFolderNG opens a view to local file system through Plone.

Project Description

This product is not supported anymore - please try Reflecto instead.

Copyright

2005-2006 Mikko Ohtamaa, Red Innovation <mikko@redinnovation.com>

Mike Garnsey (sourceforge id 'mgarnsey')

Andreas Jung, D-72070 Tübingen, Germany

Purpose

PloneLocalFolderNG is a tool which allows local files and folders appear in Plone. Sometimes you might want to store data directly on file system. This is especially the case if you have large amount files or big files which don't co-operate with Zope well. Also, you might have some static old HTML pages and images lying around and you want them to appear in Plone.

Essentially, PloneLocalFolderNG (aka PLFNG) provides a convenient way to make a (large) set of files accessible in a Plone site without the overhead of importing the files as Plone objects. A single PLFNG object instance in a Plone site proxies the entire contents of a specific filesystem path on the Plone site server as pseudo Plone objects. These proxy`d files are **NOT** 1st-class Plone citizens, so they don`t have the rich set of Plone object features like workflow state, individually assignable access permissions, etc.

Zope and plenty of files or large files

Zope database doesn't remove anything automatically. All little changes are kept in database history forever or until site manager goes to press "Pack database" button. If there are a lot of small files or big files, they will quickly increase database size making database look ups slower. File system doesn't have such history keeping and it's designed to serve file content, so it scales better with files.

Other products to solve these problems are

• ExternalStorage
• FileSystemStorage
• ATManagedFile

Features

• basic file / folder actions (add/upload, delete, move)
• file unpacking (just .zip format for now) - metadata for files and folders
• file backup & revisioning support (when files deleted or overwritten)
• ZCatalog text indexing of file contents (via TextIndexNG2)
• filters for hiding files and/or folders
• support for counting and/or logging file accesses
• file space quota limit support
• MD5-based file integrity testing
• support for and/or disregard of OS file permissions settings
• auto-load of a 'default file' on PLFNG base object access
• configurable settings for most PLFNG features
• adding items to Plone navigation tree and sitemap, allowing Plone friendly browsing in PLFNG (new in 0.9.0).

Requirements

• Plone 2.1.2    
• TextIndexNG2 (optional) 
•  and associated system utilites for optional zCatalog text-indexing of  local filesystem contents.
• mxmCounter     
  • mxmCounter is a nice little hit counter, and I put hooks into PLFNG's showFile() to use mxmCounter if its installed.  Note: for this to  work with mxmCounter (v1.1.0), you will need to modify/extend the  mxmCounter class by adding the following method (I have contacted the mxmCounter author to ask that this type of functionality be added to  the baseline of mxmCounter):

			def proxyObject_increase_count(self,url_path):		
			    return increase_count(url_path, self.save_interval)
  

Install

Follow standard Plone add-on product installation procedures.    

Add "PloneLocalFolderNG object to your site".

More information about PLFNG and icons, see adding icons chapter.

Cataloging files

PLGNG doesn't always detect file changes in file system. Namely, navigation tree and sitemaps don't reflect changes. However, you should see changes when you browse to a folder.

Recataloging to check in changes can be tiggered from "Catalog" tab. The current version behaves very badly until files are initially cataloged. Some exceptions are throw, you get 404 pages, etc.

Also, if you have checked "catalog file content" on options page, Plone site text search index will acquire file contents and makes those files searchable.

Note: 0.9.0 navtree code isn't tested with file delete and renames.

Adding icons

Since 0.9.0 there has been functionality to display mounted filenames and icons in site navigation tree and sitemap.

Plone core functionality (2.1.3) doesn't have enough flexibility to display locally mounted file icons. Namely, portal_navtree macro cannot fetch icons from PLFNG file proxy objects.

There is a file called in PloneLocalFolderNG/skins/plfng/xxx_portlet_navtree.pt. Rename this to portlet_navtree.pt so that icon support is actived. This feature is disabled by default to avoid colliding with default portlet_navtree without user interference.

PloneLocalFolderNG displays icons using CSS style declaration for each mimetype. For each file type, a bit of CSS is generated. If there are a plenty of mounted mimetypes there will be a plenty of CSS. See PloneLocalFolderNGTool.py and mime_icons.css.dtml for more details.

Use cases

So why use PLFNG?

Lets say you have lots of files (e.g., thousands of .pdf`s, or maybe more autogenerated javadoc/doxygen/etc files than you really care to actually count) that you need to make available onyour Plone site, but don`t want to crush your Plone server`s underlying Zope object store (ZODB) by creating a Zope object instance for each of these files. With PLFNG, you can simply upload HTMLFormUploadFile* method) through the regular portal web interface a .zip file containing all of these files, and then unpack (*unpackFile* method) its contents. If you have TextIndexNG2 installed, you can then catalog the textual contents of the proxy`d files (including filetypes like .html, .pdf, .ppt., and .doc) for search operations.

As you might imagine, these features are need-driven from a variety of real-world use cases, including (but not limited to) the following:

• I am a Plone site manager that needs to be able to quickly add a bunch of files to my Plone site.  
  • handled by adding PLFNG instance to Plone site & copying the files to Plone site server`s filesystem
• My Plone site users need to be able to upload either a couple or a bunch of files to the Plone site.
  • handled by PLFNG web-based file upload with optional ability to unpack .zip files 'in place' on the Plone site
• My Plone site users need to be able to provide useful metadata (such as comments/notes, descriptive titles, etc) to files on the Plone site.
  • handled by PLFNG's item-level metadata support
• Sometimes previous revisions of files need to be kept in case my Plone site users inadvertantly delete or overwrite them
  •  handled by PLFNG's file backup and file revision metadata mechanisms
• My Plone site users need to be able to perform key word searches on a  bunch of files on the Plone site
  • handled by PLFNG's Zcatalog text indexing mechanism, which can be configured to exclude cataloging of certain file type
• I need to hide the existance of certain files and/or folders based on  name prefix, suffix, and or exact name match from my Plone site users.
      For example, hide any files beginning with '.' (eg '.bashrc') or ending  with '.metadata'.  Also, hide all of the CVS-related folders (eg, folders
      named 'CVS'). 
  • handled by PLFNG's file and folder hiding mechanism
• I am a Plone site manager that needs to know how many times specific files on my Plone site have been accessed.
  • handled by PLFNG's support for external hit counters such as mxmCount)
• I am a Plone site manager that needs to know which users have accessed specific files
  •  handled by PLFNG's support for external logging mechanisms)   

License

Published under the Zope Public License ZPL

References

Version 0.9.0 is live usage at www.opensourceusability.com/site/code.

PloneLocalFolderNG functionality is extended with CodeBrowser product to display source code directly in Plone, inside a text frame.

Cataloging file content

Why catalog?

Cataloging allows ZCatalog  searches of the text contents of the files contained/proxy'd by PLFNG  instance(s). 

Why use TextIndexNG2?

The TextIndexNG2 product provides  a fast and extensible mechanism to quickly extract and catalog the text  contents of a variety of file types such as .html, .pdf, .ppt, .doc, etc.  It provides, out of the box, text extract/conversion for some file types  (eg, html), and relies on external system utilities (eg, wvWare, pdftotext, etc) for other file types. See DEPENDENCIES.txt for more  information.

Instructions

1. using ZMI, traverse to the 'portal_catalog'

2. select the 'Indexes' tab

3. delete the Index 'SearchableText' (type=ZCTextIndex)

4. add a 'TextIndexNG2' with defaults except:
   
    Id = 'SearchableText'
    Left truncation = enabled
    Document Converters = enabled

5. to test, go to one of your PLFNG instances  (preferrably pointing to a folder without  thousands of files in it....but with representative file types in it)

6. click the catalog tab of the PLFNG object

7. click on the 'Catalog Contents' button

8. View results of the PLFNG catalog operation  (on my box, if 'average index rate' is 10^5 KB/sec, it finished too quickly, meaning  that the something is not right with the
   config of TextIndexNG2 --ie, it didnt do  text extraction of the PLFNG files)

9. using ZMI, traverse to the 'portal_catalog' and click the 'Catalog' tab.

10. look for Object Identifier(s) matching the PLFNG files you just cataloged, Note, Object type will be 'FileProxy', and only 20 entries are listed per page, so don't  freak out if you don't see your newly
    cataloged items on the 1st page ;)

11. click on one of your newly cataloged items to bring up its catalog record, and study the 'SearchableText' key in the Index Contents section.  It should contain a list of the text words from the cataloged file

Quality assurance

0.9.0 contains a lot of internal changes. Do not put this thing to production server unless you are able to maintain and support it yourself.

There are some unit tests, but they barely cover the basics of the product usage.

Known problems

• Displaying number of cataloged files don't work
• Reflecting changes to deleted and renamed items don't probably work
• Browsing PLFNG instance through Kupu doesn't work yet
• Catalog tab throws expection when you go there first time after instance creation
• Not yet friendly PLFNG ids á la ATContentTypes