Adding new fields to Smart Folders search

If you are creating new content types you might want to add support for Plone's advanced search capabilities. This article describes how to achieve it.

Introduction

Smart folders is the end use term for Plone's advanced search capabilities delivered by ATContentTypes product. With smart folders, one can define different fields(also called indicies in the context of search) and criterias for searches. Internally Plone refers to Smart Folders as "topics".
Smart Folders are folderish objects which are actually searches executed every time users opens them. For example, if we are building human resource management system on Plone, we might want to create a Smart Folder for employee availability. In this Smart Folder, all employees are searched and three table columns are displayed: name, project and when the project ends (employee becomes available). Also, the employee availability Smart Folder could be sorted by availability so that employees who are free in the near future are displayed first.

screenshot.png

Steps

  1.  Create a custom ATContentTypes based type
  2. Add your custom fields to portal_catalog so that portal_catalog can find and index them
  3. Add binding between portal_catalog indices and Smart Folder indicies
  4. Create a Smart Folder to perform your searches


Create a custom ATContentTypes based type

Extending ATContentTypes tutorial explains how to create new content types for Plone. Unfortunately, currently this still requires knowledge of Python coding and there is not yet WYSIWYG editor for Plone content types.

Adding fields to portal_catalog

Portal_catalog is the backend tool responsible for making your site searchable. When user performs a search, portal_catalog checks from it's indices which objects match the search criteria.

Note that portal catalog catalogs items automatically when they are created or modified. If you have already existing content items at your site and you want to make them searchable, remember to manually reindex portal_catalog after creating new indicies. To do this, go to portal_catalog tool's advanced tab and hit "Update Catalog" button. If you have succesfully defined new portal_catalog indices and there exist content for them, you should see the number of indexed objects be above zero at the portal_catalog indexes tab.

Adding new search criteria

Portal_catalog is accessiable from Zope management interface. Go to portal_catalog tool, indexes tab. Add new index. The id of the new index is ATContentTypes accessor method for your field. For example, if your field is called "project" then the accessor method would be "getProject".

 screenshot.png

Different index types

  • ZCTextIndex: Text Indexes break text up into individual words, and are often referred to as full-text indexes. Text indexes sort results by score, meaning they return hits in order from the most relevant to the least relevant.
  • Keyword Indexes index a sequence of objects that act as 'keywords' for an object. A Keyword Index will return any objects that have one or more keywords specified in a search query.
  • A DateIndexindexes DateTime attributes.
  • A DateRangeIndex takes the name of two input attributes; one containing the start date of the range, the second the end of the range. This index is filled with range information based on those two markers. You can then search for objects for those where a given date falls within the range.
  • Field Indexes treat the value of an objects attributes atomically, and can be used, for example, to track only a certain subset of object values, such as 'meta_type'.
  • TextIndex is deprecated. Don't use it.

Adding search table columns

 Smart Folders has an option to display results as  a table. Columns available for this table are metadata indexes from portal_catalog. To add a custom column, the column must have a metadata index first. Add new metadata indexes from portal_catalog Metadata tab. Again, the index name must be the accessor of a field. If your field name is "project" then the accessor is "getProject".
 screenshot3.png

Adding Smart Folders bindings

Creating new portal_catalog indices is not enough to make your content Smart Folders searchable. portal_catalog is just an layer under Smart Folders mechanism to make searches available for Smart Folders. screenshot2.png
Smart Folders has a configuration view at Plone. Log-in as a adminstrator and go to your preferences page. There is a Smart Folder Settings page. Smart Folder Indexes tab controls search and sort criterias and Smart Folder Metadata contains available table columns for searches.

By default, Smart Folder Settings shows only indexes currently bind to Smart Folder fields. To show indexes you created, click "All Fields" link and check enable checkbox for those indexes. You must also add an user friendly name for your fields.

Creating a new Smart Folder

This should be pretty straightforward. Just go to the site location you wish and hit "add smart folder" for new types drop down menu. Your custom search criteria and metadata columns are available for Smart Folders if you followed instructions above.

screenshot1.png

Doing it prorammatically

To save you all going through hoops and gotchas how to get all it working in scripting, for example in quick install script, I'll have this complete example here:

    def createEmployeeSmartSearchFolder(self):
        """ Create smart search capabilities
       
        Things done here:
            - Creates new indexes (project, availability)
            - Binds indexes to smart folders
            - Creates a smart folder object for employees searches
       
        """
       
        # dummy class used with manage_addIndex
        class args:
            def __init__(self, **kw):
                self.__dict__.update(kw)
            def keys(self):
                return self.__dict__.keys()
           
        # Tune up catalog tool
       
        catalog_tool = self.portal.portal_catalog
       
# Clean up previous install if there is some old stuff
        try:
            catalog_tool._removeIndex("getProject")
        except:
            pass
   
        try:
            catalog_tool._removeIndex("getAvailability")
        except:
            pass   
   
        try:
            catalog_tool.delColumn("getProject")
        except:
            pass       
       
        try:
            catalog_tool.delColumn("getAvailability")
        except:
            pass       
       
        extra = args(doc_attr='SearchableText',
                             lexicon_id='plone_lexicon',
                             index_type='Okapi BM25 Rank')   
       
        catalog_tool.manage_addIndex("getProject", "ZCTextIndex", extra)
        catalog_tool.manage_addIndex("getAvailability", "DateIndex",extra)
        catalog_tool.manage_addColumn("getProject")
        catalog_tool.manage_addColumn("getAvailability")
           
        # Update Smart Folder settings          
        smart_folder_tool = self.portal.portal_atct
       
        # Remove existing indexes if there are some
        try:
            smart_folder_tool.removeIndex("getProject")
        except:
            pass
       
        try:
            smart_folder_tool.removeMetadata("getProject")
        except:
            pass

        try:
            smart_folder_tool.removeIndex("getAvailability")
        except:
            pass

        try:
            smart_folder_tool.removeMetadata("getAvailability")
        except:
            pass

       
        smart_folder_tool.addIndex("getProject", "Employee's project", "The current project employee is working on", enabled=True)
        smart_folder_tool.addIndex("getAvailability", "Employee's availability", "Date when employee is available next time", enabled=True)
        smart_folder_tool.addMetadata("getProject", "Employee's project", "The current project employee is working on", enabled=True)
        smart_folder_tool.addMetadata("getAvailability", "Employee's availability", "Date when employee is available next time", enabled=True)
       
        # This is ugly, but we cannot override friendly names per
        # smart folder instance. Here we use Title as Employee name
        # and to avoid confusion in table header we need to rename all
        # Title friendly names to Name
        smart_folder_tool.updateMetadata("Title", friendlyName="Name")
               

        # Create Employee Availbilty smart folder                    
        try:
            # Remove existing instance if there is one
            portal.manage_delObjects(["employee_availability"])
        except:
            pass               

        # Create smart folder to portal root
        self.portal.invokeFactory("Topic", id="employee_availability", title="Employee Availability")
        employee_availability = self.portal.employee_availability
       
        # Sort results based on availability
        employee_availability.setSortCriterion("getAvailability", reversed=False)
        employee_availability.setDescription("List employees, sorted by when they are free")
       
        # Filter results to employees
        type_criterion = employee_availability.addCriterion('Type', 'ATPortalTypeCriterion' )
        type_criterion.setValue("Employee")
           
       
        # Display as table
        employee_availability.setCustomView("True")       
        employee_availability.setCustomViewFields(("Title", "Description", "getAvailability"))
   
        self.out.write("Created employee search smart folder")

Related source files

If you want to do things programmatically (i.e. when your product quick install script is run), you should check following files as a starting point for creating the script

  • ATContentTypes/tool/topic.py
  • CMFCore/CatalogTool.py




thanks for documenting this

Posted by Darcy Clark at Feb 20, 2006 01:45 AM
very useful - you've saved me a *lot* of time and frustration :)

Automatic installation with ArchGenXML: AppInstall.py

Posted by Chris Shenton at Aug 31, 2006 05:33 AM
Very helpful for manually creating this, but I found it difficult to understant how to use the code during installation of an ArchGenXML product. perhaps these hints will help others.

AGX creates an Extensions/Install.py which tries to call AppInstall.py file with methods install(self) and uninstall(self). In those, you can access self.portal_catalog, self.portal_atct, self.invokeFactory() instead of the self.portal.* calls in the example code. Separate the example removal and creation code into the uninstall() and install() methods. Return status as a string (see StringIO).

You can save yourself a lot of typing by creating a structure with the index name, index type, label, and description of all your fields, e.g.,

INDICES = [("getFoo", "FieldIndex", "Foo", "Field Foo"), ('getBar', 'DateIndex', ...), ...]

then looping over them for your create/destroy needs.


Slick stuff; nice how it all fits together.

Plone Installation

Posted by Mikko Ohtamaa at Aug 31, 2006 10:19 AM
Good feedback,

Furthermore, I recommend using Plone Installation product which helps creating quick installer scripts like this one. The example above is more like "what happens under the hood".

More, please!

Posted by Mike Combs at Nov 22, 2006 03:19 AM
The manual walk-through was a HUGE help to me. Huge.

I'd like to apply the automated technique but it's still a little confusing. Would you mind posting the AppInstall.py broken out into the install and uninstall methods, and with the INDICES structure mentioned above? I'd like to have something I can paste with a little find & replace to get this running. I promise a lot of newcomers will sing your praises. :)

You want some more?

Posted by Mikko Ohtamaa at Nov 22, 2006 12:14 PM
Hi Mike,

Please feel free to contribute more yourself. Plone documentation area is open for everyone :)

Also, check out Custom Search product. It has a lot of code dealing with indexes and metadata.

If only there were a tip jar...

Posted by Mike Combs at Nov 22, 2006 03:10 PM
I knew my request would sound too selfish, but couldn't think of a better way to phrase it. Sorry. I *wish* I could contribute. I've suggested that the docs and products have PayPal tip jars for those of us who can't contribute Python. Much of the free world has been built on trades of beer money for skills.

I think this code is the "missing link" between making Archetypes completely usable TTW, so people like me who haven't learned the ins and outs of Python, Zope and Plone can make and display new data types.

The ideal here would be if there were some way to indicate in ArgoUML which fields you want to be indexes, and which you want to be metadata, and this script would automate installation.

I've been slogging through this process and have the beginnings of a How-To written, which can be viewed here:

http://chelmsforddtc.org/[…]/volunteer-database

Maybe even in its draft state it will help someone.

How to get fields for "friendly name"?

Posted by Almantas Pivrikas at Nov 29, 2006 06:57 PM
If you would take a look at picture sceenshot2.png, you would see the possibility to add or correct the "friendly name"s.
I run Plone bundle 2.5.1 on winXP, but I do not have that possibility.

How could I make friendly name field to be customizable?
A.

Temporarily disabled

Posted by Sean Kelly at Jan 11, 2007 12:56 PM
Editing of friendly names is temporarily disabled due to clobbering of internationalized names when using a browser with a non-English locale. Or some such. The problem is fixed in the 1.1 archetypes branch, though.

How to get fields for "friendly name"!

Posted by Mirco Kiessig at Jan 30, 2007 01:14 PM
If you want to have the opportunity to modify a "friendly name", you should go in the code of this template. For example you can do this for the "smart folder" (in the german version called "intelligenter Ordner") config-site "metadata" portal_atct/atct_manageTopicMetadata. In the ZMI you find these under the folder /portal_skins/ and are able to edit under the template /atct_manageTopicMetadata. Look there in the code for these lines:

                <tr class="" tal:attributes="class python:test(oddrow, 'even', 'odd')">
                    <td>
                        <span tal:content="indexObj/index"/>
                        <input type="hidden"
                               value=""
                               name=""
                               id=""
                               tal:attributes="id indexObj/index;
                                               name string:metadata.index:records;
                                               value indexObj/index"/>

Change the line <input type="hidden" to <input type="checkboxes"
That's all. (I know Plone sucks)

kind regards
Mirco Kießig

Returning ImageField images in metadata

Posted by John Schinnerer at Dec 31, 2006 10:21 PM
Thanks for this HowTo - very helpful!
I have one addition and one comment.

Addition is to the instructions for 'Adding Search Table Columns'.

Situation: you have a custom type that has an ImageField field and you want to return the ImageField image(s) as metadata in the results of a catalog search.
When adding the field to catalog metadata, use the actual image field name, *not* the accessor field naming convention.

For example for this schema:

ImageField('my_image',
            widget = ImageWidget(label = 'My Image'),
            sizes = {'thumbnail' : (150, 150),
                     },
           ),

If you want the thumbnail version of the image returned in catalog search metadata, then add this to catalog metadata:

my_image_thumbnail

If you try and follow the accessor naming convention and add this:

getMy_image_thumbnail

...it will not return the image as metadata in catalog search results

Comment is - when describing 'Adding Search Table Columns', don't use the word 'index' or the term 'metadata indexes'. Just say 'metadata' to avoid any confusion with the previous section on adding a catalog index, which is completely different from metadata and which difference I at least have always been easily confused by, especially when 'index' is used to describe metadata... :-)