Adding new fields to Smart Folders search

Introduction

Smart folders is the end use term for Plone's advanced search capabilities delivered by ATContentTypes product. With smart folders, one can define different fields(also called indicies in the context of search) and criterias for searches. Internally Plone refers to Smart Folders as "topics".
Smart Folders are folderish objects which are actually searches executed every time users opens them. For example, if we are building human resource management system on Plone, we might want to create a Smart Folder for employee availability. In this Smart Folder, all employees are searched and three table columns are displayed: name, project and when the project ends (employee becomes available). Also, the employee availability Smart Folder could be sorted by availability so that employees who are free in the near future are displayed first.

Steps

1.  Create a custom ATContentTypes based type
2. Add your custom fields to portal_catalog so that portal_catalog can find and index them
3. Add binding between portal_catalog indices and Smart Folder indicies
4. Create a Smart Folder to perform your searches
   

Create a custom ATContentTypes based type

Extending ATContentTypes tutorial explains how to create new content types for Plone. Unfortunately, currently this still requires knowledge of Python coding and there is not yet WYSIWYG editor for Plone content types.

Adding fields to portal_catalog

Portal_catalog is the backend tool responsible for making your site searchable. When user performs a search, portal_catalog checks from it's indices which objects match the search criteria.

Note that portal catalog catalogs items automatically when they are created or modified. If you have already existing content items at your site and you want to make them searchable, remember to manually reindex portal_catalog after creating new indicies. To do this, go to portal_catalog tool's advanced tab and hit "Update Catalog" button. If you have succesfully defined new portal_catalog indices and there exist content for them, you should see the number of indexed objects be above zero at the portal_catalog indexes tab.

Adding new search criteria

Portal_catalog is accessiable from Zope management interface. Go to portal_catalog tool, indexes tab. Add new index. The id of the new index is ATContentTypes accessor method for your field. For example, if your field is called "project" then the accessor method would be "getProject".

Different index types

• ZCTextIndex: Text Indexes break text up into individual words, and are often referred to as full-text indexes. Text indexes sort results by score, meaning they return hits in order from the most relevant to the least relevant.
  
• Keyword Indexes index a sequence of objects that act as 'keywords' for an object. A Keyword Index will return any objects that have one or more keywords specified in a search query.
• A DateIndexindexes DateTime attributes.
  
• A DateRangeIndex takes the name of two input attributes; one containing the start date of the range, the second the end of the range. This index is filled with range information based on those two markers. You can then search for objects for those where a given date falls within the range.
• Field Indexes treat the value of an objects attributes atomically, and can be used, for example, to track only a certain subset of object values, such as 'meta_type'.
• TextIndex is deprecated. Don't use it.

Adding search table columns

 Smart Folders has an option to display results as  a table. Columns available for this table are metadata indexes from portal_catalog. To add a custom column, the column must have a metadata index first. Add new metadata indexes from portal_catalog Metadata tab. Again, the index name must be the accessor of a field. If your field name is "project" then the accessor is "getProject".
 

Adding Smart Folders bindings

Creating new portal_catalog indices is not enough to make your content Smart Folders searchable. portal_catalog is just an layer under Smart Folders mechanism to make searches available for Smart Folders.
Smart Folders has a configuration view at Plone. Log-in as a adminstrator and go to your preferences page. There is a Smart Folder Settings page. Smart Folder Indexes tab controls search and sort criterias and Smart Folder Metadata contains available table columns for searches.

By default, Smart Folder Settings shows only indexes currently bind to Smart Folder fields. To show indexes you created, click "All Fields" link and check enable checkbox for those indexes. You must also add an user friendly name for your fields.

Creating a new Smart Folder

This should be pretty straightforward. Just go to the site location you wish and hit "add smart folder" for new types drop down menu. Your custom search criteria and metadata columns are available for Smart Folders if you followed instructions above.

Doing it prorammatically

To save you all going through hoops and gotchas how to get all it working in scripting, for example in quick install script, I'll have this complete example here:

    def createEmployeeSmartSearchFolder(self):
        """ Create smart search capabilities
        
        Things done here:
            - Creates new indexes (project, availability)
            - Binds indexes to smart folders
            - Creates a smart folder object for employees searches
        
        """ 
        
        # dummy class used with manage_addIndex
        class args:
            def __init__(self, **kw):
                self.__dict__.update(kw)
            def keys(self):
                return self.__dict__.keys()
            
        # Tune up catalog tool
        
        catalog_tool = self.portal.portal_catalog
        
	# Clean up previous install if there is some old stuff
        try:
            catalog_tool._removeIndex("getProject")
        except:
            pass
    
        try:
            catalog_tool._removeIndex("getAvailability")
        except:
            pass    
    
        try:
            catalog_tool.delColumn("getProject")
        except:
            pass        
        
        try:
            catalog_tool.delColumn("getAvailability")
        except:
            pass        
        
        extra = args(doc_attr='SearchableText',
                             lexicon_id='plone_lexicon',
                             index_type='Okapi BM25 Rank')    
        
        catalog_tool.manage_addIndex("getProject", "ZCTextIndex", extra)
        catalog_tool.manage_addIndex("getAvailability", "DateIndex",extra)
        catalog_tool.manage_addColumn("getProject")
        catalog_tool.manage_addColumn("getAvailability")
            
        # Update Smart Folder settings           
        smart_folder_tool = self.portal.portal_atct
        
        # Remove existing indexes if there are some
        try:
            smart_folder_tool.removeIndex("getProject")
        except:
            pass
        
        try:
            smart_folder_tool.removeMetadata("getProject")
        except:
            pass

        try:
            smart_folder_tool.removeIndex("getAvailability")
        except:
            pass

        try:
            smart_folder_tool.removeMetadata("getAvailability")
        except:
            pass

        
        smart_folder_tool.addIndex("getProject", "Employee's project", "The current project employee is working on", enabled=True)
        smart_folder_tool.addIndex("getAvailability", "Employee's availability", "Date when employee is available next time", enabled=True)
        smart_folder_tool.addMetadata("getProject", "Employee's project", "The current project employee is working on", enabled=True)
        smart_folder_tool.addMetadata("getAvailability", "Employee's availability", "Date when employee is available next time", enabled=True)
        
        # This is ugly, but we cannot override friendly names per 
        # smart folder instance. Here we use Title as Employee name
        # and to avoid confusion in table header we need to rename all
        # Title friendly names to Name 
        smart_folder_tool.updateMetadata("Title", friendlyName="Name")
                

        # Create Employee Availbilty smart folder                     
        try:
            # Remove existing instance if there is one
            portal.manage_delObjects(["employee_availability"])
        except:
            pass                

        # Create smart folder to portal root
        self.portal.invokeFactory("Topic", id="employee_availability", title="Employee Availability")
        employee_availability = self.portal.employee_availability
        
        # Sort results based on availability
        employee_availability.setSortCriterion("getAvailability", reversed=False)
        employee_availability.setDescription("List employees, sorted by when they are free")
        
        # Filter results to employees
        type_criterion = employee_availability.addCriterion('Type', 'ATPortalTypeCriterion' )
        type_criterion.setValue("Employee")
            
        
        # Display as table
        employee_availability.setCustomView("True")        
        employee_availability.setCustomViewFields(("Title", "Description", "getAvailability"))
    
        self.out.write("Created employee search smart folder")

Related source files

If you want to do things programmatically (i.e. when your product quick install script is run), you should check following files as a starting point for creating the script

• ATContentTypes/tool/topic.py
• CMFCore/CatalogTool.py