#158: Recurring events

1. datetime is always rendered in GUI with current timezone: some problem occurs when we try to calculate future thisDay datetime objects. We observe that for long period of recurrence, which goes across timezone boundaries (summer/winter time), Plone renders datetime shifted plus/minus 1 hour. This happens becouse local server datetime at winter time is different than at summer time.

2. recurrence of last two days of month in 'monthly' mode: if we try to recue 31'th May when should we land? At 30'th June or at 1'st July? Escalated problems arrives near February.

3. leap year: February could have different length and the whole year also.

4. naive shifting 'datetime.now() + n' algorithm: becouse of point 2. and 3. We can not use it. Suggestion: n should be calculated in context of year and month pair.

5. quering catalog to build calendar view: in standard portal_calendar view we make maximum 31 queries but with CalendarX at multi-month view we could reach houndres of queries. Multiply it by 10 request/s and it will kill our sweet Plone even if we will have few, but recured during very long time period, events - every day for 2 years for example.

6. indexing recurring events: I've found and tried two paths, the dark and the light one. The dark path is to build indexes for fields which describes recurring attributes - we don't use much memory and catalog query is fast but when we want to query for events for 'thisDay' we have to filter all results and make acceptance test. Going this way I've visited the hell named 'objectify' and during acceptance test I've awaken RecurringEvent objects from ZODB to invoke method 'isRecueForDay(thisDay)' - simple hermetized and clear desing but naive. The light path was to index list of specially prepared numbers which represents number of day since epoch for every day for RecurringEvent when event appears. So quering for thisDay was very simple and very fast, but use more memory for indexes - but it was acceptable cost - imho.

7. separation of logic from content: it could be hard to play with RecurringEvent if the user have to know too much details about how to query for events in proper way. My suggestion is to build specialized tool for that with hermetized queries like: listEventsFor(thisDay) and isAnyEventFor(thisDay). These two methods would be enought to implement most of calendar's views.

8. iCal/vCal import/export: RFC2445 is the source. Not only for import/export but also for other things like attendees, recue exceptions and so on. There is also microlanguage for specifing recurrence modes, exceptions and much more. Maybe we should consider to implement attribute of type lines where per line we will have specified recurrence rule? And embed knowledge about it into portal tool?

9. attendees: even ATEvent have this field but do nothing with it, even doesn't use it in export iCal/vCal. We should use it becouse it is very important field and adds many features to Plone collective related aspects. We should be able to invite attendees to visit and import our instance of RecurringEvent - here appears the need for additional workflow. If the person we invite have e-mail she should be notified, if the person is member of our Plone portal maybe we should consider management of LocalRoles of our instace object ot give she ability to view private events.

Tommorow I'll write additional comments ;)

Regards

Pawel Marzec

Pawel, thanks for your comments you bring up some good points - I'll answer the ones I have ideas for and leave the rest open for suggestions.

   1. datetime is always rendered in GUI with current timezone: some problem occurs when we try to calculate future thisDay datetime objects. We observe that for long period of recurrence, which goes across timezone boundaries (summer/winter time), Plone renders datetime shifted plus/minus 1 hour. This happens becouse local server datetime at winter time is different than at summer time.
   
   I have seen this problem with my current recurring event project just this past month, but I think a large part of this is that we were creating event start and end times using the regular event schema. Perhaps we should override the ATEvent schema and use our own fields with the Operating Systems timezone function. This option would require that we convert the start and end time into UTC for storing it and then converting it back from UTC to the local time of the operating system when displaying. Sounds messy but it may actually solve a problem that currently exists? Ultimately this problem sounds like one stemming from ATEvent rather than any recurring event? The time of the event shouldn't be dependant upon the time the server is currently set at and I don't know if it is the scope of this proposal to try and "fix" it? If we are extending ATEvent we may need to get some help from the core developers of ATEvent to support recurring event time stamps? I'm open to any suggestions on this one from everyone involved.
   
   2. recurrence of last two days of month in monthly mode: if we try to recue 31'th May when should we land? At 30'th June or at 1'st July? Escalated problems arrives near February.
   
   This is a problem that is rather difficult. The proposal calls that we should do "Monthly" mode based upon the WEEK and DAY of the month rather than the numeric date of the month. By utilizing first, second, third, fourth, or last weeks of the month we can avoid confusion as to where an event should land. Also the majority of recurring events occur on a "specific week of the month" rather than a specific "day" of the month as the day is always changing each month. I've never seen a meeting on the 30th of each month for example. However my company bagel meeting is the first Monday of each month and my condominium association meets on the third Wednesday of each month. Let me know if you feel this is the best way to handle this as well. Recurring Events v 1.0 uses logic that has already been tested to solve this problem and handles all months that we can see (including February in leap years).
   
   3. leap year: February could have different length and the whole year also.
   
   This is handled from the same logic as number two's solution. By using weeks and days of the month we are not dependant upon a numeric number of days within each month. The business rule logic such as "recurs every first Monday monthly" or "recurs last Friday monthly" helps to keep the logic behind the recurring event seperate from any specific dates within a given month.
   
   4. naive shifting datetime.now() + n algorithm: becouse of point 2. and 3. We can not use it. Suggestion: n should be calculated in context of year and month pair.
   
   I'm going to assume that you are referring to the proposal's logic within the implementation that states, "thisDate is correct number of days away from StartDate (7 for weekly, 14 bi-weekly, etc)". This is far from naive, rather this is a very simple summary of what we have done with recurring event 1.0. The logic behind it is a lot more complex than I'd like to admit to as there are arrays for each month and how many days are in each month based upon leap year, regular year, etc. You can look at recurring event v 1.0 for more details on our business logic on ensuring that an event is created at its next frequency correctly. There are quite a few tests to determine which month it is, how many days are in the month, when the last date was, when the new date should be, etc. All of this logic can be pretty easily converted for use as function to determine if the startDate is the appropriate length away from the date (thisDate) being queried by the calendar. In other words, we are basically doing a more specific version of what you have suggested - not just adding 7 or 14. I was trying to keep the wording of the implementation simple for the proposal as it should not be limited just to how I've implemented recurring events v. 1.0 - as long as we can make sure the recurring events display correctly it shouldn't matter how we accomplish it.

   5. quering catalog to build calendar view: in standard portal_calendar view we make maximum 31 queries but with CalendarX at multi-month view we could reach houndres of queries. Multiply it by 10 request/s and it will kill our sweet Plone even if we will have few, but recured during very long time period, events - every day for 2 years for example.
   
   I would think that the best things we could do is find a way to create ONE query that grabs ALL recurring events and then find a way to use the results of that one query within each day that needs to be tested. I feel this is the simplest way to ensure that our logic is as efficient as possible. Requerying the catalog each day for all the recurring events is sort of pointless as we have to test EACH recurring event anyway to see if it is appropriate to display on each day. This is the biggest drawback to a single object implementation - it will take longer to display than it does now, the goal is to keep that extra length as minimal as possible. One solution that comes to mind for me is that we use the start date of the month and the end date of the month in our query to retrieve only recurring events that occur infinitely, or have an end date in the right range. This slightly more complex query would let us avoid having to do a check later in the calendar logic on whether or not the event has a recurring end date that fits in the current display dates or not. Also, this ensures that every recurring event we do get back will need to be displayed at some point - we will just need to use our functions for testing if the event should be displayed against our array of results rather than repeating multiple queries.
   
   Another option (better the more I think about it as it is inspired by number 6 below - indexes) that comes to mind is to create a function that is invoked towards the beginning of the calendar portlet that grabs ALL recurring events from the catalog and then uses the calendar's start date and end date to test each recurring event and create a temporary index of the days it will need to be displayed. This may be the simplest way as we only test each recurring event once and there is only one query catalog. After this single test for each recurring event (that has to happen sooner or later) we are left with a tuple of dates that it needs to be displayed. Integrating this tuple into any calendar system (Plone default portlet, CalendarX, etc) will probably be much easier than redesigning our query for each calendar system. Let me know if you agree with this idea to be the most efficient?
   
   6. indexing recurring events: I've found and tried two paths, the dark and the light one. The dark path is to build indexes for fields which describes recurring attributes - we don't use much memory and catalog query is fast but when we want to query for events for thisDay we have to filter all results and make acceptance test. Going this way I've visited the hell named objectify and during acceptance test I've awaken RecurringEvent objects from ZODB to invoke method isRecueForDay(thisDay) - simple hermetized and clear desing but naive. The light path was to index list of specially prepared numbers which represents number of day since epoch for every day for RecurringEvent when event appears. So quering for thisDay was very simple and very fast, but use more memory for indexes - but it was acceptable cost - imho.
   
   I agree that the idea of querying for thisDay on each recurring event is going to take to long. It was my biggest concern going into a single object implementaion. See number five's response above and let me know if you think that indexing the dates a recurring event will display in a given date range from the calendar is the simplest way to do this infinitely? The logic in the function could easily use the logic from recurring event v. 1.0 where we actually created individual event objects for each recurring event based upon a start and end date range. Modifying that to simply create a tuple of dates would be a simple solution to creating a temporary index that requires only one query and one while loop for each recurring event.
   
   7. separation of logic from content: it could be hard to play with RecurringEvent if the user have to know too much details about how to query for events in proper way. My suggestion is to build specialized tool for that with hermetized queries like: listEventsFor(thisDay) and isAnyEventFor(thisDay). These two methods would be enought to implement most of calendar's views.
   
   This is a great proposal. One thing that has been bothering me was how we would get our recurring event object to display in smart folders as "upcoming events". This kind of function would be a good solution and if we could possibly find a way to add it into the smart folder display view somehow (not sure off-hand) then it would provide a possible solution for this as well as other calendars that come out. Perhaps it may be simple enough to say "getEventsForRange(startDate,endDate)". This function would be the same as described in 5 & 6 responses above and would work for a date range or one day just as easily and would obviously return the same format for dates that each recurring event occurs in the given range.
   
   8. iCal/vCal import/export: RFC2445 is the source. Not only for import/export but also for other things like attendees, recue exceptions and so on. There is also microlanguage for specifing recurrence modes, exceptions and much more. Maybe we should consider to implement attribute of type lines where per line we will have specified recurrence rule? And embed knowledge about it into portal tool?
   
   I have read over RFC2445 quite a bit and think you make some good oints. I am limited on my knowledge as to supporting other calendar programs, if we could find a way to improve the iCal/vCal support of recurring events that would be great. I think that it may be easiest for us to create our recurring event schemata and implementation first and then find a way to use our stored data to create the iCal / vCal formatted data from our stored data (a function that is invoked only when they want to download it perhaps)? Since iCal and vCal download links are only visible while viewing a specific event it would be easy to invoke these functions from a custom view we already have to provide for recurring events (since we already have to display recurrence frequency, recurring end date / infinite, and exception dates).
   
   9. attendees: even ATEvent have this field but do nothing with it, even doesn't use it in export iCal/vCal. We should use it becouse it is very important field and adds many features to Plone collective related aspects. We should be able to invite attendees to visit and import our instance of RecurringEvent - here appears the need for additional workflow. If the person we invite have e-mail she should be notified, if the person is member of our Plone portal maybe we should consider management of LocalRoles of our instace object ot give she ability to view private events.
   
   I would like to see attendees used more as well; however, I don't know if the scope of this proposal provides for that any more than it provides for fixing start/end times of ATEvent. Although this is a good improvement I don't think we should necessarily worry about it in a recurring events implementation as we are just extinding ATEvent for its pre-existing fields to ensure maximum compatibility with current event infrastructure in Plone.
   
Thanks for your suggestions and I'm looking forward to your help/comments/etc,
   
Steven Haddox

For an attendee implementation take a look in mxmCalendarTypes. I have a sensible implementation of attendees in there.

It also has many more parts of the icalendar spec implemented than other calendar content types.

With regard to multiple queries in the catalog. That is just poor coding practice. Get all the events for eg. a month, then save them in a dict with the same resolution as the calendar. Like:

    resolution_key = (event.year(), event.month(), event.day(), event.hour())
    events[resolution_key] = event

For a resolution of an hour.

Then it is simple to get the event for a specific hour:

    hour_event = sequence.sort(events.get(resolution_key))

With regards to recurring events, please just follow the iCal standard. It is more than sufficient for what a normal user can comprehend. No reason to reinvent the wheel. An easy to use interface would be good enough.

Thanks for pushing the iCal standards, I will ensure that I look at your products related to this (mxmCalendarTypes and mxm-iCal-Tool) as an example for attendees and iCal exporting abilities. I think you are absolutely correct that we should simply reuse the code from iCal standards, it will require a little more research but I have a feeling it will make the recurring event much more compatible and standard compliant (big issues to me). Thanks for pushing it.

-Steven

Much of the difficult computation dealing with recurrence calculations can be done using the dateUtil library (http://labix.org/python-dateutil). A mixin class that provides support for generating a list of dates from a given set of recurrence rules (which dateUtil does), an index that is optimized for storing and querying long lists of dates (essentially a KeywordIndex that does a conversion of dates to integers (probably with 1 minute resolution), and a nice ui for setting recurrence rules (the hardest part probably, but it can start by supporting only a limited set of rules) is probably 90% of what's needed for this (leaving out support for showing these things in calendars, which shouldn't be too hard once you have the above). It would also be nice if there was a way to traverse to a particular recurrence, so that clicking on a calendar link for a specific date yielded information specific to the clicked date. This would also make it easy for changes to be made to a specific recurrence, in which case a new single event would be spawned and the original date would be added to the exclusion list.

The PLIP is now updated and changed rather radically. :)

We (Steve and Larry) are also working on integration with CalendarX(Plone 3 version).