We have a web event calendar where people can schedule events (for far into the future). When people input an event, they specify the date/time it will begin/end. The application then converts the date/time into a unix timestamp and stores it in the database. Later, when an event is being viewed, the timestamp is converted back to a textual date/time.
The conversion from local time to timestamp is done via PHP functions, which uses the systems timezone file. The OS patch to fix this problem simply updates the timezone file and everything should automatically work.
This is fine and dandy for most things, but I ran into one small glitch. For any events that are scheduled between the new start of DST and the old start of DST (roughly a 3 week period), if they were created BEFORE the patch was applied, they are now off by 1 hour AFTER the patch is applied. (the same would be true of anything in the week or so between the new and old END of DST also)
The 1 hour difference is simple enough, and I might not have even noticed...or if I had, I probably would have just blamed it on user error when entering the data. However, this caused a strange side effect. The application built a secondary index to simplify searching for events. The index consisted of the event_id and the starting timestamp truncated to midnight. Of course, for those events in that 3 week period that were created before the patch was applied, they now had an index timestamp that was actually 1 AM. Of course, this caused events to mysteriously disappear from the calendar. You could search for them, but not see them when browsing.
I wasted a good couple of hours on this problem (thinking at first that it was isolated to a single event) before I finally did an analysis of the entire index and discovered that almost all events in that 3 week period were wrong. Then it instantly dawned on me what had happened.
You see, a subtle bug is much harder to fix than a blatent one. We've got one very subtle bug headed our way. It'll be annonying at mininum, and could be a really big mess... lets hope it stays small.
Thanks for your time and attention.
--Mike--