Resolved -
Summary: Between 15 January at 18:00 CET and 15 January at 19:00 CET, a small subset of CEP trigger automations was delayed for up to an hour.
Only trigger automations using timers and with timers scheduled to fire between 18:00 and 19:00 CET were affected; this represents less than 2% of all trigger automations during this time period.
Impact for the affected trigger automations:
* Timer firing was delayed by up to one hour.
* As a consequence, the delivery of emails, SMS, and push notifications was delayed by up to one hour.
Root Cause: some timers were not accepted by our database system due to an edge case in their data, which blocked the entire timer scheduling pipeline.
Resolution: a fix was deployed on 15 January at 18:55 CET, which resolved the issue. By 19:00 CET, the backlog was cleared.
Jan 15, 17:00 UTC