Postmortem -
Read details
Aug 21, 07:11 UTC
Resolved -
The system has been stable for the last 12 hours.
As of August 14, 18:04 CEST:
- Late MEP & CEP Trigger Automation messages have been sent
- Some MEP Push Campaigns & Recurring Automations have been partially send and will not be retried
- API error rates are back to normal
- Webhooks are back to normal
This incident is now considered as resolved.
Aug 15, 05:37 UTC
Monitoring -
The system has now fully processed the backlog of CEP and MEP automation messages, including Email, SMS, and Push v1 and v2.
All previously delayed messages have been sent.
We are now moving into a monitoring phase, during which the on-call team will closely watch system performance.
Aug 14, 16:04 UTC
Update -
The system is still catching up with the backlogged work.
All late CEP & MEP Automation messages (Email, SMS, Push v1 & v2) will be sent.
We will provide further updates as soon as we have more information or in 1 hour, whichever comes first.
Aug 14, 15:36 UTC
Update -
The database issue has been resolved.
All components are gradually returning to normal.
Processing speed is being increased progressively, with priority given to maintaining system stability.
We will provide further updates as soon as we have more information or in 1 hour, whichever comes first.
Aug 14, 14:38 UTC
Update -
We are still working on fixing the database problem.
The affected components are still the same.
Aug 14, 13:33 UTC
Identified -
We have been experiencing a database issue since August 14th, 12:10 CEST.
We are working on a complete report of what is affected.
In the meantime, here is a brief overview of the platform’s status
* CEP Automations (Email, SMS, Push v2) are not being sent
* MEP Automations (Push v1) are not being sent
* MEP Campaigns are partially being sent
* CEP Campaigns are partially being sent
* The transactional API (MEP) is working
* Data sent on APIs for which we sent back a 200 HTTP status code may not be processed live but is enqueued and will eventually be processed
* Some APIs might experience elevated error rate over the afternoon
* Webhooks are partially working
We will provide further updates as soon as we have more information or in 1 hour, whichever comes first.
Aug 14, 12:23 UTC