Push delivery delays and increased API error rates
Incident Report for Batch
Postmortem

Summary

Starting at 18:00 UTC we experienced significant push notification delivery delays as well as increased API error rates.

Between 18:00 UTC and 20:15 UTC customers might have experienced delivery delays of 35 minutes at worst.

Customers using the following APIs might have experienced increased error rates:

  • Custom Audience API
  • Custom Data API
  • App Data API
  • GDPR API
  • Export API

Root cause

This was due to a network connectivity issue with our hosting provider which impacted some of our database servers.

After 20:15 UTC the situation improved enough for our database servers to work correctly again but we were still seeing some small
connectivity issues.
Our hosting provider worked overnight and resolved the issue on their side; the connectivity issues were completely resolved on the 2021-06-14 at around 00:30 UTC.

Posted Jun 14, 2021 - 12:16 UTC

Resolved
The issue has been resolved with our hosting provider during the night and our database servers are fully operational again. We didn't encounter push notificaition delivery delays or increase API error rates since the previous status update.
Posted Jun 14, 2021 - 07:59 UTC
Monitoring
The situation has improved and we're no longer seing delivery delays or increased API error rates, however our database servers don't have 100% recovered yet. Our hosting provider is still working on resolving the issue on their side.
Posted Jun 13, 2021 - 20:58 UTC
Update
We are having issues communicating with some of our databases servers. We're still working on resolving the problem.
Posted Jun 13, 2021 - 19:44 UTC
Identified
The issue has been identified with our hosting provider. We're currently working on a resolution.
Posted Jun 13, 2021 - 19:22 UTC
Investigating
We are aware of push notification delivery delays as well as increased API error rates. We are currently investigating.
Posted Jun 13, 2021 - 19:00 UTC
This incident affected: Webservices (SDK, Inbox, Custom attributes, In-app messaging, Web push static resources), Delivery (Transactional push, Push campaigns, In-app messaging), and API (Transactional API, Campaigns API, Custom Data API, Transactional API for Partners).