Summary

Starting at 18:00 UTC we experienced significant push notification delivery delays as well as increased API error rates.

Between 18:00 UTC and 20:15 UTC customers might have experienced delivery delays of 35 minutes at worst.

Customers using the following APIs might have experienced increased error rates:

Custom Audience API
Custom Data API
App Data API
GDPR API
Export API

Root cause

This was due to a network connectivity issue with our hosting provider which impacted some of our database servers.

After 20:15 UTC the situation improved enough for our database servers to work correctly again but we were still seeing some small
connectivity issues.
Our hosting provider worked overnight and resolved the issue on their side; the connectivity issues were completely resolved on the 2021-06-14 at around 00:30 UTC.

Posted Jun 14, 2021 - 12:16 UTC

Resolved

The issue has been resolved with our hosting provider during the night and our database servers are fully operational again. We didn't encounter push notificaition delivery delays or increase API error rates since the previous status update.

Posted Jun 14, 2021 - 07:59 UTC

Monitoring

The situation has improved and we're no longer seing delivery delays or increased API error rates, however our database servers don't have 100% recovered yet. Our hosting provider is still working on resolving the issue on their side.

Posted Jun 13, 2021 - 20:58 UTC

Update

We are having issues communicating with some of our databases servers. We're still working on resolving the problem.

Posted Jun 13, 2021 - 19:44 UTC

Identified

The issue has been identified with our hosting provider. We're currently working on a resolution.

Posted Jun 13, 2021 - 19:22 UTC

Investigating

We are aware of push notification delivery delays as well as increased API error rates. We are currently investigating.

Posted Jun 13, 2021 - 19:00 UTC

This incident affected: API (Transactional API, Campaigns API, Custom Data API, Transactional API for Partners), Webservices (SDK, Inbox, Custom attributes, In-app messaging, Web push static resources), and Delivery (Transactional push, Push campaigns, In-app messaging).