After a long period of observation, we're confident that every service are back to a nominal status. Incident went from 8:15am to 11:45am UTC.
Posted Jun 19, 2020 - 14:17 UTC
Update
We're aware of a partial impact of that outage on statistics. We're taking measures to address that impact as soon as possible. More information will come regarding potential data loss during our downtime. Dashboards are operational. API and campaigns are fully available.
Posted Jun 19, 2020 - 11:25 UTC
Monitoring
Restore is complete, we're monitoring the situation and ensuring that there is nothing else in incident's scope.
Posted Jun 19, 2020 - 10:37 UTC
Update
While restoration is still ongoing, we wanted to provide more information about this database incident impact:
* New push campaigns can not be created or existing push campaigns can not be edited at the moment. * Dashboard and editorial dashboard have been put in maintenance mode and are not available. * Third-party connectors relying on the Transactional API for Partners are not working as they rely on the Push campaigns.
Other services are not impacted such as: * Push campaigns (scheduled, recurring, trigger) created/edited before today ~08:15 UTC are working and sending as expected. * Transactional API is working and push are sent as expected. * In-app campaigns are working: synced and displayed to the end-users devices as expected. * All data coming from the SDKs (such as new optins, tagging plan) or from the Custom Data and Audience API are indexed as expected.
We'll share an ETA for a full resolution as soon as we can.
Posted Jun 19, 2020 - 10:30 UTC
Update
Restoration is going on well, we have several backup contingency plans ready to be executed if there was any unforeseen caveats on the currently implemented fix.
Posted Jun 19, 2020 - 10:04 UTC
Update
We are continuing to work on a fix for this issue.
Posted Jun 19, 2020 - 09:02 UTC
Identified
A human error triggered loss of data, we're restoring data and will keep you posted about progress.
Posted Jun 19, 2020 - 08:56 UTC
Investigating
We're encountering a technical issue on our dashboard, preventing users to log-in or use the dashboard. We're investigating and will keep you posted.
Posted Jun 19, 2020 - 08:20 UTC
This incident affected: Dashboard (Main dashboard, Editorial dashboard) and API (Campaigns API, Transactional API for Partners).