What happened?

In an effort to improve the performance of our iOS push notification infrastructure, we started rolling out an upgrade of our backend applications.

This phased rollout targeted 2% of our daily volume of iOS Campaigns & Recurring automations.

The new version contained a bug that caused applications sharing the same APNs* p8** push configuration across multiple apps to experience a high bounce rate, leading to incorrect “apns2_device_token_not_for_topic” errors.

Apps using p12 based push configurations were not affected.

Once we were made aware of the problem, we stopped the rollout and reverted it.

‌

*APNs = Apple Push Notification service

**p8 = private key used to authenticate against Apple’s servers

‌

Impact on our platform

From September 26, 14:45 GMT+2 to September 27, 10:30 GMT+2, part of our iOS Campaign & Recurring automation traffic returned “apns2_device_token_not_for_topic” errors while they should not have.

We have not tried to send the failed notifications a second time. The analytics of the affected Campaign/Recurring automations are accurate.

‌

All other components worked as expected:

• iOS Transactional & Trigger automation pushes
• Android & Webpush: all push notifications
• In-apps
• Email
• SMS
• APIs
• Dashboard
• Analytics

‌

Timeline

For clarity, this timeline only lists the most important events. All times are GMT+2

September 26, 14:45

We begin the gradual deployment of our improvements to the iOS push notification architecture. Starting with 2% of our daily volume of iOS Campaigns & Recurring automations.

September 27, 09:57

We notice a significant number of bounces on the campaigns since September 26 at 18:30.At this point we check if it is related to our deployment from the day before.

September 27, 10:05

We identify a campaign impacted by this problem to analyze the cause.

September 27, 10:20

We open an incident and confirm that it is related to yesterday's deployment.

September 27, 10:30

We are rolling back the changes to the previous iOS push notification architecture.

September 27, 11:34

We open a status page incident. We are still analyzing the impacted applications

September 27, 14:50

The problem is reclassified as major, concerns few applications but the applications concerned were unable to send a major part of their campaign & recurring automation push notifications.

‌

Actions

The bug has been fixed but postponed the rollout for a later time (we reverted the code to its original state before introducing it).

We are looking into improving our alaming to catch this specific kind of error at low volumes.