Push campaign delivery issues
Incident Report for Batch
Our team has identified a bug in the language targeting and the assignment of message translations for push campaigns which could have significantly impacted the number of users targeted by such campaigns.

Here are the details:
* users on iOS, Windows and the web were affected no matter which SDK version
* users on Android and using an application with a SDK version >= 1.18 were affected
* users on Android and using an application with a SDK version < 1.18 were _not_ affected
* push campaigns created both via the dashboard and the API were affected

Since the 2021/12/03 at 13:30 UTC and until the 2021/12/06 at 16:00 UTC a bad version of a background service responsible for managing the user data was deployed in production.
This version introduced a new feature for improved language targeting, intended to be released later in the year; this feature stores data in a way that is not directly compatible with other backend services responsible for sending push campaigns.
Due to a release management issue this background service was deployed earlier than it should have been, thus causing the issues with the dependent services.

As soon as our team was made aware of this problem we implemented a workaround in the dependent services.
As of the 2021/12/06 at 16:00 UTC all language targetings work as expected, for all users.

In the future we will work on making our deployment process more resilient to user errors so that incompatible app versions cannot be deployed together.
Posted Dec 03, 2021 - 13:30 UTC
This incident affected: Delivery (Push campaigns).