Whilst conducting a routine maintenance (cleaning some stale data) on our Cassandra database cluster, a part of one of our clusters started to consume a large amount of memory which led to a partial collapse.

At first, we didn't suspected this simple database operation was the cause, and we focused on getting the cluster back on its feet. A further investigation revealed that this outage was clearly due to human intervention.

During this outage :

• the transactional and campaign API requests were still accepted
• campaign creations and updates through our dashboard were still working
• the push notification delivery were significantly slowed down (during 28min)
• the Inbox service was unavailable during the whole incident (during 48 min)
• almost the totality of impacted notifications have been sent but with up to a 1h delay