Previous incidents

We have stopped the deployment and rolled-back to the previous version. We're no longer seeing dropped requests.

A previous deployment changed some indexes to make some queries faster, but it also resulted in slower queries in other areas.
We have performed a series of optimizations that should lead to faster queries overall.
We've added load testing for the most common queries to catch future regressions and are monitoring timings in production.

We've identified the cause and are working on a fix.

We're seeing very slow write queries at peak traffic that have bad consequences.

We released a first mitigation to limit the impact of those queries and are working on understanding why those queries are slow.

The API was not available during 2 time chunks

chunk 1: from 12:34:33pm to 12:35:08pm (35 seconds)
chunk 2: from 12:35:46pm to 12:36:42 (56 seconds)
The first chunk was the result of an unexpectedly large analytics query from one of our customers combined with an improper thread management at the server level (blocking other queries).

The second chunk was a consequence of the accumulation of requests during the first chunk that overloaded the replicas once they started processing requests a...