Previous incidents

The incident has been solved.
The issue arose after a change in the client's data.
This change triggered messaging calls for heavy queries, leading to an overload of the workers.

We have mitigated the situation for now and are working on a more systematic fix for later.

The bug regarding Routines has been fixed, but unfortunately the correct data from that period cannot be recovered.

We will assist the recovery of our customers' operations in the following way:
- Providing a list of the affected Routine Executions that have invalid data.
- Offering to remove the affected Routine Executions.
- Rebuilding the data on Dynamic KPIs that are based on those Routines.

The root cause comes from a feature that was supposed to improve the loading time of images and avatars. This feature was intended to enhance the way thumbnails are managed in the legacy code. This improvement has been postponed and will be implemented in the new stack.

We are found the root cause which a specific endpoint. We will work on this endpoint to prevent future slowdowns.