Incidents reported on status page for Fabriq — Lonza environment https://lonza-status.fabriq.tech/ en https://lonza-status.fabriq.tech/incident/575827 Thu, 22 May 2025 10:30:00 -0000 https://lonza-status.fabriq.tech/incident/575827#d66b79272ac3ac162b52b9dec08cc6071a50cb1e292f62f3f66b8f120f606cf7 All stalled indicator computations have been performed, and the incident is now over. https://lonza-status.fabriq.tech/incident/575827 Thu, 22 May 2025 10:30:00 -0000 https://lonza-status.fabriq.tech/incident/575827#d66b79272ac3ac162b52b9dec08cc6071a50cb1e292f62f3f66b8f120f606cf7 All stalled indicator computations have been performed, and the incident is now over. https://lonza-status.fabriq.tech/incident/575827 Thu, 22 May 2025 10:15:00 -0000 https://lonza-status.fabriq.tech/incident/575827#3270a6cefa10f2f76d98cd4c6b2d87aab9a2bfbcf71590305418a607d91eb873 Redis capacity has been increased. Ticket creation is back to normal and indicator computations that were stalling and now being processed https://lonza-status.fabriq.tech/incident/575827 Thu, 22 May 2025 10:15:00 -0000 https://lonza-status.fabriq.tech/incident/575827#3270a6cefa10f2f76d98cd4c6b2d87aab9a2bfbcf71590305418a607d91eb873 Redis capacity has been increased. Ticket creation is back to normal and indicator computations that were stalling and now being processed https://lonza-status.fabriq.tech/incident/575827 Thu, 22 May 2025 09:55:00 -0000 https://lonza-status.fabriq.tech/incident/575827#1a2c052723f4b104852eb89c4196785ff498d648cfb35cdce9c2a2d394e5fbc2 We have identified the root cause of the incident. It is due to a lack of capacity in Redis, the technology we use for caching and to perform long-running operations, such as exports and computation of dynamic indicators. While the infrastructure was under capacity, computation of dynamic indicators stalled. We have increased the capacity of Redis and the new capacity is being deployed. https://lonza-status.fabriq.tech/incident/575827 Thu, 22 May 2025 09:55:00 -0000 https://lonza-status.fabriq.tech/incident/575827#1a2c052723f4b104852eb89c4196785ff498d648cfb35cdce9c2a2d394e5fbc2 We have identified the root cause of the incident. It is due to a lack of capacity in Redis, the technology we use for caching and to perform long-running operations, such as exports and computation of dynamic indicators. While the infrastructure was under capacity, computation of dynamic indicators stalled. We have increased the capacity of Redis and the new capacity is being deployed. https://lonza-status.fabriq.tech/incident/575827 Thu, 22 May 2025 09:50:00 -0000 https://lonza-status.fabriq.tech/incident/575827#4e9be17f5e30fdf9026dedcc66d3a7d1571d4065d9c77ec39cbc1c0aebbccbd5 Indicators are stuck in a recomputing state and tickets are failing to get created https://lonza-status.fabriq.tech/incident/575827 Thu, 22 May 2025 09:50:00 -0000 https://lonza-status.fabriq.tech/incident/575827#4e9be17f5e30fdf9026dedcc66d3a7d1571d4065d9c77ec39cbc1c0aebbccbd5 Indicators are stuck in a recomputing state and tickets are failing to get created https://lonza-status.fabriq.tech/ Wed, 21 May 2025 08:12:09 +0000 https://lonza-status.fabriq.tech/#a8c086b549257df4717e7521f2611b0357484017e5764c3e6cf56b655e7254ca API — Lonza recovered https://lonza-status.fabriq.tech/ Wed, 21 May 2025 08:11:35 +0000 https://lonza-status.fabriq.tech/#a8c086b549257df4717e7521f2611b0357484017e5764c3e6cf56b655e7254ca API — Lonza went down https://lonza-status.fabriq.tech/ Wed, 21 May 2025 08:04:08 +0000 https://lonza-status.fabriq.tech/#543f7fea1ed4b65ee873b1c1ba3d018db6eed83757678d173aca6d7620299d9b API — Lonza recovered https://lonza-status.fabriq.tech/ Wed, 21 May 2025 07:54:40 +0000 https://lonza-status.fabriq.tech/#543f7fea1ed4b65ee873b1c1ba3d018db6eed83757678d173aca6d7620299d9b API — Lonza went down https://lonza-status.fabriq.tech/ Thu, 08 May 2025 09:07:33 +0000 https://lonza-status.fabriq.tech/#51d7ff1286fc6dc069451bfef4a7762033ee2445e890395d45928b6b7c45ad25 API — Lonza recovered https://lonza-status.fabriq.tech/ Thu, 08 May 2025 09:07:30 +0000 https://lonza-status.fabriq.tech/#51d7ff1286fc6dc069451bfef4a7762033ee2445e890395d45928b6b7c45ad25 API — Lonza went down https://lonza-status.fabriq.tech/ Thu, 01 May 2025 13:00:11 +0000 https://lonza-status.fabriq.tech/#d5c7918beaeba708b7b9a390fac0768e85f79b71d280b1989558ee1649df0412 API — Lonza recovered https://lonza-status.fabriq.tech/ Thu, 01 May 2025 12:59:39 +0000 https://lonza-status.fabriq.tech/#d5c7918beaeba708b7b9a390fac0768e85f79b71d280b1989558ee1649df0412 API — Lonza went down https://lonza-status.fabriq.tech/ Tue, 29 Apr 2025 11:58:28 +0000 https://lonza-status.fabriq.tech/#0bfbde0b78ad29421c65b7b319105cf2f27206d1c48fca4c223cbbb778b13ba3 API — Lonza recovered https://lonza-status.fabriq.tech/incident/553770 Tue, 29 Apr 2025 11:58:00 -0000 https://lonza-status.fabriq.tech/incident/553770#10b1116919f2fde3c7a773ff619d4bdcab397dc6f4bdc26d16dd34bb88f30c1b We have identified the root cause and have reverted the changes made that affected our API. The root cause was linked to a routine server dependency update, that was reverted to resolve the incident https://lonza-status.fabriq.tech/incident/553770 Tue, 29 Apr 2025 11:58:00 -0000 https://lonza-status.fabriq.tech/incident/553770#10b1116919f2fde3c7a773ff619d4bdcab397dc6f4bdc26d16dd34bb88f30c1b We have identified the root cause and have reverted the changes made that affected our API. The root cause was linked to a routine server dependency update, that was reverted to resolve the incident https://lonza-status.fabriq.tech/ Tue, 29 Apr 2025 11:49:26 +0000 https://lonza-status.fabriq.tech/#0bfbde0b78ad29421c65b7b319105cf2f27206d1c48fca4c223cbbb778b13ba3 API — Lonza went down https://lonza-status.fabriq.tech/ Tue, 29 Apr 2025 11:44:58 +0000 https://lonza-status.fabriq.tech/#2f6b8341c5a0185d3a0c98459bc58f625d27362876aec78dbe74b569048c0509 API — Lonza recovered https://lonza-status.fabriq.tech/ Tue, 29 Apr 2025 11:42:57 +0000 https://lonza-status.fabriq.tech/#2f6b8341c5a0185d3a0c98459bc58f625d27362876aec78dbe74b569048c0509 API — Lonza went down https://lonza-status.fabriq.tech/incident/553770 Tue, 29 Apr 2025 11:42:00 -0000 https://lonza-status.fabriq.tech/incident/553770#13afd333f521c82e40bb513330af7f7c781dc5c508d112a512ff25d73bc5e66c We are currently experiencing an incident affecting our API, causing it to be unavailable. Our team is investigating the issue and working to restore service as quickly as possible. We will provide further updates shortly. https://lonza-status.fabriq.tech/incident/553770 Tue, 29 Apr 2025 11:42:00 -0000 https://lonza-status.fabriq.tech/incident/553770#13afd333f521c82e40bb513330af7f7c781dc5c508d112a512ff25d73bc5e66c We are currently experiencing an incident affecting our API, causing it to be unavailable. Our team is investigating the issue and working to restore service as quickly as possible. We will provide further updates shortly. https://lonza-status.fabriq.tech/incident/549580 Wed, 23 Apr 2025 07:36:00 -0000 https://lonza-status.fabriq.tech/incident/549580#c18574b993b42317b532b34beab4373d35d273761a83001ba0cf30ed1cbacbbf The performance issues (including slowness and potential access problems) reported earlier this morning, following our recent system maintenance, have now been fully resolved. Our team has completed the necessary corrective actions, and all services have returned to normal operation. We have confirmed that system performance is stable. We apologize for the disruption this may have caused and thank you for your patience and understanding while we worked to resolve it. https://lonza-status.fabriq.tech/incident/549580 Wed, 23 Apr 2025 07:36:00 -0000 https://lonza-status.fabriq.tech/incident/549580#c18574b993b42317b532b34beab4373d35d273761a83001ba0cf30ed1cbacbbf The performance issues (including slowness and potential access problems) reported earlier this morning, following our recent system maintenance, have now been fully resolved. Our team has completed the necessary corrective actions, and all services have returned to normal operation. We have confirmed that system performance is stable. We apologize for the disruption this may have caused and thank you for your patience and understanding while we worked to resolve it. https://lonza-status.fabriq.tech/incident/549580 Wed, 23 Apr 2025 05:17:00 -0000 https://lonza-status.fabriq.tech/incident/549580#accbc75af551e8b12a8c1234b3a837edc9a0a586268259db30f459fc862900fd Our monitoring systems have detected performance issues impacting services following overnight maintenance completed early on April 23rd. Users may experience slowness or difficulty accessing the platform. Our engineers are investigating urgently. We apologize for the inconvenience and will share more information shortly. https://lonza-status.fabriq.tech/incident/549580 Wed, 23 Apr 2025 05:17:00 -0000 https://lonza-status.fabriq.tech/incident/549580#accbc75af551e8b12a8c1234b3a837edc9a0a586268259db30f459fc862900fd Our monitoring systems have detected performance issues impacting services following overnight maintenance completed early on April 23rd. Users may experience slowness or difficulty accessing the platform. Our engineers are investigating urgently. We apologize for the inconvenience and will share more information shortly. https://lonza-status.fabriq.tech/ Tue, 01 Apr 2025 12:25:07 +0000 https://lonza-status.fabriq.tech/#a109e461a3ebcc72b7683f66f7dc6b0fb44675a4224c27f00ba14acc1a3e4805 API — Lonza recovered https://lonza-status.fabriq.tech/ Tue, 01 Apr 2025 12:21:06 +0000 https://lonza-status.fabriq.tech/#a109e461a3ebcc72b7683f66f7dc6b0fb44675a4224c27f00ba14acc1a3e4805 API — Lonza went down https://lonza-status.fabriq.tech/ Thu, 27 Mar 2025 14:27:16 +0000 https://lonza-status.fabriq.tech/#91fa714131f435b46748d725252183ada4515dc0fd34f3eb72acff9d2cd1c654 API — Lonza recovered https://lonza-status.fabriq.tech/incident/535497 Thu, 27 Mar 2025 14:27:00 -0000 https://lonza-status.fabriq.tech/incident/535497#c4511d603fd921b2f230262110931eac462193739788821e75656a1737f25413 Incident has been mitigated by doubling the number of server instances. We are carrying an investigation to identify the root cause. https://lonza-status.fabriq.tech/incident/535497 Thu, 27 Mar 2025 14:27:00 -0000 https://lonza-status.fabriq.tech/incident/535497#c4511d603fd921b2f230262110931eac462193739788821e75656a1737f25413 Incident has been mitigated by doubling the number of server instances. We are carrying an investigation to identify the root cause. https://lonza-status.fabriq.tech/ Thu, 27 Mar 2025 14:23:44 +0000 https://lonza-status.fabriq.tech/#91fa714131f435b46748d725252183ada4515dc0fd34f3eb72acff9d2cd1c654 API — Lonza went down https://lonza-status.fabriq.tech/incident/535497 Thu, 27 Mar 2025 14:23:00 -0000 https://lonza-status.fabriq.tech/incident/535497#ce0b0e36f5335c4aa89ce1e7b494ca778bc634e7d89a2ec407b807013267021e Some requests take multiple minutes instead of less than a second. Several of our server instances are unhealthy. We are currently investigating. https://lonza-status.fabriq.tech/incident/535497 Thu, 27 Mar 2025 14:23:00 -0000 https://lonza-status.fabriq.tech/incident/535497#ce0b0e36f5335c4aa89ce1e7b494ca778bc634e7d89a2ec407b807013267021e Some requests take multiple minutes instead of less than a second. Several of our server instances are unhealthy. We are currently investigating. https://lonza-status.fabriq.tech/ Sun, 19 Jan 2025 00:19:14 +0000 https://lonza-status.fabriq.tech/#65c0e48d8a00a890bef181df905f4668c4f04cae332591fba3190061be15adfb API — Lonza recovered https://lonza-status.fabriq.tech/ Sun, 19 Jan 2025 00:15:44 +0000 https://lonza-status.fabriq.tech/#65c0e48d8a00a890bef181df905f4668c4f04cae332591fba3190061be15adfb API — Lonza went down https://lonza-status.fabriq.tech/incident/497474 Fri, 17 Jan 2025 14:58:00 -0000 https://lonza-status.fabriq.tech/incident/497474#875626f07d954bd391d7373fa077d2c5139257f1c288738ed8b6ce879e9cea6a We have confirmed that no users were affected than the issues except our internal Customer Operations user. Therefore, this was a false alarm, and in fact no downtime was experienced. Thank you for your understanding and apologies for the inconvenience. https://lonza-status.fabriq.tech/incident/497474 Fri, 17 Jan 2025 14:58:00 -0000 https://lonza-status.fabriq.tech/incident/497474#875626f07d954bd391d7373fa077d2c5139257f1c288738ed8b6ce879e9cea6a We have confirmed that no users were affected than the issues except our internal Customer Operations user. Therefore, this was a false alarm, and in fact no downtime was experienced. Thank you for your understanding and apologies for the inconvenience. https://lonza-status.fabriq.tech/incident/497474 Fri, 17 Jan 2025 14:45:00 -0000 https://lonza-status.fabriq.tech/incident/497474#128f51d1db4381f2deaece444d9f7c02a45f7c62e97211fb5644934617980aa9 We have identified the root cause of the issue and are currently mitigating it. We can confirm that most users are now able to login and use the application. We are evaluating the impact for other users. https://lonza-status.fabriq.tech/incident/497474 Fri, 17 Jan 2025 14:45:00 -0000 https://lonza-status.fabriq.tech/incident/497474#128f51d1db4381f2deaece444d9f7c02a45f7c62e97211fb5644934617980aa9 We have identified the root cause of the issue and are currently mitigating it. We can confirm that most users are now able to login and use the application. We are evaluating the impact for other users. https://lonza-status.fabriq.tech/incident/497474 Fri, 17 Jan 2025 14:15:00 -0000 https://lonza-status.fabriq.tech/incident/497474#b74d9f3030c0e3cffda9ffa483782298021980eb2c9faedb055b844e0a125be2 We are currently experiencing an authentication issue that prevents the mobile and web applications, as well as our API, from working. Our teams are actively investigating the root cause and working to resolve the issue as quickly as possible. We will keep you updated on the progress. Thank you for your understanding. https://lonza-status.fabriq.tech/incident/497474 Fri, 17 Jan 2025 14:15:00 -0000 https://lonza-status.fabriq.tech/incident/497474#b74d9f3030c0e3cffda9ffa483782298021980eb2c9faedb055b844e0a125be2 We are currently experiencing an authentication issue that prevents the mobile and web applications, as well as our API, from working. Our teams are actively investigating the root cause and working to resolve the issue as quickly as possible. We will keep you updated on the progress. Thank you for your understanding. https://lonza-status.fabriq.tech/incident/419745 Mon, 09 Dec 2024 11:26:00 -0000 https://lonza-status.fabriq.tech/incident/419745#960b80bba13a1e08acf0e4223f286de8d1cd99d0b8fc851b0c238353af776fa9 The incident has been solved. The issue arose after a change in the client's data. This change triggered messaging calls for heavy queries, leading to an overload of the workers. We have mitigated the situation for now and are working on a more systematic fix for later. https://lonza-status.fabriq.tech/incident/419745 Mon, 09 Dec 2024 10:10:00 -0000 https://lonza-status.fabriq.tech/incident/419745#14679f152c14bee17400d2dcaaad5556aa9beb1f6815e70fe01a393083fccbc1 A short unavailability has been experienced during less than 1 minute. Our team is working on finding the root cause of this issue. https://lonza-status.fabriq.tech/ Mon, 09 Dec 2024 10:09:00 +0000 https://lonza-status.fabriq.tech/#ac7f9629370b1c541fd983586e8c11e3b799447767f76908069a41e81b58214b API — Lonza recovered https://lonza-status.fabriq.tech/ Mon, 09 Dec 2024 10:08:41 +0000 https://lonza-status.fabriq.tech/#ac7f9629370b1c541fd983586e8c11e3b799447767f76908069a41e81b58214b API — Lonza went down https://lonza-status.fabriq.tech/incident/419745 Tue, 27 Aug 2024 08:02:00 -0000 https://lonza-status.fabriq.tech/incident/419745#ba58ce40c4b526248412a0cdd3a828d8ceafd2a095ea48898798a6d5b2b9238c We upgraded the number of allocated instances from one to two at 09:59. The API is available again, we are investigating the root causes to https://lonza-status.fabriq.tech/ Tue, 27 Aug 2024 08:01:19 +0000 https://lonza-status.fabriq.tech/#4e0a2ee4434314bedaad3ac3f55d599f7a31b0cdd66573496a480a165b3952e9 API — Lonza recovered https://lonza-status.fabriq.tech/ Tue, 27 Aug 2024 07:58:59 +0000 https://lonza-status.fabriq.tech/#4e0a2ee4434314bedaad3ac3f55d599f7a31b0cdd66573496a480a165b3952e9 API — Lonza went down https://lonza-status.fabriq.tech/incident/419745 Tue, 27 Aug 2024 07:58:00 -0000 https://lonza-status.fabriq.tech/incident/419745#bddafb8b09e8bf3fdb8d7990f06f38e82ea724f7e2f979add352daa7118fdc70 We're experiencing an issue that put the API down again. We are making actions to resolve the issue and investigate. https://lonza-status.fabriq.tech/incident/419745 Tue, 27 Aug 2024 07:54:00 -0000 https://lonza-status.fabriq.tech/incident/419745#aa411f8ffdd266edf7efa7460fa039e8a17ff5fd5192a6a273a92f17a49ebb72 The incident was automatically resolved, we are investigating the root causes. https://lonza-status.fabriq.tech/ Tue, 27 Aug 2024 07:53:06 +0000 https://lonza-status.fabriq.tech/#7c845bb4db09f221c4f845ca371d1d0718065f25f84058fb698b2cfccbcd0cde API — Lonza recovered https://lonza-status.fabriq.tech/ Tue, 27 Aug 2024 07:52:15 +0000 https://lonza-status.fabriq.tech/#7c845bb4db09f221c4f845ca371d1d0718065f25f84058fb698b2cfccbcd0cde API — Lonza went down https://lonza-status.fabriq.tech/incident/419745 Tue, 27 Aug 2024 07:51:00 -0000 https://lonza-status.fabriq.tech/incident/419745#dfdcabfef0040d83e2715292ced10cd71f3243eb5e6655f73550259f04cb5897 We're experiencing an issue that put the API down. We are making actions to resolve the issue and investigate. https://lonza-status.fabriq.tech/incident/417633 Thu, 22 Aug 2024 06:24:00 -0000 https://lonza-status.fabriq.tech/incident/417633#f2047c01881c21a4fdca168421f02a357e8911d0fc826eb06234e205388b9579 The bug regarding Routines has been fixed, but unfortunately the correct data from that period cannot be recovered. We will assist the recovery of our customers' operations in the following way: - Providing a list of the affected Routine Executions that have invalid data. - Offering to remove the affected Routine Executions. - Rebuilding the data on Dynamic KPIs that are based on those Routines. https://lonza-status.fabriq.tech/incident/417633 Tue, 20 Aug 2024 23:04:00 -0000 https://lonza-status.fabriq.tech/incident/417633#6f84c2bf75836a2a7207602fcfc870980e8fef79656048b1d2a940a4863e7150 Due to a regression introduced by a change in the application, from Wednesday 2024-08-21 01:04 PM UTC to Thursday 2024-08-22 8:24 AM UTC, any usage of Routines resulted in invalid data being recorded. This was due to a bug that displayed incorrect Routine Steps when starting, submitting, or editing a Routine Execution; and when editing the Steps of Routines. When used during that time period, Routines were missing some Steps, or had Steps that were displayed or recorded data in an unexpected way. Therefore, the data on those Routine Executions is invalid and cannot be exploited. Dynamic KPIs based on those Routine Executions will also display incorrect data. The issue affected the Webapp, the Mobile Application, and the ability to retrieve data from Routines on our public API. https://lonza-status.fabriq.tech/incident/405427 Mon, 29 Jul 2024 09:59:00 -0000 https://lonza-status.fabriq.tech/incident/405427#0f93e1db8532b6b8214de38f036e3ef0f948d0b4277d892a572b1dac105affe6 The root cause comes from a feature that was supposed to improve the loading time of images and avatars. This feature was intended to enhance the way thumbnails are managed in the legacy code. This improvement has been postponed and will be implemented in the new stack. https://lonza-status.fabriq.tech/incident/405427 Mon, 29 Jul 2024 09:14:00 -0000 https://lonza-status.fabriq.tech/incident/405427#bc2a36096b7b060390e8ac69cc3e5dc848747f3067a8c7181ca9c07980fa50a6 Service on the API is restored. We are investigating the cause of the incident. https://lonza-status.fabriq.tech/ Mon, 29 Jul 2024 08:59:38 +0000 https://lonza-status.fabriq.tech/#5a7fdd0416793fc5e9e86e21317586d8022f6133bd051e96f195fa22d5b00912 API — Lonza recovered https://lonza-status.fabriq.tech/incident/405427 Mon, 29 Jul 2024 08:59:00 -0000 https://lonza-status.fabriq.tech/incident/405427#af66dc75e113b86325e51970f02476d4c0319c1b66683ae57de2dd0ae775e91c We're experiencing an issue that put the API down. We are making actions to resolve the issue and investigate. https://lonza-status.fabriq.tech/ Mon, 29 Jul 2024 08:58:28 +0000 https://lonza-status.fabriq.tech/#5a7fdd0416793fc5e9e86e21317586d8022f6133bd051e96f195fa22d5b00912 API — Lonza went down https://lonza-status.fabriq.tech/incident/404260 Fri, 26 Jul 2024 20:14:00 -0000 https://lonza-status.fabriq.tech/incident/404260#fd2f3bf246a49a43671a1ae7932f664a71339e89a4f96d9f2fb98ad43921a2ad We are found the root cause which a specific endpoint. We will work on this endpoint to prevent future slowdowns. https://lonza-status.fabriq.tech/incident/404260 Fri, 26 Jul 2024 19:59:00 -0000 https://lonza-status.fabriq.tech/incident/404260#a7fcaad6238954806639215d647996cd0d9ea53ff2e9dc47752d050117db148f The API was very slow for a short period of time, causing a congestion. We are investigating the root cause. https://lonza-status.fabriq.tech/ Fri, 26 Jul 2024 19:42:33 +0000 https://lonza-status.fabriq.tech/#bf78c8dac930e200ecf4d2a20b2ea428f389f243ee10efc11d890d0e968633de API — Lonza recovered https://lonza-status.fabriq.tech/ Fri, 26 Jul 2024 19:40:29 +0000 https://lonza-status.fabriq.tech/#bf78c8dac930e200ecf4d2a20b2ea428f389f243ee10efc11d890d0e968633de API — Lonza went down