Gladia - Notice history

This incident has been resolved, and latencies are now back to normal ⚡

One of our cloud provider load balancer had issues resulting in only a part of the system to be available, resulting in lower ressources available for a high number of queries in parallels.

We're in touch with our cloud provider to make sure that this doesn't happen or is mitigated and fix as soon as possible if it does.

We implemented a fix and are currently monitoring the result.

Latencies should be slower than usual, we're still working on it.

This incident has been resolved, and latencies should be back to normal for the last 2 hours now.

It seems that the root cause is one of our provider having issues with bandwith limitations, resulting in some large audio struggle to get passed between our services. We're still investigating to make sure it doesn't happen and we're putting mitigation in place for this.

If some of your request have failed due to timeout issues, those have been automatically replayed.

This incident has been resolved.

One of our internal services had a fallback on the wrong database resulting in important queries to be ran on this one.
This long query resulted in a high CPU pressure, which delayed some of the DB queries that ran during few minutes.

The internal service which caused the issue have been disabled for the moment and will be resumed during next Monday with a patch ensuring it never reach one of our production DB.

No data loss nor noticeable high latency should have occurred during the incident.

Thanks for your understanding.

This incident has been resolved, we're currently investigating to understand why some users couldn't reach the API during few minutes.

This incident has been resolved, it was caused by a cache issue resulting in wrong image pulled for one of our services.

This incident has been resolved.

Post-mortem will be shared here soon.

We've fixed the issue. We are actively monitoring the recovery.