Gladia - API unstabilities – Incident details

All systems operational

API unstabilities

Resolved
Degraded performance
Started 7 days agoLasted about 4 hours

Affected

API

Degraded performance from 6:05 PM to 6:28 PM, Operational from 6:28 PM to 7:02 PM, Degraded performance from 7:02 PM to 7:26 PM, Operational from 7:26 PM to 9:49 PM

Pre-Recorded v2

Degraded performance from 6:05 PM to 6:28 PM, Operational from 6:28 PM to 9:49 PM

Real-Time v1

Degraded performance from 6:05 PM to 6:28 PM, Operational from 6:28 PM to 9:49 PM

Real-Time v2

Degraded performance from 6:05 PM to 6:28 PM, Operational from 6:28 PM to 9:49 PM

App

Degraded performance from 6:05 PM to 6:28 PM, Operational from 6:28 PM to 9:49 PM

Addons

Degraded performance from 6:05 PM to 6:28 PM, Operational from 6:28 PM to 9:49 PM

Updates
  • Resolved
    Resolved

    This incident has been resolved.

    One of our internal services had a fallback on the wrong database resulting in important queries to be ran on this one.
    This long query resulted in a high CPU pressure, which delayed some of the DB queries that ran during few minutes.

    The internal service which caused the issue have been disabled for the moment and will be resumed during next Monday with a patch ensuring it never reach one of our production DB.

    No data loss nor noticeable high latency should have occurred during the incident.

    Thanks for your understanding.

  • Monitoring
    Monitoring
    We implemented a fix and are currently monitoring the result.
  • Investigating
    Investigating
    We are currently investigating this incident.
  • Resolved
    Resolved

    This incident has been resolved, we're currently investigating to understand why some users couldn't reach the API during few minutes.

  • Monitoring
    Monitoring
    We implemented a fix and are currently monitoring the result.
  • Identified
    Identified
    We are continuing to work on a fix for this incident.