Gladia - API unreachable – Incident details

All systems operational

API unreachable

Resolved
Partial outage 0
Started 10 days agoLasted about 3 hours

Affected

API

Major outage from 6:58 AM to 7:04 AM, Operational from 7:04 AM to 8:19 AM, Partial outage from 8:19 AM to 9:18 AM, Operational from 9:18 AM to 9:57 AM

Pre-Recorded v2

Major outage from 6:58 AM to 7:04 AM, Operational from 7:04 AM to 8:19 AM, Partial outage from 8:19 AM to 9:18 AM, Operational from 9:18 AM to 9:57 AM

Real-Time v1

Major outage from 6:58 AM to 7:04 AM, Operational from 7:04 AM to 9:57 AM

Real-Time v2

Major outage from 6:58 AM to 7:04 AM, Operational from 7:04 AM to 8:19 AM, Partial outage from 8:19 AM to 9:18 AM, Operational from 9:18 AM to 9:57 AM

App

Major outage from 6:58 AM to 7:04 AM, Operational from 7:04 AM to 9:57 AM

Addons

Major outage from 6:58 AM to 7:04 AM, Operational from 7:04 AM to 9:57 AM

Updates
  • Postmortem
    Postmortem

    We had a regression introduced in our file downloading service resulting in larger file to not be removed from our storage. In the same-time , a specific user sent unsupported larger files resulting in our storage being filled way faster than usual.

    This impacted files upload services and increased our services CPU usage and pressure over multiples services including Solaria & Pre-recorded resulting in multiples issues like:

    - Some request returning 404 codes
    - File upload failures Session opening failures on Solaria

    We fixed the introduced regression to make sur large files are removed as it should be.

  • Resolved
    Resolved

    This incident has been resolved.

    A post-mortem will be shared soon.

  • Monitoring
    Monitoring
    We implemented a fix and are currently monitoring the result.
  • Investigating
    Investigating
    We are currently investigating this incident.
  • Monitoring
    Monitoring
    We implemented a fix and are currently monitoring the result.
  • Investigating
    Investigating
    We are currently investigating this incident.