Singularity - AWS outage causing API failures – Incident details

All systems operational

AWS outage causing API failures

Resolved
Major outage
Started almost 3 years agoLasted about 11 hours

Affected

Carbonara App

Major outage from 5:00 PM to 3:50 AM

Singularity API

Major outage from 5:00 PM to 3:50 AM

Updates
  • Resolved
    Resolved

    API performance appears to have stabilized. We will continue monitoring to ensure there are no lingering effects of the outage.

  • Update
    Update

    Things are slowly returning to normal, but we're still seeing errors on ~25% of API requests. We expect our services to recover at the same pace as AWS recovers.

  • Update
    Update

    AWS reports they have deployed a fix, and we are seeing some improvement in our systems. However, our public-facing API Gateway is still severely degraded, and API requests will continue to fail until that service returns to normal operation.

  • Update
    Update

    The AWS outage continues, and the majority of API calls will continue you fail. We are still monitoring the situation. Our data ingestion infrastructure is not affected by the outage, so we will have up-to-date data available as soon as the API returns to normal operation.

  • Monitoring
    Monitoring

    Most API requests are currently failing with 5XX errors. This is caused by an AWS outage. We are monitoring the situation and actively investigating mitigation options.