Studio API Availability Impaired

Incident Report for Apollo Graph, Inc.

Resolved

This incident has been resolved
Posted Mar 28, 2024 - 15:36 UTC

Update

We have completed running through the backlog of outstanding check requests and all systems have returned to operational. We will continue to monitor for 30 minutes and resolve the incident.
Posted Mar 28, 2024 - 15:17 UTC

Update

We are continuing to work through our checks backlog. Presently, based on our burn rate through our queue, we are estimating less than 15 minutes till we are entirely caught back up. We will update here again in 15 minutes, or once our queue is through.
Posted Mar 28, 2024 - 14:52 UTC

Update

Publishes have recovered, still working through the backlog of check requests
Posted Mar 28, 2024 - 14:24 UTC

Update

We're making progress on our backlog of builds. ETA 15-30 minutes
Posted Mar 28, 2024 - 14:12 UTC

Monitoring

Fix has been shipped. We are beginning to make progress against our backlog of publish requests. Will provide an ETA for full recovery when available.
Posted Mar 28, 2024 - 13:48 UTC

Update

We are still in the process of deploying the code fixes. Publishes and Checks continue to be degraded
Posted Mar 28, 2024 - 13:35 UTC

Update

We have identified another scaling issue in our build systems, and are deploying a fix now. Publishes and Checks continue to be degraded
Posted Mar 28, 2024 - 12:44 UTC

Update

Our systems are working through a backlog of build requests. Publishing and Checks continue to be degraded
Posted Mar 28, 2024 - 12:04 UTC

Update

We continue to monitor recovery to most systems. Publishing and checks are still degraded.
Posted Mar 28, 2024 - 11:44 UTC

Update

We have begun seeing recovery to most systems. Publishing and checks are still degraded
Posted Mar 28, 2024 - 10:45 UTC

Identified

We believe to have root caused our availability issues and are working on a fix
Posted Mar 28, 2024 - 10:23 UTC

Investigating

We are currently investigating this issue.
Posted Mar 28, 2024 - 09:59 UTC
This incident affected: Schema Publishing, Schema Checks, GraphQL API, and Cloud Routing.