We have completed running through the backlog of outstanding check requests and all systems have returned to operational. We will continue to monitor for 30 minutes and resolve the incident.
Posted Mar 28, 2024 - 15:17 UTC
Update
We are continuing to work through our checks backlog. Presently, based on our burn rate through our queue, we are estimating less than 15 minutes till we are entirely caught back up. We will update here again in 15 minutes, or once our queue is through.
Posted Mar 28, 2024 - 14:52 UTC
Update
Publishes have recovered, still working through the backlog of check requests
Posted Mar 28, 2024 - 14:24 UTC
Update
We're making progress on our backlog of builds. ETA 15-30 minutes
Posted Mar 28, 2024 - 14:12 UTC
Monitoring
Fix has been shipped. We are beginning to make progress against our backlog of publish requests. Will provide an ETA for full recovery when available.
Posted Mar 28, 2024 - 13:48 UTC
Update
We are still in the process of deploying the code fixes. Publishes and Checks continue to be degraded
Posted Mar 28, 2024 - 13:35 UTC
Update
We have identified another scaling issue in our build systems, and are deploying a fix now. Publishes and Checks continue to be degraded
Posted Mar 28, 2024 - 12:44 UTC
Update
Our systems are working through a backlog of build requests. Publishing and Checks continue to be degraded
Posted Mar 28, 2024 - 12:04 UTC
Update
We continue to monitor recovery to most systems. Publishing and checks are still degraded.
Posted Mar 28, 2024 - 11:44 UTC
Update
We have begun seeing recovery to most systems. Publishing and checks are still degraded
Posted Mar 28, 2024 - 10:45 UTC
Identified
We believe to have root caused our availability issues and are working on a fix
Posted Mar 28, 2024 - 10:23 UTC
Investigating
We are currently investigating this issue.
Posted Mar 28, 2024 - 09:59 UTC
This incident affected: Schema Publishing, Schema Checks, GraphQL API, and Cloud Routing.