Resolved
This incident has been resolved
Monitoring
We have completed running through the backlog of outstanding check requests and all systems have returned to operational. We will continue to monitor for 30 minutes and resolve the incident.
Monitoring
We are continuing to work through our checks backlog. Presently, based on our burn rate through our queue, we are estimating less than 15 minutes till we are entirely caught back up. We will update here again in 15 minutes, or once our queue is through.
Monitoring
Publishes have recovered, still working through the backlog of check requests
Monitoring
We're making progress on our backlog of builds. ETA 15-30 minutes
Monitoring
Fix has been shipped. We are beginning to make progress against our backlog of publish requests. Will provide an ETA for full recovery when available.
Identified
We are still in the process of deploying the code fixes. Publishes and Checks continue to be degraded
Identified
We have identified another scaling issue in our build systems, and are deploying a fix now. Publishes and Checks continue to be degraded
Identified
Our systems are working through a backlog of build requests. Publishing and Checks continue to be degraded
Identified
We continue to monitor recovery to most systems. Publishing and checks are still degraded.
Identified
We have begun seeing recovery to most systems. Publishing and checks are still degraded
Identified
We believe to have root caused our availability issues and are working on a fix
Investigating
We are currently investigating this issue.