Error rates are elevated.
Incident Report for Apollo
Resolved
This incident has been resolved. Thank you for your patience.
Posted Sep 21, 2021 - 14:10 PDT
Update
Our Experimental Performance alerts feature is now back online. We are working on fully restoring the last of our beta features now.
Posted Sep 21, 2021 - 12:53 PDT
Update
Our E-Mail delivery is back to healthy. We are now working on turning back on our various beta/experimental features.
Posted Sep 21, 2021 - 12:30 PDT
Update
Our GraphQL API is back within our regular bounds. We are still working on bringing up our experimental features and E-Mail.
Posted Sep 21, 2021 - 11:38 PDT
Update
We are seeing success in our Metrics Ingestion and Frontend Web Applications. We are continuing to monitor our GraphQL API as we roll out more changes there to further reduce traffic and load. E-mail's are still disabled, but we are now actively working on restoring this functionality.
Posted Sep 21, 2021 - 11:16 PDT
Monitoring
Our changes are now in production and we are monitoring.
Posted Sep 21, 2021 - 11:02 PDT
Identified
We have identified a process which has caused high load on our ingestion servers and managed to cascade its impact across our fleet. We previously thought we had cordoned it such that this would no longer be a problem, however we found a few more code paths which were only exhibited over the past 24 hours.

We are rolling out this patch to our fleet and hope to have more telemetry here in about 20 minutes.
Posted Sep 21, 2021 - 10:42 PDT
Investigating
We have seen elevated levels of errors in our ingestion and API. We are starting back from root causes and working to assess how the changes we made earlier are not handling this current increase in load.
Posted Sep 21, 2021 - 10:30 PDT
Monitoring
We are seeing our health as generally available. We are now working to bring back online our email functionality, our experimental performance alerts, and our beta launches features.
Posted Sep 21, 2021 - 09:00 PDT
Investigating
We are currently investigating this issue.
Posted Sep 21, 2021 - 05:54 PDT
This incident affected: Metrics Ingestion, GraphQL API, Notifications, and Studio UI.