Historical data has been fully reloaded. Current data has been stable for about the past 6-7 hours. We will be conducting a post mortem on this this in the coming days. As always, your patience is greatly appreciated.
Posted Aug 17, 2021 - 11:57 UTC
Update
We are continuing to see progress in historical data being loaded, and expect to have all data loaded by tomorrow. We shall give an update on progress tomorrow around 9AM PDT. We thank you for your patience and understanding during this incident.
Posted Aug 17, 2021 - 05:51 UTC
Update
Most historical data sources have returned to full availability, although some will take a few more hours to return to full availability. Most Studio actions (such as publishes and checks) and dashboards should be functional.
Posted Aug 17, 2021 - 05:05 UTC
Monitoring
Our historical data servers are reloading data, and they appear to be stable. You should notice data availability returning over the course of the next few hours.
Posted Aug 17, 2021 - 04:34 UTC
Update
We have recreated all our historical data servers and we are starting to reload the data. They appear to be stable.
Posted Aug 17, 2021 - 04:09 UTC
Update
We are over halfway through decommissioning and recreating historical data servers, and the recreated historical data servers have been stable.
Posted Aug 17, 2021 - 03:32 UTC
Identified
We have identified and decommissioned faulty historical data servers, and are waiting for them to be recreated. Note that this will not incur historical data loss.
Posted Aug 17, 2021 - 02:54 UTC
Investigating
Our metrics historical data may be slow or unavailable. Ingestion is not impacted, but checks and metrics dashboards will be severely impacted/may not load.
Posted Aug 17, 2021 - 02:18 UTC
Monitoring
A fix has been implemented and we are monitoring the results.
Posted Aug 17, 2021 - 02:00 UTC
Update
We are continuing to investigate the root cause of the delays, though problems persist and may continue to be experienced across a variety of Apollo Studio features.
Posted Aug 17, 2021 - 00:32 UTC
Investigating
Our metrics historical data may be slow or unavailable. Ingestion is not impacted, but checks and metrics dashboards will be severely impacted/may not load.