We're seeing things as stable, though our data servers are still rebalancing, we aren't seeing negative impacts across our stack. We have verified that all data sent to us over the past hours are fully available in our UI, and especially via our api.
We know that these kinds of outages have impacts beyond just our organization, and we are truly grateful for the patience shown as we continue to improve and scale up Studio to meet ever higher demands and need. Thanks as always folks!
Posted Sep 17, 2021 - 00:03 UTC
Update
The experimental performance alerts feature is now operational. We will continue to monitor for 15 minutes before moving this incident to resolved. Thank you for your patience.
Posted Sep 16, 2021 - 23:46 UTC
Monitoring
We have fully enabled our system and are seeing our normal resilience and reporting rates as stable. Our experimental production-alerts feature has been reenabled and we are watching how this impacts the entirety of our system. We will update again in about 15 minutes as to the health of this incident. As always, we appreciate your continued patience.
Posted Sep 16, 2021 - 23:33 UTC
Update
We have identified the problem with our backend data server. We are going to bring all our services back online over the next few hours. We appreciate your patience.
Posted Sep 16, 2021 - 23:07 UTC
Identified
We are seeing an elevated level of success in our services. We are going to continue monitoring for now as we start to bring back online the rest of our services.
Posted Sep 16, 2021 - 21:37 UTC
Update
We have implemented a fix to bring some more capacity online for our historical data servers. We're waiting to see the impact of this before determining next steps. As always, your patience is appreciated.
Posted Sep 16, 2021 - 21:21 UTC
Update
The experimental performance alerts feature is currently unavailable.
Posted Sep 16, 2021 - 20:19 UTC
Update
We are continuing to investigate this issue.
Posted Sep 16, 2021 - 20:08 UTC
Update
The impacted features include checks which are experiencing errors, and publishes which have limited operations with static diffs.
Posted Sep 16, 2021 - 18:43 UTC
Update
We are continuing to investigate this issue.
Posted Sep 16, 2021 - 18:00 UTC
Investigating
We are currently investigating this issue
Posted Sep 16, 2021 - 17:58 UTC
This incident affected: GraphQL API and Studio UI.