Our GraphQL API has not experienced latency anomalies for the past 2 hours, and we believe the incident to be resolved.
The effect of this incident is that from 9:30AM PT to 9:00PM PT, our GraphQL API experienced high latency, leading to timeouts in several components of our systems. This includes Studio, Rover CLI commands, and schema reporting. The root cause of this outage has been identified as a combination of database resource exhaustion and internal scaling issues.
We have collected initial notes during this incident that already point to improvements we can make on our side, and we will be conducting a thorough postmortem to ensure that such an incident does not occur again. Thank you to everyone for your patience and support. We are committed to meeting high standards of reliability and excellence and using the learnings from this incident to improve.
Posted Oct 30, 2022 - 06:06 UTC
Monitoring
We have identified the causes of the latency, and have implemented fixes. We will continue to monitor our GraphQL API latency for anomalies.
Posted Oct 30, 2022 - 04:27 UTC
Investigating
Our GraphQL API is experiencing increased latency, and we are currently investigating the cause. You may experience Studio UI fail to load as a result, in addition to high latency when running Rover commands that connect to Studio.
Posted Oct 29, 2022 - 20:45 UTC
This incident affected: GraphQL API and Studio UI.