We have seen all stats return to normal. We are closing this incident while we continue to bolster our infrastructure for this particular issue in the future.
Posted Dec 08, 2020 - 19:58 UTC
Monitoring
We have scaled our lagging internal service and are now seeing regular operational status. We will continue to monitor things and work to address the root issue out of band. Thanks for your patience through all of this.
Posted Dec 08, 2020 - 17:27 UTC
Update
We are continuing to investigate this issue. We have limited this down to two internal services, and are deploying additional monitoring to one to get us an update on what's going on.
Posted Dec 08, 2020 - 16:59 UTC
Investigating
Since 11:25 UTC we have been seeing increased rates of 5XX in our GraphQL API. This is effecting downstream components minorly. The increase is from loosely 0, to under 1.7% at most. This is not impacting our ability to ingest metrics, and appears to be limited to a subset of our resolvers. As we learn more we will update here. As always, thank you for your patience!
Posted Dec 08, 2020 - 16:40 UTC
This incident affected: GraphQL API and Studio UI.