Resolved
We have seen all stats return to normal. We are closing this incident while we continue to bolster our infrastructure for this particular issue in the future.
Monitoring
We have scaled our lagging internal service and are now seeing regular operational status. We will continue to monitor things and work to address the root issue out of band. Thanks for your patience through all of this.
Investigating
We are continuing to investigate this issue. We have limited this down to two internal services, and are deploying additional monitoring to one to get us an update on what's going on.
Investigating
Since 11:25 UTC we have been seeing increased rates of 5XX in our GraphQL API. This is effecting downstream components minorly. The increase is from loosely 0, to under 1.7% at most. This is not impacting our ability to ingest metrics, and appears to be limited to a subset of our resolvers. As we learn more we will update here. As always, thank you for your patience!