Resolved
We've seen our ingestion reporting stable operational aspects for the last little while. As such, we're closing this out. Thanks for the patience folks!
Monitoring
We have identified the issue, and our metrics reporting endpoint should be functional at this time. Metrics and trace data may have been lost from 6:45AM-7AM PT. We are currently monitoring for further issues.
Investigating
Our reporting endpoint is completely down and returning 5XXs at this time. There is likely metrics loss, and we are currently investigating.
It's possible your Apollo Server or Apollo Gateway instances may have increased memory usage at this time due to buffering metrics and trace data. If this is causing your fleet to run out of memory/crash, we recommend disabling the usage reporting plugin at this time. You can find guidance on this at https://www.apollographql.com/docs/apollo-server/v3/api/plugin/usage-reporting#disabling-the-plugin