Field stats data is partially available
Incident Report for Apollo
Resolved
This incident is, at long last, resolved.

We have confirmed that our improvements for metrics ingestion (ie, our prioritization lanes for certain types of critical feature data) are all healthy and working as expected. We have confirmed that our Fields page data is back up and running with no reported issues. We have also confirmed that some backend changes to alleviate strain on certain portions of our application are working as expected.

We truly appreciate everyone's patience in this as we continue to work on scaling to meet your needs.
Posted Sep 02, 2021 - 12:29 PDT
Monitoring
We have been able to confirm that all ingestion data is being persisted, and will be coming back online. Our Checks features are completely available. Our fields page is loading, but for some customers portions of request rate data will be unavailable. Recent data is completely available.

We have verified that our prioritized task lanes are working as expected.

As always, we appreciate your patience.
Posted Sep 02, 2021 - 08:01 PDT
Update
We are still working on reloading historical data. We have prioritized tasks pertaining to these sources and are waiting to see them pick up new work.

As always, we appreciate your patience.
Posted Sep 02, 2021 - 07:21 PDT
Update
At this time we are seeing success when querying for recent data for the Fields page. We are fully up on Checks and other trace related data. We are actively working to reload portions of our Fields data which is presently unavailable.
Posted Sep 02, 2021 - 06:31 PDT
Identified
We are investigating the issue and have begun to identify the root cause of the unavailability of field stats data. We are confident that no data is permanently lost from ingestion, and we are working hard to restart the processing pipeline to make more recent field usage data available. Currently, the effects of this disruption are limited to features that rely on this data (fields page in Studio, VS Code timing estimations, Explorer timing estimations)
Posted Sep 02, 2021 - 04:42 PDT
Investigating
Recent field level metrics are unavailable. When querying the api this will show up as no data for things on the fields page or when using field timings hints in the vscode extension or explorer.

This should not effect schema publications or schema checks or other operation level metric information shown in studio.
Posted Sep 02, 2021 - 01:03 PDT
This incident affected: Metrics Ingestion.