The backlock of schema coordinate (object field, input object field, and enum value) metrics has now cleared and there is no more metric ingestion delay. The metrics for input object fields, enum values, and the operation and client breakdown of object fields is inaccurate for June 27 4-5am and 8-9pm, and missing between 5am to 8pm (all times in UTC). All other metrics were not impacted.
Resolved
The backlock of schema coordinate (object field, input object field, and enum value) metrics has now cleared and there is no more metric ingestion delay. The metrics for input object fields, enum values, and the operation and client breakdown of object fields is inaccurate for June 27 4-5am and 8-9pm, and missing between 5am to 8pm (all times in UTC). All other metrics were not impacted.
Monitoring
The backlog of object field, input object field, and enum value metrics are now being processed but we expect the backlog to take 8-10 hours to be fully processed. For a period of time between 4 am and 9pm on June 27 UTC we expect there to be data loss for input object field and enum value metrics, as well as the client and operation breakdown of object field metrics. No other metrics (operation, trace, etc.) are impacted.
Monitoring
Field and coordinate usage metric ingestion is functional and lag is trending downwards. We estimate 8-10 hours for full recovery.
Investigating
We are investigating suspected data loss for field and coordinate usage metrics for all customers.
