Resolved
This incident has been resolved.
Monitoring
On March 15th, at 9:01 AM Pacific, Apollo performed maintenance on our AWS Uplink configuration delivery system. The change was to migrate our internal loadbalancers to best practice Edge optimized routing.
In doing so, we removed a route used by default Routers, thus making their fallback to AWS fail with a 4XX.
From our observations of our systems, this impacted some of our traffic but with our knowledge of our rates of success/failure with our GCP Uplink, we do not believe any Routers running in our customer's systems were critically impacted.
No Apollo Gateways were impacted by this change at all.
On March 23rd, at around 2:30 PM Pacific, we patched this issue and confirmed seeing our 4XX error rate drop back to almost 0%.
We apologize for any inconvenience this caused by this escaped bug, and will be working in the coming days on our postmortem. As always, we appreciate your patience!