Our US-West cluster has been stable since the failed server was replaced. We are still investigating the reason that a significant subset of phones didn't automatically fail over to healthy servers. More information will be made available via the post mortem notes on this incident report. This incident is resolved.
Posted May 03, 2021 - 01:08 PDT
Monitoring
Our US-West zone is fully operational once again. We are moving all traffic back to US-West and will monitor closely to ensure that services remain stable.
Posted May 02, 2021 - 20:20 PDT
Identified
We have received reports of unregistered phones on our US-West cluster. One of the servers in this cluster had a hardware failure that is preventing phones from registering. We have initiated a route change to divert these registrations to US-Central, and the affected US-West server is being rebuilt. We expect that all impacted US-West phones should register to US-Central within the next 5 minutes.