Services have been up and stable for over 24 hours at this point. We discovered a firewall issue in our US-West datacenter that slowed down its ability to handle inbound traffic. When our other redundant clusters attempted to send data such as user configurations or call records to US-West to keep the clusters in sync, the slowdown in US-West caused a backlog on those redundant clusters as well. This eventually led to failures across all 3 zones until the links between US-West and the other zones were severed.
We have identified the configuration issue that resulted in the US-West slowdown and rectified it. We kept that cluster out of active service most of the day yesterday to ensure that everything was healthy, and we have been running at full capacity across all 3 zones since last night. Longer term, we will be investigating the backlog sync issue to ensure that a single cluster slowdown does not impact the other zones.
Posted Nov 13, 2018 - 12:04 PST
Monitoring
This issue has been resolved. Calls should be completing normally at this time. We will be able to provide a more detailed description of the issue and the steps taken to resolve it after further review.
Posted Nov 12, 2018 - 07:38 PST
Identified
We have identified an issue causing call completion problems for a large subset of customers. We are currently working to resolve this issue and will have an update shortly.