All systems have been stable for over 24 hours. The firewall configuration issue that prevented the automatic failover of services from our Midwest datacenter to our two redundant datacenters has been corrected. At this time we consider this incident fully resolved.
Posted Aug 29, 2018 - 15:27 PDT
Update
We have more information regarding the incident this morning. Inbound calls to a subset of numbers homed to our Midwest datacenter returned busy signals from about 11:00am CDT to 11:45am CDT due to a component malfunction in the Midwest DC. Calls should have failed over automatically to our East and West Coast datacenters, but due to a firewall configuration issue, that failover traffic was rejected by both DCs. Inbound calls to East and West Coast numbers were not affected, and all outbound traffic was unaffected. We are working to ensure that failover functions as expected in the future, and we will continue to monitor the Midwest DC.
Posted Aug 28, 2018 - 16:20 PDT
Monitoring
Reports of issues have stopped, and we believe that the inbound busy issue is resolved. Our engineering team is still working to determine the root cause. We will continue to monitor this issue, and you will receive a Reason for Outage notification when our investigation is complete.
Posted Aug 28, 2018 - 10:53 PDT
Update
At this time it appears that calls are ringing in to all customers normally. We have not yet found the root cause of this incident. It appears to have been isolated to inbound calls and related to a single carrier. We are still actively gathering information to ensure that we have everything restored to full service.
Posted Aug 28, 2018 - 10:02 PDT
Investigating
We are currently experiencing an issue in which some users are unable to receive incoming calls. Our engineering team is working to isolate the cause of the issue as well as the scope. We will update you as soon as we have more information.