We received reports of errors and slow response times within our web portal on Thursday, March 23. We quickly isolated these web portal issues to a backend API being handled primarily servers in our US-West datacenter. We redirected that API traffic to US-East in an attempt to mitigate the issues. Unfortunately, we started to receive reports that call traffic was being impacted as well.  

After isolating these call reports to our US-West datacenter, our engineering team was able to determine that our US-West application servers, which handle web traffic and interface with our SIP devices, had entered a bad state that limited the amount of traffic they could process. We were able to restart the affected services and restore voice and web traffic to 100%. 

In looking into the underlying cause, we found that the network connection between our application servers and the virtual host running one of our database servers was degraded during the initial outage period and intermittently afterward. This slowed response times for database lookups involved in both API handling and call routing, resulting in the observed behavior, and eventually resulting in the faulty service state on our application servers. 

We apologize for the frustration caused by these disruptions to both your customers and your end users. Any downtime is bad for business, and minimizing the impact of any interruptions in service is our highest priority. Since these issues occurred, we have begun work to migrate our database servers from their current virtual hosts to new physical hosts that will minimize the points of failure between the DB servers and the rest of our platform stack, in addition to providing improved performance. You should receive maintenance notifications for these migrations over the next several weeks.

Thank you for your patience as we continue to improve our systems and processes to best serve you and your customers. As always, we welcome any feedback, which may be submitted to our Support team or to your Account Manager.