We received reports of inbound and outbound call failures as well as errors and slow response times within our web portal on Wednesday, June 14. These reports spanned all three of our availability zones.  Our engineering team was able to determine that our US-West application servers, which handle web traffic and interface with our SIP devices, were having difficulty reaching their preferred database server and had entered a bad state. This limited the amount of traffic they could process. Due to the trouble connecting to this database server, we had some difficulty in restarting its services or removing it from our global datasource pool, which delayed recovery of our services. Once we fixed connectivity to the database server, we were able to restart the affected services on our application servers and restore voice and web traffic to 100%. 

This is not the first time we have encountered application state issues due to a faulty connection to our US-West database servers. In fact, we have been working to migrate our database servers in US-West to new hardware that will also minimize the number of network hops to our application servers. That migration is scheduled for Monday, June 26th, and you will be receiving maintenance window notifications shortly. 

We are also implementing additional monitoring to allow us to catch similar issues as they occur so that our engineers may be engaged before our Support team starts to receive customer calls. 

Thank you for your patience as we continue to improve our systems and processes to best serve you and your customers. As always, we welcome any feedback, which may be submitted to our Support team or to your Account Manager.