SimpleVoIP encountered a database indexing issue on Monday morning at around 10:30am CST that caused some frequently-running queries to slow down, resulting in a traffic jam-like climb in waiting queries. After hitting the database connection limit, our other services were no longer able to connect in a timely fashion, which caused significant failures to our web services, including some services involved in routing calls for a subset of our customers. We resolved this by fixing the underlying indexing issue and clearing the list of backlogged queries, which brought the connection count and our response times back to normal levels at around 11:15am CST.

This specific issue was resolved for good thanks to the change applied as part of the fix. Longer term, our engineers are continuing work on an ongoing project to migrate as many of our database operations as possible to redundant replicas to reduce load on the primary database and prevent similar issues from occurring in the future, both by preventing the lockup of an individual database and by providing redundancy for our critical services in the event that a database does experience issues.

We apologize for the disruption that this caused to you and to your customers. Keeping your call services stable is our highest priority, and we thank you for your patience as we continue to improve the stability of our platform.