SV Portal and Call Routing
Incident Report for SimpleVoIP LLC

SimpleVoIP encountered a database indexing issue on Monday morning at around 10:30am CST that caused some frequently-running queries to slow down, resulting in a traffic jam-like climb in waiting queries. After hitting the database connection limit, our other services were no longer able to connect in a timely fashion, which caused significant failures to our web services, including some services involved in routing calls for a subset of our customers. We resolved this by fixing the underlying indexing issue and clearing the list of backlogged queries, which brought the connection count and our response times back to normal levels at around 11:15am CST.

This specific issue was resolved for good thanks to the change applied as part of the fix. Longer term, our engineers are continuing work on an ongoing project to migrate as many of our database operations as possible to redundant replicas to reduce load on the primary database and prevent similar issues from occurring in the future, both by preventing the lockup of an individual database and by providing redundancy for our critical services in the event that a database does experience issues.

We apologize for the disruption that this caused to you and to your customers. Keeping your call services stable is our highest priority, and we thank you for your patience as we continue to improve the stability of our platform.

Posted Nov 19, 2020 - 10:09 PST

All services are now stable and have remained so for several hours. A formal RFO will be available through this status page within the next couple of days following some internal investigation. We apologize for any disruption this caused you, your users, or your customers.
Posted Nov 16, 2020 - 13:22 PST
Services have been restored. We are keeping a close eye on this to ensure it remains stable.
Posted Nov 16, 2020 - 09:33 PST
We are still working to get these services up and running again as quickly as possible.
Posted Nov 16, 2020 - 09:28 PST
We have identified an issue impacting the SimpleVoIP web portal that is also disrupting a portion of our phone traffic. Our engineering team is working to resolve this now.
Posted Nov 16, 2020 - 09:14 PST
This incident affected: SimpleVoIP Hosted PBX and SimpleVoIP Admin Portal.