On Sunday, Sept 17th, our monitoring system showed a massive increase in outbound call failures starting at around 12:40pm CDT. Upon investigation, our engineers discovered an issue with one of our redundant carriers used for outbound call delivery. At the time, this carrier was our primary route for outbound calls. This primary route was responding to each call attempt with a failure code that should have resulted in a retry over one of our redundant routes, which would not have been subject to the same issue our experienced by our primary route, and should have resulted in transparent delivery of the call for the SimpleVoIP user. 

However, our system treated the error codes from our primary route as hard failure codes, terminating the call entirely rather than attempting redelivery over a backup route. Once our engineers identified that automatic failover was not occurring, we manually deprioritized the affected route, moving a redundant carrier route to the primary position, and fully restored outbound service at around 1:40pm CDT.

The underlying issue with our primary carrier was addressed, and the manual routing change was reverted Monday morning, Sept 18th. 

Our engineers are still investigating why we failed to reattempt these calls over our redundant routes. We have confirmed that calls that are undeliverable via our primary carrier are being reattempted via our redundant routes as expected. We have also identified opportunities for monitoring improvements that should reduce our response times and allow us to engage Engineering more directly if similar calling patterns are observed in the future.