tag:status.simplevoip.us,2005:/historySimpleVoIP LLC Status - Incident History2024-03-28T22:23:37-07:00SimpleVoIP LLCtag:status.simplevoip.us,2005:Incident/203809112024-04-01T23:00:00-07:002024-03-27T11:49:26-07:00Enable New Server IPs<p><strong>THIS IS A SCHEDULED EVENT Apr <var data-var='date'>1</var>, <var data-var='time'>23:00</var> PDT - Apr <var data-var='date'>2</var>, <var data-var='time'>01:00</var> PDT</strong></p><p><small>Mar <var data-var='date'>27</var>, <var data-var='time'>11:49</var> PDT</small><br><strong>Scheduled</strong> - SimpleVoIP is introducing new IP addresses for our VoIP gateway servers, as outlined in the advance notifications sent over the past month. Customer firewalls must be configured to allow SIP and RTP traffic to and from our new IP ranges to avoid issues preventing phone registration or interfering with delivery of voice traffic. Please see our updated Wiki at https://support.simplevoip.com/hc/en-us/articles/360046002753-SimpleVoIP-Firewall-Setup for details. The 161.38.* ranges will be added to active routing during this maintenance window.<br /><br />If you have any questions or concerns, please reach out to our support team at support@simplevoip.com.</p>tag:status.simplevoip.us,2005:Incident/202947012024-03-18T22:21:00-07:002024-03-18T22:21:00-07:00Unregistered Phones | Texas | Possible Comcast Outage<p><small>Mar <var data-var='date'>18</var>, <var data-var='time'>22:21</var> PDT</small><br><strong>Resolved</strong> - After some time, it appears all connections from Comcast IPs in the Houston region have restored.</p><p><small>Mar <var data-var='date'>18</var>, <var data-var='time'>14:21</var> PDT</small><br><strong>Update</strong> - For a brief time, devices had re-registered, although shortly after all connections in the greater Houston area that are on comcast have gone down again. <br /><br />We still highly recommend failing over to a secondary internet connection while the issue is investigated further.</p><p><small>Mar <var data-var='date'>18</var>, <var data-var='time'>13:54</var> PDT</small><br><strong>Update</strong> - The issue is isolated to customers using comcast business in the Texas region. We recommend customers with unregistered devices fail over to a secondary internet provider if available while we continue to investigate.</p><p><small>Mar <var data-var='date'>18</var>, <var data-var='time'>13:10</var> PDT</small><br><strong>Investigating</strong> - We are investigating reports of unregistered phones on our system. We will provide updates as soon as we have identified the cause and impact of this event.<br /><br />Initial investigation appears all phones going offline were previously registered through a comcast business IP address. Customers failing over to a secondary internet connection are showing their connections restoring.</p>tag:status.simplevoip.us,2005:Incident/196140172024-01-08T05:00:25-08:002024-01-08T05:00:25-08:00Voice Platform Upgrade<p><small>Jan <var data-var='date'> 8</var>, <var data-var='time'>05:00</var> PST</small><br><strong>Completed</strong> - The scheduled maintenance has been completed.</p><p><small>Jan <var data-var='date'> 8</var>, <var data-var='time'>01:00</var> PST</small><br><strong>In progress</strong> - Scheduled maintenance is currently in progress. We will provide updates as necessary.</p><p><small>Jan <var data-var='date'> 4</var>, <var data-var='time'>10:18</var> PST</small><br><strong>Scheduled</strong> - We will be upgrading our core voice platform in order to stay current on security patches and bug fixes. The upgrades should be staggered across our 3 zones so as to minimize the impact to our customers - that is, each zone will be offline for a period, but phones will automatically re-register to the nearest healthy zone until their primary zone is back online.<br /><br />Please reach out to us at support@simplevoip.us or 855-899-8647 if you have any questions or concerns.</p>tag:status.simplevoip.us,2005:Incident/188171802023-10-16T14:43:47-07:002023-10-16T14:43:47-07:00Call Completion<p><small>Oct <var data-var='date'>16</var>, <var data-var='time'>14:43</var> PDT</small><br><strong>Resolved</strong> - All zones have been restored to full status and are handling traffic normally.</p><p><small>Oct <var data-var='date'>16</var>, <var data-var='time'>14:02</var> PDT</small><br><strong>Monitoring</strong> - Following the pause of our US-West zone, all calls should now be completing successfully. We will finalize the restart of US-West once all calls in progress have wrapped up, at which point traffic will be restored to primary patterns across all zones. Please reach out to our Support team if your users continue to experience any issues.</p><p><small>Oct <var data-var='date'>16</var>, <var data-var='time'>13:16</var> PDT</small><br><strong>Update</strong> - Services have been restarted in US-East and traffic has been restored. We have verified that the applied fixes have corrected both the issues with outbound 911 connectivity and the one-way/no-way media issues experienced by certain customers.<br /><br />Engineers are currently working to restart US-Central with minimal impact. Once that is done, we will proceed with US-West.</p><p><small>Oct <var data-var='date'>16</var>, <var data-var='time'>12:41</var> PDT</small><br><strong>Update</strong> - In order to resolve the routing issue impacting outbound calls to e911 services, we must restart services which may impact calls in progress. Due to the sensitive nature of the calls being impacted, we are electing to perform these restarts ASAP. The restarts will be performed in a rolling fashion to minimize the impact to your users, but calls lasting longer than 5 minutes will likely be cut off when the restart occurs. We will begin the restarts with our US-East zone at 3:45pm Eastern/12:45pm Pacific and roll through US-Central and US-West shortly after.</p><p><small>Oct <var data-var='date'>16</var>, <var data-var='time'>11:39</var> PDT</small><br><strong>Identified</strong> - We have identified a routing issue following our platform upgrade last night that is impacting outbound call delivery to e911 services and causing call completion or one-way/no-way audio issues for certain customers. Our engineers are working to adjust this routing to restore all services as quickly as possible.</p>tag:status.simplevoip.us,2005:Incident/187660892023-10-16T03:00:17-07:002023-10-16T03:00:17-07:00Voice Platform Upgrade<p><small>Oct <var data-var='date'>16</var>, <var data-var='time'>03:00</var> PDT</small><br><strong>Completed</strong> - The scheduled maintenance has been completed.</p><p><small>Oct <var data-var='date'>15</var>, <var data-var='time'>23:00</var> PDT</small><br><strong>In progress</strong> - Scheduled maintenance is currently in progress. We will provide updates as necessary.</p><p><small>Oct <var data-var='date'>12</var>, <var data-var='time'>07:42</var> PDT</small><br><strong>Scheduled</strong> - We will be upgrading our core voice platform in order to stay current on security patches and bug fixes. The upgrades should be staggered across our 3 zones so as to minimize the impact to our customers - that is, each zone will be offline for a period, but phones will automatically re-register to the nearest healthy zone until their primary zone is back online.<br /><br />Please reach out to us at support@simplevoip.us or 855-899-8647 if you have any questions or concerns.</p>tag:status.simplevoip.us,2005:Incident/185282362023-09-17T12:56:03-07:002023-09-22T11:25:58-07:00Call Completion<p><small>Sep <var data-var='date'>17</var>, <var data-var='time'>12:56</var> PDT</small><br><strong>Resolved</strong> - This incident has been resolved.</p><p><small>Sep <var data-var='date'>17</var>, <var data-var='time'>12:07</var> PDT</small><br><strong>Update</strong> - Inbound and outbound calls are completing at normal on the platform. Our team will continue to monitor for stability.</p><p><small>Sep <var data-var='date'>17</var>, <var data-var='time'>11:50</var> PDT</small><br><strong>Monitoring</strong> - We have implemented a fix for outbound calling. We are actively investigating potential issues pertaining to some inbound calls failing.</p><p><small>Sep <var data-var='date'>17</var>, <var data-var='time'>11:43</var> PDT</small><br><strong>Identified</strong> - The issue has been identified and a fix is being implemented.</p><p><small>Sep <var data-var='date'>17</var>, <var data-var='time'>11:21</var> PDT</small><br><strong>Investigating</strong> - We are investigating reports of calls not completing on our system. We will provide updates as soon as we have identified the cause and impact of this event.</p>tag:status.simplevoip.us,2005:Incident/179472172023-07-24T20:59:19-07:002023-07-24T20:59:19-07:00Call Completion<p><small>Jul <var data-var='date'>24</var>, <var data-var='time'>20:59</var> PDT</small><br><strong>Resolved</strong> - Site-to-site dialing has been fully functional since the fix was applied, and all traffic patterns have normalized. We apologize for the inconvenience today. An RFO will be posted on this status page once we have had time to perform an in-depth analysis.</p><p><small>Jul <var data-var='date'>24</var>, <var data-var='time'>17:25</var> PDT</small><br><strong>Monitoring</strong> - Our Engineering team has implemented a fix for the site-to-site dialing issue. All tests show normal behavior at this time. If you have any locations that continue to have issues reaching other locations, please let our Support team know.</p><p><small>Jul <var data-var='date'>24</var>, <var data-var='time'>13:21</var> PDT</small><br><strong>Update</strong> - Our Engineering team is still working to resolve the issue impacting store-to-store calling for a subset of our locations.</p><p><small>Jul <var data-var='date'>24</var>, <var data-var='time'>10:31</var> PDT</small><br><strong>Identified</strong> - Our engineers have identified a routing issue that is affecting the delivery of some inbound calls. This seems to be primarily impacting calls to the US-West region. We are implementing a quick route change to divert inbound traffic to US-Central to alleviate the issue while we continue to work the underlying root cause.</p><p><small>Jul <var data-var='date'>24</var>, <var data-var='time'>10:12</var> PDT</small><br><strong>Update</strong> - Our engineers are continuing to work to isolate the root cause of this issue. It appears to be impacting a subset of inbound calls to SimpleVoIP numbers. As more information is available regarding the scope of the issue or the time to resolve, it will be posted here.</p><p><small>Jul <var data-var='date'>24</var>, <var data-var='time'>09:40</var> PDT</small><br><strong>Investigating</strong> - We are investigating reports of some calls not completing on our system. We will provide updates as soon as we have identified the cause and impact of this event.</p>tag:status.simplevoip.us,2005:Incident/177764732023-07-10T04:01:32-07:002023-07-10T04:01:32-07:00Database Server Migration - US-West<p><small>Jul <var data-var='date'>10</var>, <var data-var='time'>04:01</var> PDT</small><br><strong>Completed</strong> - The scheduled maintenance has been completed.</p><p><small>Jul <var data-var='date'>10</var>, <var data-var='time'>00:00</var> PDT</small><br><strong>In progress</strong> - Scheduled maintenance is currently in progress. We will provide updates as necessary.</p><p><small>Jul <var data-var='date'> 5</var>, <var data-var='time'>19:53</var> PDT</small><br><strong>Scheduled</strong> - We will migrate our US-West database servers to new hardware to improve performance and stability. With our US-Central and US-East database servers available to pick up the slack, we expect there to be no customer impact as a result of this maintenance. No action needs to be taken on your part.<br /><br />This is a rescheduling of the maintenance originally scheduled for June 26th.</p>tag:status.simplevoip.us,2005:Incident/176355652023-06-26T00:53:26-07:002023-06-26T00:53:26-07:00Database Server Migration - US-West<p><small>Jun <var data-var='date'>26</var>, <var data-var='time'>00:53</var> PDT</small><br><strong>Completed</strong> - Unfortunately we hit a small technical snag which will prevent us from completing this migration within the allotted window. We will have to postpone tonight's maintenance. It will be rescheduled and advance notifications will be sent once this has been addressed.</p><p><small>Jun <var data-var='date'>25</var>, <var data-var='time'>23:00</var> PDT</small><br><strong>In progress</strong> - Scheduled maintenance is currently in progress. We will provide updates as necessary.</p><p><small>Jun <var data-var='date'>20</var>, <var data-var='time'>15:53</var> PDT</small><br><strong>Scheduled</strong> - We will migrate our US-West database servers to new hardware to improve performance and stability. With our US-Central and US-East database servers available to pick up the slack, we expect there to be no customer impact as a result of this maintenance. No action needs to be taken on your part.</p>tag:status.simplevoip.us,2005:Incident/175773132023-06-15T06:24:16-07:002023-06-20T15:48:24-07:00Call Completion<p><small>Jun <var data-var='date'>15</var>, <var data-var='time'>06:24</var> PDT</small><br><strong>Resolved</strong> - All services remain stable following the fix implemented last night. We will be scheduling a maintenance operation to address the underlying root cause as soon as possible.</p><p><small>Jun <var data-var='date'>14</var>, <var data-var='time'>19:52</var> PDT</small><br><strong>Monitoring</strong> - A fix has been implemented, and our monitoring system is showing normal call traffic at this time. We will continue to keep a close eye on this to ensure everything remains stable.</p><p><small>Jun <var data-var='date'>14</var>, <var data-var='time'>19:43</var> PDT</small><br><strong>Update</strong> - Our engineering team is still working to resolve the intermittent connectivity issues affecting inbound and outbound calls.</p><p><small>Jun <var data-var='date'>14</var>, <var data-var='time'>18:10</var> PDT</small><br><strong>Identified</strong> - Our engineering team has identified a connectivity issue between some of our servers. This is causing intermittent inbound and outbound call failures across all 3 of our availability zones. We are working to resolve this as quickly as possible.</p><p><small>Jun <var data-var='date'>14</var>, <var data-var='time'>17:35</var> PDT</small><br><strong>Update</strong> - We are continuing to investigate inbound and outbound call failure.</p><p><small>Jun <var data-var='date'>14</var>, <var data-var='time'>17:16</var> PDT</small><br><strong>Investigating</strong> - We are investigating reports of some calls not completing on our system. We will provide updates as soon as we have identified the cause and impact of this event.</p>tag:status.simplevoip.us,2005:Incident/172803762023-05-17T19:00:30-07:002023-05-17T19:00:30-07:00Administration Portal<p><small>May <var data-var='date'>17</var>, <var data-var='time'>19:00</var> PDT</small><br><strong>Resolved</strong> - We have identified the source of our slow response times to an API slowdown issue caused by a degraded database host. The underlying issue has been resolved, and we have seen no delays on API calls since approximately 11:15 CDT.</p><p><small>May <var data-var='date'>17</var>, <var data-var='time'>09:31</var> PDT</small><br><strong>Monitoring</strong> - Our monitoring systems show that response times have returned to normal. We are continuing to investigate the cause of this spike in response times, but there should be no impact to services currently.</p><p><small>May <var data-var='date'>17</var>, <var data-var='time'>09:16</var> PDT</small><br><strong>Investigating</strong> - We are investigating slow response times from our administration portals. We will provide updates as soon as we have identified the cause and impact of this event.</p>tag:status.simplevoip.us,2005:Incident/171544592023-05-05T08:56:24-07:002023-05-05T08:56:24-07:00Call Completion - Eastern<p><small>May <var data-var='date'> 5</var>, <var data-var='time'>08:56</var> PDT</small><br><strong>Resolved</strong> - This incident has been resolved.</p><p><small>May <var data-var='date'> 5</var>, <var data-var='time'>07:05</var> PDT</small><br><strong>Monitoring</strong> - We received reports at approximately 8:30 am CST that inbound callers were unable to reach locations homed in the Eastern region. After investigating, we determined the issue and implemented a fix at 8:50 am CST. Calls are routing appropriately at this time. Please report any further issues to support@simplevoip.com.</p>tag:status.simplevoip.us,2005:Incident/166194482023-03-23T20:38:12-07:002023-04-10T14:44:36-07:00Administration Portal<p><small>Mar <var data-var='date'>23</var>, <var data-var='time'>20:38</var> PDT</small><br><strong>Resolved</strong> - The system has been stable since traffic was moved from US-West. Our engineers performed a service restart on the US-West cluster once it was safe to do so, and all services there are stable once again. We will give it a bit more time before moving all traffic back to default routes. More information will be available in the post-mortem, which will be posted following further analysis by our Engineering team.</p><p><small>Mar <var data-var='date'>23</var>, <var data-var='time'>14:57</var> PDT</small><br><strong>Monitoring</strong> - We have moved phone registrations and voice traffic off of the US-West zone. At this point all call traffic should be routing normally. Please contact our Support team if your users are experiencing any further issues.</p><p><small>Mar <var data-var='date'>23</var>, <var data-var='time'>14:23</var> PDT</small><br><strong>Update</strong> - We have received examples of long post-dial delay specific to our US-West cluster. We are routing traffic away from US-West so that we can work on that server without disrupting active traffic. This should also fix the post-dial delay issues.</p><p><small>Mar <var data-var='date'>23</var>, <var data-var='time'>14:08</var> PDT</small><br><strong>Identified</strong> - Our engineers have identified a network issue impacting our database servers that coincides with the period of call disruption and are working to fully resolve that. We still see some sporadic API timeouts, but we believe that the inbound/outbound call issues are resolved at this time. Please let our support team know if your users are continuing to experience call completion issues.</p><p><small>Mar <var data-var='date'>23</var>, <var data-var='time'>13:45</var> PDT</small><br><strong>Investigating</strong> - We are still seeing some limited timeouts from the API impacting Portal operations. In addition, we saw a brief period during which call traffic was severely impacted. Some users are reporting ongoing post-dial delay on outbound calls. The engineering team is working to isolate the cause.</p><p><small>Mar <var data-var='date'>23</var>, <var data-var='time'>11:00</var> PDT</small><br><strong>Monitoring</strong> - Our engineering team has fixed the issue with the affected database server and restarted our API application. Admin portal functionality should be fully restored at this time.</p><p><small>Mar <var data-var='date'>23</var>, <var data-var='time'>10:04</var> PDT</small><br><strong>Update</strong> - Diverting voice traffic away from our US-West zone has resolved the outbound dialing issues. We are still working on restoring full functionality to the admin portal.</p><p><small>Mar <var data-var='date'>23</var>, <var data-var='time'>09:07</var> PDT</small><br><strong>Update</strong> - We have received examples of outbound call failures. At this time we believe these failures are isolated to our US-West zone. Our engineers have routed phone traffic away from this zone to attempt to mitigate the issue.</p><p><small>Mar <var data-var='date'>23</var>, <var data-var='time'>08:32</var> PDT</small><br><strong>Identified</strong> - Our engineers have identified a database server with elevated response times. They are working to resolve this issue as quickly as possible.</p><p><small>Mar <var data-var='date'>23</var>, <var data-var='time'>07:55</var> PDT</small><br><strong>Investigating</strong> - We are investigating reports of issues with our Administration Portal located at https://portal.simplevoip.us. We will provide updates as soon as we have identified the cause and impact of this event.<br /><br />**This should not be impacting voice or messaging services.**</p>tag:status.simplevoip.us,2005:Incident/165568642023-03-20T07:58:58-07:002023-03-20T07:58:58-07:00Inbound and Outbound No-Way Audio Failures<p><small>Mar <var data-var='date'>20</var>, <var data-var='time'>07:58</var> PDT</small><br><strong>Resolved</strong> - This incident has been resolved.</p><p><small>Mar <var data-var='date'>18</var>, <var data-var='time'>10:56</var> PDT</small><br><strong>Monitoring</strong> - A fix has been implemented and we are now monitoring this issue for stability.</p><p><small>Mar <var data-var='date'>18</var>, <var data-var='time'>09:15</var> PDT</small><br><strong>Update</strong> - We still believe this issue to be only affecting inbound calls on certain numbers. Effected calls will experience 'no-way audio.' We are still working to resolve this issue at this time.</p><p><small>Mar <var data-var='date'>18</var>, <var data-var='time'>08:15</var> PDT</small><br><strong>Update</strong> - We've performed mitigating steps to reduce the effects of this issue by moving outbound traffic to a non-affected route. Issue at this time likely only pertains to a certain subset of numbers, and on inbound calls only. We are still working on identifying and resolving this as quickly as possible.</p><p><small>Mar <var data-var='date'>18</var>, <var data-var='time'>07:50</var> PDT</small><br><strong>Identified</strong> - We are investigating reports of inbound and outbound calls experiencing no-way audio. We've identified this issue down to a specific upstream carrier and are working to resolve as quickly as possible.</p>tag:status.simplevoip.us,2005:Incident/165330442023-03-17T08:58:30-07:002023-03-17T08:58:30-07:00One-way and No-way Audio | T-Mobile<p><small>Mar <var data-var='date'>17</var>, <var data-var='time'>08:58</var> PDT</small><br><strong>Resolved</strong> - No further reports pertaining to this issue have been received. Monitoring indicates that this issue is now resolved.</p><p><small>Mar <var data-var='date'>16</var>, <var data-var='time'>16:58</var> PDT</small><br><strong>Monitoring</strong> - A fix has been implemented and we are now monitoring for stability.</p><p><small>Mar <var data-var='date'>16</var>, <var data-var='time'>16:19</var> PDT</small><br><strong>Update</strong> - The upstream carrier has noted that the implemented fix has not resolved this issue.</p><p><small>Mar <var data-var='date'>16</var>, <var data-var='time'>14:48</var> PDT</small><br><strong>Update</strong> - The upstream carrier has identified this issue and is working on implementing a fix. We will provide further updates as they become available.</p><p><small>Mar <var data-var='date'>16</var>, <var data-var='time'>13:52</var> PDT</small><br><strong>Update</strong> - At this time, we're seeing that this is affecting the majority of calls inbound, but only from T-Mobile callers. We're working to escalate with the upstream carrier to resolve this issue.</p><p><small>Mar <var data-var='date'>16</var>, <var data-var='time'>13:02</var> PDT</small><br><strong>Identified</strong> - We are investigating reports of one-way audio occurring on some calls inbound from T-Mobile callers. We are working to further identify this issue and resolve as quickly as possible. We will provide updates shortly.</p>tag:status.simplevoip.us,2005:Incident/157651062023-01-17T03:00:28-08:002023-01-17T03:00:29-08:00Voice Platform Upgrade<p><small>Jan <var data-var='date'>17</var>, <var data-var='time'>03:00</var> PST</small><br><strong>Completed</strong> - The scheduled maintenance has been completed.</p><p><small>Jan <var data-var='date'>16</var>, <var data-var='time'>22:00</var> PST</small><br><strong>In progress</strong> - Scheduled maintenance is currently in progress. We will provide updates as necessary.</p><p><small>Jan <var data-var='date'> 3</var>, <var data-var='time'>09:22</var> PST</small><br><strong>Scheduled</strong> - We will be upgrading our core voice platform in order to stay current on new features, security patches and bug fixes. The upgrades will be staggered across our 3 zones to minimize the impact to our customers - that is, each zone will be offline for a period, but phones will automatically re-register to the nearest healthy zone until their primary zone is back online. <br /><br />Please reach out to us at support@simplevoip.com or 855-899-8647 if you have any questions or concerns.</p>tag:status.simplevoip.us,2005:Incident/133795172022-11-28T10:50:19-08:002022-11-28T10:50:19-08:00Call Completion<p><small>Nov <var data-var='date'>28</var>, <var data-var='time'>10:50</var> PST</small><br><strong>Resolved</strong> - This incident has been resolved.</p><p><small>Nov <var data-var='date'>20</var>, <var data-var='time'>17:59</var> PST</small><br><strong>Update</strong> - The US-Central cluster has proven stable over the weekend. Our engineering team has restored all traffic to primary routes. This incident is now resolved.</p><p><small>Nov <var data-var='date'>18</var>, <var data-var='time'>15:53</var> PST</small><br><strong>Monitoring</strong> - We have stabilized all services on secondary routes, and all phone traffic should be processing normally at this time. We will keep voice traffic on its secondary route while we continue to closely monitor our US-Central servers.</p><p><small>Nov <var data-var='date'>18</var>, <var data-var='time'>15:20</var> PST</small><br><strong>Identified</strong> - Our engineers have routed traffic away from our US-central cluster, and preliminary metrics indicate this has restored normal call behavior across all 3 zones. We will continue to investigate the underlying cause of the failed calls on US-Central, but at this time we believe that all calls will complete without issue.</p><p><small>Nov <var data-var='date'>18</var>, <var data-var='time'>15:14</var> PST</small><br><strong>Investigating</strong> - We are investigating reports of some calls to US-Central not completing on our system. We will provide updates as soon as we have identified the cause and impact of this event.</p>tag:status.simplevoip.us,2005:Incident/126980602022-11-03T08:20:21-07:002022-11-03T08:20:21-07:00Call Completion<p><small>Nov <var data-var='date'> 3</var>, <var data-var='time'>08:20</var> PDT</small><br><strong>Resolved</strong> - Services have been stable since restoring default routing yesterday. This incident is now resolved.</p><p><small>Nov <var data-var='date'> 2</var>, <var data-var='time'>14:07</var> PDT</small><br><strong>Monitoring</strong> - Once traffic had been routed away from our servers in US-West, our engineers were able to restart services on the affected servers. After testing all US-West services to confirm that the restart resolved the issue, we have restored traffic to default routing. All traffic is now completing normally over standard routes. We will continue to keep a close eye on these servers over the course of the evening to ensure that all services remain stable.</p><p><small>Nov <var data-var='date'> 2</var>, <var data-var='time'>13:01</var> PDT</small><br><strong>Identified</strong> - Our engineers have routed traffic away from our US-West cluster, and preliminary metrics indicate this has restored normal call behavior across all 3 zones. We will continue to investigate the underlying cause of the failed calls on both US-West and US-East, but at this time we believe that all calls will complete without issue.</p><p><small>Nov <var data-var='date'> 2</var>, <var data-var='time'>12:51</var> PDT</small><br><strong>Investigating</strong> - We are investigating reports of some calls to US-East and US-West not completing on our system. We will provide updates as soon as we have identified the cause and impact of this event.</p>tag:status.simplevoip.us,2005:Incident/126502612022-10-28T23:20:16-07:002022-10-28T23:20:16-07:00Call Completion<p><small>Oct <var data-var='date'>28</var>, <var data-var='time'>23:20</var> PDT</small><br><strong>Resolved</strong> - Our engineers have determined it safe to restore traffic to US-Central, so the emergency routing change has been reverted. All traffic is now flowing over normal routes. We have detected no issues with inbound or outbound calls, and this incident is considered resolved.</p><p><small>Oct <var data-var='date'>28</var>, <var data-var='time'>15:00</var> PDT</small><br><strong>Monitoring</strong> - Our monitoring systems show that calls are completing normally following the routing change. Our engineers are still working on the underlying issue on US-Central, but we do not believe it is causing any impact at this time. If your users are experiencing any issues, please let our support team know.<br /><br />We will continue to monitor the status of the US-East and US-West servers extra closely as they handle the additional call volume. Informational updates will be posted on this page as we determine the action required to restore US-Central to active service. We will spare you the email notifications for those updates unless any additional service impact is discovered.</p><p><small>Oct <var data-var='date'>28</var>, <var data-var='time'>14:49</var> PDT</small><br><strong>Identified</strong> - The current call failures are isolated to our US-Central cluster. We are making a routing change to divert traffic away from that cluster until we are able to identify and resolve the underlying issue. We should have another update shortly.</p><p><small>Oct <var data-var='date'>28</var>, <var data-var='time'>14:41</var> PDT</small><br><strong>Investigating</strong> - We are investigating reports of some calls not completing on our system. We will provide updates as soon as we have identified the cause and impact of this event.</p>tag:status.simplevoip.us,2005:Incident/108860002022-08-23T03:00:46-07:002022-08-23T03:00:46-07:00Service Maintenance in US-East, US-Central, US-West<p><small>Aug <var data-var='date'>23</var>, <var data-var='time'>03:00</var> PDT</small><br><strong>Completed</strong> - The scheduled maintenance has been completed.</p><p><small>Aug <var data-var='date'>23</var>, <var data-var='time'>00:01</var> PDT</small><br><strong>In progress</strong> - Scheduled maintenance is currently in progress. We will provide updates as necessary.</p><p><small>Aug <var data-var='date'>22</var>, <var data-var='time'>12:42</var> PDT</small><br><strong>Scheduled</strong> - We will be performing service maintenance in our US-East, US-Central, and US-West datacenter's to further stabilize our network. Phone registrations and call traffic will be routed to the nearest healthy cluster to avoid any service impact caused by the connectivity interruption.</p>tag:status.simplevoip.us,2005:Incident/101334042022-06-02T21:34:01-07:002022-06-07T11:22:31-07:00Call Completion in US-East<p><small>Jun <var data-var='date'> 2</var>, <var data-var='time'>21:34</var> PDT</small><br><strong>Resolved</strong> - The US-East cluster has been stable for the past 12 hours. Our engineering team has restored all traffic to primary routes. This incident is now resolved.</p><p><small>Jun <var data-var='date'> 2</var>, <var data-var='time'>10:39</var> PDT</small><br><strong>Monitoring</strong> - We have stabilized all services on US-East, and all phone and web traffic should be processing normally at this time. We will keep voice traffic on its secondary route while we continue to closely monitor our US-East servers.</p><p><small>Jun <var data-var='date'> 2</var>, <var data-var='time'>09:37</var> PDT</small><br><strong>Identified</strong> - Our engineers have temporarily redirected phone traffic away from US-East while we continue working to resolve this issue. Inbound and outbound calls are completing normally over their redundant routes. Call Center users may notice slow portal performance while this issue is active.</p><p><small>Jun <var data-var='date'> 2</var>, <var data-var='time'>09:25</var> PDT</small><br><strong>Investigating</strong> - We are investigating reports of some calls not completing on our US-East zone. We will provide updates as soon as we have identified the cause and impact of this event.</p>tag:status.simplevoip.us,2005:Incident/101268902022-06-01T19:43:39-07:002022-06-07T11:22:10-07:00Call Completion<p><small>Jun <var data-var='date'> 1</var>, <var data-var='time'>19:43</var> PDT</small><br><strong>Resolved</strong> - All services have been stable for the past 6 hours, which includes our peak traffic period. We now consider this incident fully resolved.</p><p><small>Jun <var data-var='date'> 1</var>, <var data-var='time'>13:51</var> PDT</small><br><strong>Monitoring</strong> - The rolling restarts have completed, and all services in US-East are now functioning normally.</p><p><small>Jun <var data-var='date'> 1</var>, <var data-var='time'>13:40</var> PDT</small><br><strong>Update</strong> - Our engineering team is now performing a rolling service restart on our US-East cluster to resolve this issue with as little disruption as possible. Phones should re-register without interruption, while web users may be asked to log in again.</p><p><small>Jun <var data-var='date'> 1</var>, <var data-var='time'>13:33</var> PDT</small><br><strong>Identified</strong> - We are investigating reports of some calls not completing on our system. We believe to have identified the cause and are performing steps to resolve this issue.</p>tag:status.simplevoip.us,2005:Incident/92384232022-02-04T14:19:30-08:002022-02-04T14:19:30-08:00Call Completion<p><small>Feb <var data-var='date'> 4</var>, <var data-var='time'>14:19</var> PST</small><br><strong>Resolved</strong> - The issue has been resolved and Verizon Wireless is routing traffic to all of our numbers appropriately. If any locations are having continued issues, please reach out to our support team at support@simplevoip.com for assistance.</p><p><small>Feb <var data-var='date'> 4</var>, <var data-var='time'>10:02</var> PST</small><br><strong>Identified</strong> - We have isolated the issue to roughly 500 numbers that are unreachable from Verizon Wireless. We are working with them to ensure that calls from Verizon Wireless subscribers are delivered appropriately to our network. At present we are unable to provide an ETR since we do not directly control the affected routing, but we are escalating as aggressively as possible to get this corrected.</p><p><small>Feb <var data-var='date'> 4</var>, <var data-var='time'>08:44</var> PST</small><br><strong>Investigating</strong> - We are investigating reports of some calls not completing on our system. Most of the reports we have received have been from Verizon Wireless users unable to reach certain SimpleVoIP numbers. We are working to determine whether the issue exists on the Verizon Wireless network or with SimpleVoIP's carrier network. We will provide updates as soon as we have identified the cause and impact of this event.</p>tag:status.simplevoip.us,2005:Incident/91295932022-01-20T22:57:19-08:002022-01-20T22:57:19-08:00Call Completion in US-East<p><small>Jan <var data-var='date'>20</var>, <var data-var='time'>22:57</var> PST</small><br><strong>Resolved</strong> - The underlying routing issue has been resolved. Our testing confirmed the fix, and we have migrated all US-East numbers back to their optimal routes. If you have any questions or concerns, please reach out to our support team at 855-899-8647.</p><p><small>Jan <var data-var='date'>20</var>, <var data-var='time'>08:09</var> PST</small><br><strong>Monitoring</strong> - All affected US-East numbers have been temporarily rerouted to US-Central, and testing shows that inbound traffic to the impacted numbers is now being handled properly. Please reach out to our support team to report any issues. We will continue to monitor this traffic closely to ensure system stability, and confirmation will be posted once the underlying issue has been resolved and all traffic is reverted to its preferred routing.</p><p><small>Jan <var data-var='date'>20</var>, <var data-var='time'>07:56</var> PST</small><br><strong>Identified</strong> - Our operations team has identified a voice routing issue impacting numbers that route preferentially to our US-East cluster. We are moving that traffic temporarily to US-Central, which we expect will resolve the inbound completion issues until the underlying issue is repaired. An update will be posted once the reroute is complete.</p><p><small>Jan <var data-var='date'>20</var>, <var data-var='time'>07:51</var> PST</small><br><strong>Investigating</strong> - We are investigating reports of some inbound calls not completing to the US-East region. We will provide updates as soon as we have identified the cause and impact of this event.</p>tag:status.simplevoip.us,2005:Incident/90025162022-01-04T13:56:08-08:002022-01-04T13:56:08-08:00Admin Portal and Call-Center Administration Portal Down<p><small>Jan <var data-var='date'> 4</var>, <var data-var='time'>13:56</var> PST</small><br><strong>Resolved</strong> - We have received no further indication or reports of problems after the fix was implemented. At this time, we are considering this issue resolved.</p><p><small>Jan <var data-var='date'> 4</var>, <var data-var='time'>12:52</var> PST</small><br><strong>Monitoring</strong> - A fix has been implemented for call-center/PBX portal, now allowing users to log in. We will continue to monitor this incident to confirm there are no peripheral services remaining down after implementation of this fix.</p><p><small>Jan <var data-var='date'> 4</var>, <var data-var='time'>12:15</var> PST</small><br><strong>Update</strong> - The administration portal at admin.simplevoip.us functionality has restored. At this time, we're still experiencing issues with users logging into the call-center/PBX portal. Our engineers have already identified a fix for this issue as well and are working to implement the fix.</p><p><small>Jan <var data-var='date'> 4</var>, <var data-var='time'>11:56</var> PST</small><br><strong>Identified</strong> - We have identified an issue preventing users from logging into the administration and call-center portals. At this time all PBX systems are fully operational, calls are being delivered as normal and no registrations are being affected by this portal outage. Our engineers have identified the issue and are working to implement a fix.</p>