Final Update Received from our upstream InterLATA carrier Friday 08/19/22
We have received a final update on the voice issues from last Monday (08/15/22) from our InterLATA carrier with some additional information regarding the issues.
08/19/22
Dear Valued Partner
As your InterLATA carrier for your global communication needs I know you count on us as your trusted communications provider. We are the backbone of your business communications, so when our network goes down so does yours. Our mission is to delight our customers. This week we are very disappointed that, due to external circumstances, we let you down. I want to personally apologize and assure you we will fix it and do better.
As you are aware, we experienced an unprecedented network-wide outage which affected many of our customers’ inbound and outbound voice calls on August 15th between 12:19 p.m. ET – 4:15 p.m. ET. Our technical team restored partial call processing at 2:15 p.m. ET and all service was fully back up-and-running by 4:15 p.m. ET.
Our team investigated every element within our network to determine the root cause of the outage and learned that it was triggered due to an impairment and a change that was made, without proper notifications, by our STIR/SHAKEN vendor (Neustar). We and many other communications providers use Neustar to validate and authenticate phone numbers – to combat spoofed robocalls and scams with caller ID authentication. In addition to disrupting our business, multiple other telecommunications providers also experienced an outage that impacted their customers’ inbound/outbound phone calls. This is unacceptable and we are evaluating this vendor’s processes and notifications to ensure we are protecting your business with the reliable and resilient solutions you have come to expect from us.
The review of this incident is being directly overseen by the Executive Leadership team. We are also evaluating ways to further protect our network against any potential future incidents with our vendors.
Ultimately, we understand that you have always trusted us to maintain the highest level of reliability independent of any issues with third parties. It is our responsibility to preserve that trust.
We also understand the need to improve our communications strategy to provide real-time mass communications to our customers and partners when unprecedented, complete network outages or other events occur. We are implementing new tools to help with instantaneous communications and will provide additional details in the coming weeks.
I know how difficult it is to face a customer and apologize for an unforeseen and unexpected failure, as you had to do this week. Our commitment is to work diligently to restore your trust in our continued partnership.
———————————————————————–
To our valued customers,
At around 1:00 PM Peregrine Networks starting seeing problems with inbound & outbound calls. Investigations determine this is was nationwide affecting a multitude of voice providers. We have received the following from our InterLATA voice carrier that provides origination & termination of phone calls across the globe.
Dear Voice Provider
Today, we experienced a network impairment triggered by an industry event attributed to a leading STIR/SHAKEN vendor. The full extent of the impact is still being investigated and we are developing the mitigating steps to identify and isolate re-occurrence of malformed messages generated by the vendor which we have not experienced before today.
At 12:19 pm EST, we began to see degradation to our network. Due to the nature of the failure, we began investigating every element within our network. We then isolated it to our voice routing engine of which there are multiple nodes distributed geographically for full redundancy. As an FCC requirement, all these nodes are required to be connected to the vendor for STIR/SHAKEN processing capabilities. Due to the corrupt and malformed messaging received from the STIR/SHAKEN vendor, this impacted all our nodes simultaneously. We determined a plan of action and then had to rebuild and re-sync each voice routing engine individually. Initial call processing began to be restored at 14:15 EST. As each node was brought back up, service began restoring fully since our core routing architecture is designed to operate on just one node but designed with additional redundancy. All critical production nodes were fully restored at 16:15 ET.
As of this point, our voice network is normalized and call processing is performing as expected. We has disassociated services for STIR/SHAKEN temporarily until we get formal remediation steps from the STIR/SHAKEN vendor. As we get a more formal release and analysis from the vendor, we will be releasing a formal RFO as soon as possible.
Peregrine Networks engineers will be monitoring the situation closely. Once we receive the formal Reason For Outage (RFO) from our InterLATA voice provider it will be posted here.
We apologize for inconvenience this may have caused our valued customers as we strive for reliability and performance in our infrastructure. The upstream voice carrier we use to transport voice calls across the globe has been used by us for over half-decade without incident and we expect the incident on 08/15/22 will further enhanced geographical redundancy.
In the mean time if you have an additional questions please reach out to us at 603-296-0760 opt 2 or email us at support@peregrinenetworks.net