INC000100663181 Multiple regions - Loss of data and voice services - Card failure


Current Status:Closed
We've identified an issue impacting customer connectivity.

Our initial analysis shows the issue is impacting service in multiple regions in the UK. We're currently working through the impact and issue and will look to provide more detail as soon as we have it.

We'll keep you informed and updated on developments. Our priority is to restore service to normal operations with minimal disruption as soon as possible.

-General Update-

A bridge call with Core Network Engineering is currently ongoing. Following a further impact assessment it has been identified that approx 50% of the Voice base is impacted resulting in the loss of voice service for those LLU and FTTP customers.

Core Networks have identified issues in 2 NER routers in the Birmingham region. Due to the impact this has now been upgraded to an MSO.

Issues have been identified in 2 NER routers in the Birmingham region impacting the loss of Voice service for LLU and FTTP customers.

An action has been taken to re load the 2 NER cards again.

Further information will be provided within the next update due time.

-General Update-

Both NER001.bir and NER002.bir are up and stable since 09:44 and 10:01 respectively however, we are still experiencing issues with Telco Cloud reaching the application Virtual machines which host Metswitch and Radius applications. This causing a loss of voice service to a large number of MPF and VOIP customers across group.

Investigations are continuing into the cause of the loss of connectivity between NXS edge and VMs is on-going and possible mitigations to move services to Irlam to restore service to the customers. Following this it will require application clean-up work post change.

Further information will be provided within the next update due time.

-Incident Closure-

Approximately 50% of our customers experienced a loss of inbound and outbound voice services.

Corrective actions have been taken to restore service to our Voice Platform as of 11:27. Service to all impacted voice customers has now been restored.

This incident will now be closed with root cause analysis being tracked via the Problem Management process.

This incident will now be closed.

WorkLog Submit Date : 08/02/2024 12:16:58

Start date:09:23 8th Feb 2024
End date:12:21 8th Feb 2024