13775433 - Core Network Fibre Disturbance

Details:

Current Status:Closed
Summary
A fibre disturbance has been identified in the core network and may be impacting Business customers nationwide

Date Created - 07/04/2021 23:19:31
Incident Duration - 0 day(s) 3 hour(s) 11 minute(s).
Service Affecting Status - Incident is not Service Affecting.
Service Affecting Duration - 0 day(s) 3 hour(s) 44 minute(s).
Tickets Raised to Date: 0

Technical / Suspected Root Cause
A fibre disturbance has been identified in the Core Network impacting Business customers nationwide. TTB customers on the following sites will experience issues: nge001.don, nge001.hex, nge001.mal, nge001.new, nge001.sto, nge001.tcm, npe001.tnn, npe001.hex, nge001.hex, nge001.wre, nge002.ash and nge001.wol. At 22:23, IOC identified a drop in light on the Huawei DWDM (Dense Wavelength Division Multiplexing) which impacted on Voice also. All alarms had cleared shortly after on the Huawei DWDM, however we were still seeing impact on the Core Network at 22:45 where it was identified a disturbance on the DWDM Core Network (we were still getting reports of services still being down). Core Networks had then been engaged and had identified an issue on ner00.msp (Core Router) in Manchester science park who had raised this with Juniper under JTAC - 2021-0407-0771. As of 01:42, Core Networks had advised that circuits were then starting to come back up where IOC and Service Assurance teams had advised a number of ‘up’ alerts had been seen however Core Networks advised we still had issues on nge002.ash, the rest of the nodes had then recovered (only that one box had issues). As of 02:15 confirmation that nge002.ash was also back up. Core Networks had advised that in order to restore service, they carried out a switchover on the node to recover. Root cause was due to a flap on DWDM which caused issues on the nodes, however RCA will be with Juniper to investigate. The incident will now be resolved as all systems are available with no further issues reported. Monitoring will remain in place to ensure stability.

Start date:23:28 7th Apr 2021
End date:02:32 8th Apr 2021