IT Services Alerts - Resolved: Eircom service disruption affecting inbound and outbound phone calls, September 2nd

Updated: Thursday, 3rd September 2015, 9am

The nationwide issues affecting inbound and outbound phone calls to Trinity yesterday were resolved by Eircom at approximately 6pm.  Below is a report on the disruption from Noel Donnelly, Eircom Director of Customer Service and Assurance.  Noel has since confirmed that there were no further issues overnight and that they are confident the service will continue to work under full load.

Good Evening,

I want to update you and provide some background on the outage which has affected a number of our Next Generation Intelligent Network (NGIN) services since this morning and which we have been working with our network engineers to resolve throughout the day.

I am glad to say at the outset that since approximately 6 pm all affected services have now been restored and are stable.

Since the problem first appeared we have treated this as a highest order priority Incident and have followed a well-established logical process of fault localisation involving a team of our most senior engineers including our vendors to identify and resolve the problem. A brief summary is as follows;

  • At approximately 9.52 am one of the IN elements in the network which governs the operation of key services including 18xx, 15xx, SIP/VoIP, VPN and eMobile began to perform in a non-standard and unpredictable manner which ultimately caused the affected services to shut down. This is a condition that we have never observed previously and nor has the vendor who provides the underlying technology that supports each services.
  • Because of the complexity of the processes involved including applications, switching and signalling, it took a protracted period of time to isolate the location of the problem.
  • We are now confident that we have localised the issue and have taken the necessary action to stabilise the situation.

We are confident that we can maintain the stability of the service and we will be working through the night to ensure the Next Generation platform is prepared to accommodate normal business volumes without further disruption.  We will further confirm the outcome of our efforts before start of business tomorrow.

Following full stabilisation we are undertaking a full Root Cause Analysis which we will share with you over the coming weeks. We will also be available to meet to discuss this when the results of this analysis is to hand.

Regards,

Noel Donnelly
(Eircom Director of Customer Service and Assurance)

Update: Wednesday, 2nd September 2015, 4.30pm

Eircom has informed us that they have identified an issue and put a workaround in place that should restore service to the Dublin (01) area.

Update: Wednesday, 2nd September 2015, 3pm

Based on the latest information from Eircom, there are still intermittent issues making outbound phone calls from Trinity phones, and making inbound phone calls from outside Trinity.

Updates will be posted here.  There is also more detailed information from Eircom.

First Posted: Wednesday, 2nd September 2015, 10.50am

Eircom are experiencing a nationwide ISDN and mobile service disruption at the moment with the result that it is not possible to make calls into Trinity from outside, nor to make external calls from within Trinity.

Internal Trinity phone services are not affected by this.

Updates on this situation will be posted here.