NLA eClips Service Incident - Report

Problem:

The NLA’s London data centre lost connectivity to the Internet on Friday 20^th March at 18:05. Connectivity was fully restored at 19:39. All NLA services were unavailable during this period.

Cause:

The NLA’s Internet Service Provider detected a problem with one of their upstream peers which affected some downstream customers. NLA was not one of the customers affected by this problem. While the problem was being investigated, the ISP decided to failover all customers (including the NLA) to an alternate peer at 18:00 on Friday. The failover process did not complete as expected and resulted in Internet connectivity failures for some customers - including the NLA.

Solution:

Customers of the NLA’s Internet Service Provider, which were not affected by the original upstream peering issue, were then failed back to the primary peer at 19:20. Normal connectivity was restored throughout the NLA environment by 19:39. The NLA has received assurances from its Internet Service Provider that a review of the actions taken on Friday has been completed and that the procedure for peering changes has been modified to avoid a similarly impacting event in future. Operational steps associated with changes in this procedure will be scheduled for testing during a maintenance window soon.