NLA eClips Service Incident - Report

Problem:

Some core content for publication date 11th Feb 2009 was missing from the eClips database, and from eClips feeds, by the NLA KPI deadlines. Certain pages were missing from three core titles:

Daily Mirror

The Daily Telegraph

The Guardian

Cause:

A core, automated NLA service malfunctioned on Tuesday 10th February 2009 at 19:24. NLA engineers were alerted to the malfunction at 19:52 and had the service running normally by 20:16. Unfortunately, the nature of the malfunction resulted in the corruption of pages delivered by publishers, to the NLA, within the aforementioned period.

When the corruption became apparent, publishers were engaged for retransmission of the affected pages. The normal NLA escalation process was not followed when recovering pages from The Daily Telegraph, which resulted in these pages being made available much later than expected. All the missing pages have now been re-delivered to the NLA and are being processed.

Solution:

To mitigate the risk and effects of a similar event occurring in future, the automated monitoring strategy for the affected service will be modified to alert NLA engineers of impending failure, rather than upon failure. The publisher escalation process will be reviewed to reduce the possibility of deviation from process under similar circumstances. Finally, the architecture of the affected service will be enhanced to make it more robust.