eClips Incident Announcement - Delayed content supply [CLOSED]

Time:

An incident was declared today at 23:20

Incident Description:

An incident is occurring which is delaying eClips content supply. All core and regional titles have been affected. 

Cause:

The cause is unknown at this time & NLA engineers are investigating. 

Expected Resolution Time:

Unknown at this time.

Update on Friday, September 13, 2013 at 0:12 by eClips Team

The issue is currently ongoing and impacting IDS and XML core titles. Our engineers continue to work on the issue. 

Please expect the next update at 1a.m.

Update on Friday, September 13, 2013 at 1:03 by eClips Team

The incident is on-going as at 1a.m. Our engineers continue to investigate. We apologise for the inconvenience.

Please expect the next update at 3a.m. 

Update on Friday, September 13, 2013 at 3:13 by eClips Team

Processing of content has resumed, however there is a large backlog and we estimate completed delivery by 9a.m.

The root cause is unknown at this time and we will continue to diagnose. We will provide a incident report as soon as possible. 

Please expect the next update at 10a.m.

Update on Friday, September 13, 2013 at 4:21 by eClips Team

Content is processing however delivery behaviour appears intermittent and therefore delayed. We apologise for this ongoing inconvenience. Our engineers are troubleshooting further. 

We will provide an update as soon as possible, but please expect one no later than 10a.m. 

Update on Friday, September 13, 2013 at 8:39 by eClips Team

The issue has been rectified. However, the backlog of content is still being processed.

We apologize for inconvenience caused & estimate that delivery of all content will be complete by 10:00am UK. An incident report will follow later in the day.

Update on Friday, September 13, 2013 at 12:29 by eClips Team

NLA eClips Service Incident - Report

This incident has now been closed and normal services has resumed.

Problem:
On the night of 12th Sept, it was discovered that the number of pages flowing through to the Production Centre had greatly reduced. NLA engineers started to investigate the issue & an incident was declared at 11:30pm. Since the problem occurred during the peak production window, there was a backlog of content waiting to be delivered. The issue was rectified around 2:30am and content then started to get delivered. All core processing was completed by 10:30am

Cause:
The issue was caused by the knock on effects of internal problems at one of our publishers which caused an NLA media access processing backlog that eventually led to an effective denial of service.  After resolution, given the time of the incident, it took a long time for the backlog to clear.

Solution:
In order to prevent a similar situation and impact in future, we will be deploying inbound bandwidth throttling for publisher uploads, and additional filters to check and alert us on unacceptable workflow volumes.

We apologize for the inconvenience caused. Please contact clientservices@nla.co.uk if you have further questions.