Incident Announcement - thesun.co.uk content loading into eClips Web [CLOSED]

Time:

An incident was declared today at 16:30

Incident Description:

An incident has occurred which is affecting new content from thesun.co.uk loading into eClips Web. New content from thesun.co.uk has not been populated into the eClips Web service since earlier today.

Cause:

Publisher supply has been disrupted

Expected Resolution Time:

Unknown at this time. We have escalated this to News UK and are awaiting an update.

Next Update:

Please expect the next update by 18:30

Update on Friday, October 25, 2013 at 10:25 by Incident Management Team

Incident Update:

New content from thesun.co.uk is still not being loaded into eClips Web. We have re-engaged the publisher this morning and they are actively trying to resolve as soon as possible.

Expected Resolution Time:

Unknown at this time

Next Update:

Please expect the next update by 12:00 today

Regards,

NLA Service Operations Management

Update on Friday, October 25, 2013 at 14:52 by Incident Management Team

Incident Update:

New articles from thesun.co.uk are now flowing into the eClips Web database, including the backlog of articles that were not loading into the database yesterday (24^th)

Cause:

The cause of the incident is still under investigation at News UK.

Resolution Time:

The issue was declared as fixed by the publisher at 11:00 this morning, and the NLA incident resolved at 12:00.

Incident Notification: eClips content processing delay [CLOSED]

Time:

An incident was declared today at 1a.m

Incident Description:

Content processing is delayed and 1st edition content may not be available at 2:30a.m. We expect a delay of up to 2 hours but will provide a further assessment at approx. 2:15a.m. 

Expected Resolution Time:

3:30a.m

Update on Saturday, October 19, 2013 at 2:21 by eClips Team

Time:

An incident was declared today at 1a.m

Update:

Content processing has resumed however due to large backlog it remains likely that we will miss the 2:30a.m target. We will continue to monitor and provide a further update approx. 5a.m or sooner. 

Expected Resolution Time:

Between 5:00a.m  - 7a.m.

Update on Saturday, October 19, 2013 at 10:15 by eClips Team

Update:

The issue has been rectified. However, the backlog of content continues to be processed.

Our next update will be to inform when the backlog has been processed and it's time is unknown. 

We apologise for inconvenience caused. A detailed incident report will follow in the week.

Update on Saturday, October 19, 2013 at 13:45 by eClips Team

Closure Note:

All content, with the exception of missing pages was reported to be processed and distributed at 13:15. 

Missing pages have been escalated to publishers and will be processed and distributed soon. 

We are taking steps to try ensure this does not re-occur tonight. However in the event that it does, we will communicate through the incident notification channels.

A detailed incident report will follow during the following week once a full investigation has been carried out.

Once again we apologise for the inconvenience caused.  

eClips Incident Announcement - Delayed content supply [CLOSED]

Time:

An incident was declared today at 23:20

Incident Description:

An incident is occurring which is delaying eClips content supply. All core and regional titles have been affected. 

Cause:

The cause is unknown at this time & NLA engineers are investigating. 

Expected Resolution Time:

Unknown at this time.

Update on Friday, September 13, 2013 at 0:12 by eClips Team

The issue is currently ongoing and impacting IDS and XML core titles. Our engineers continue to work on the issue. 

Please expect the next update at 1a.m.

Update on Friday, September 13, 2013 at 1:03 by eClips Team

The incident is on-going as at 1a.m. Our engineers continue to investigate. We apologise for the inconvenience.

Please expect the next update at 3a.m. 

Update on Friday, September 13, 2013 at 3:13 by eClips Team

Processing of content has resumed, however there is a large backlog and we estimate completed delivery by 9a.m.

The root cause is unknown at this time and we will continue to diagnose. We will provide a incident report as soon as possible. 

Please expect the next update at 10a.m.

Update on Friday, September 13, 2013 at 4:21 by eClips Team

Content is processing however delivery behaviour appears intermittent and therefore delayed. We apologise for this ongoing inconvenience. Our engineers are troubleshooting further. 

We will provide an update as soon as possible, but please expect one no later than 10a.m. 

Update on Friday, September 13, 2013 at 8:39 by eClips Team

The issue has been rectified. However, the backlog of content is still being processed.

We apologize for inconvenience caused & estimate that delivery of all content will be complete by 10:00am UK. An incident report will follow later in the day.

Update on Friday, September 13, 2013 at 12:29 by eClips Team

NLA eClips Service Incident - Report

This incident has now been closed and normal services has resumed.

Problem:
On the night of 12th Sept, it was discovered that the number of pages flowing through to the Production Centre had greatly reduced. NLA engineers started to investigate the issue & an incident was declared at 11:30pm. Since the problem occurred during the peak production window, there was a backlog of content waiting to be delivered. The issue was rectified around 2:30am and content then started to get delivered. All core processing was completed by 10:30am

Cause:
The issue was caused by the knock on effects of internal problems at one of our publishers which caused an NLA media access processing backlog that eventually led to an effective denial of service.  After resolution, given the time of the incident, it took a long time for the backlog to clear.

Solution:
In order to prevent a similar situation and impact in future, we will be deploying inbound bandwidth throttling for publisher uploads, and additional filters to check and alert us on unacceptable workflow volumes.

We apologize for the inconvenience caused. Please contact clientservices@nla.co.uk if you have further questions.

NLA media access - Incident Announcement

Time:

An incident was declared today at 11:45

Incident Description:

An incident has occurred which is affecting Domain Name resolution. There is intermittent loss of access to our products.

Cause:

This is due to a problem at our external DNS provider, Network Solutions and is outside the NLA’s control.

Expected Resolution Time:

Unknown at this time. We have escalated this to Network Solutions & are awaiting a response. Note that users who have accessed sites in the past 24 hours should still be able to access them now.

Next Update:

Please expect the next update by 13:00

Incident Report - eClips missing content

Problem:

Several pages were missing in the eClips feed for publication date 2^nd May 2013. The problem was initially discovered around 1am but due to there being less than ten missing pages at the time, it was not deemed a wider issue. It was only much later that the impact of the issue was discovered & more pages were found to be missing. An incident was declared at 9:15am & the problem was resolved by 10:30am. Missing content was then reprocessed & loaded into the eClips database with all content completed by 4pm

Cause:

A planned change took place yesterday which enabled processing of new content. Unfortunately, one of the configuration files in the change erroneously enabled a dormant part of NLA infrastructure. Despite passing through a quality checking and approval process, this subtle error went unnoticed. When the dormant system became active, it moved some content out of the production workflow, resulting in severe processing delays. The error was corrected and content was returned to the production workflow.

Solution:

In order to prevent a similar situation and impact in future, the change deployment process has been modified to include specific checks for this type of condition. Additional automated monitoring has been put in place to identify this and similar types of unexpected behaviour. Finally, the dormant parts of the system in question will be removed entirely next month.