Incident Notification: eClips content processing delay [CLOSED]

Time:

An incident was declared today at 1a.m

Incident Description:

Content processing is delayed and 1st edition content may not be available at 2:30a.m. We expect a delay of up to 2 hours but will provide a further assessment at approx. 2:15a.m. 

Expected Resolution Time:

3:30a.m

Update on Saturday, October 19, 2013 at 2:21 by eClips Team

Time:

An incident was declared today at 1a.m

Update:

Content processing has resumed however due to large backlog it remains likely that we will miss the 2:30a.m target. We will continue to monitor and provide a further update approx. 5a.m or sooner. 

Expected Resolution Time:

Between 5:00a.m  - 7a.m.

Update on Saturday, October 19, 2013 at 10:15 by eClips Team

Update:

The issue has been rectified. However, the backlog of content continues to be processed.

Our next update will be to inform when the backlog has been processed and it's time is unknown. 

We apologise for inconvenience caused. A detailed incident report will follow in the week.

Update on Saturday, October 19, 2013 at 13:45 by eClips Team

Closure Note:

All content, with the exception of missing pages was reported to be processed and distributed at 13:15. 

Missing pages have been escalated to publishers and will be processed and distributed soon. 

We are taking steps to try ensure this does not re-occur tonight. However in the event that it does, we will communicate through the incident notification channels.

A detailed incident report will follow during the following week once a full investigation has been carried out.

Once again we apologise for the inconvenience caused.  

eClips Incident Announcement - Delayed content supply [CLOSED]

Time:

An incident was declared today at 23:20

Incident Description:

An incident is occurring which is delaying eClips content supply. All core and regional titles have been affected. 

Cause:

The cause is unknown at this time & NLA engineers are investigating. 

Expected Resolution Time:

Unknown at this time.

Update on Friday, September 13, 2013 at 0:12 by eClips Team

The issue is currently ongoing and impacting IDS and XML core titles. Our engineers continue to work on the issue. 

Please expect the next update at 1a.m.

Update on Friday, September 13, 2013 at 1:03 by eClips Team

The incident is on-going as at 1a.m. Our engineers continue to investigate. We apologise for the inconvenience.

Please expect the next update at 3a.m. 

Update on Friday, September 13, 2013 at 3:13 by eClips Team

Processing of content has resumed, however there is a large backlog and we estimate completed delivery by 9a.m.

The root cause is unknown at this time and we will continue to diagnose. We will provide a incident report as soon as possible. 

Please expect the next update at 10a.m.

Update on Friday, September 13, 2013 at 4:21 by eClips Team

Content is processing however delivery behaviour appears intermittent and therefore delayed. We apologise for this ongoing inconvenience. Our engineers are troubleshooting further. 

We will provide an update as soon as possible, but please expect one no later than 10a.m. 

Update on Friday, September 13, 2013 at 8:39 by eClips Team

The issue has been rectified. However, the backlog of content is still being processed.

We apologize for inconvenience caused & estimate that delivery of all content will be complete by 10:00am UK. An incident report will follow later in the day.

Update on Friday, September 13, 2013 at 12:29 by eClips Team

NLA eClips Service Incident - Report

This incident has now been closed and normal services has resumed.

Problem:
On the night of 12th Sept, it was discovered that the number of pages flowing through to the Production Centre had greatly reduced. NLA engineers started to investigate the issue & an incident was declared at 11:30pm. Since the problem occurred during the peak production window, there was a backlog of content waiting to be delivered. The issue was rectified around 2:30am and content then started to get delivered. All core processing was completed by 10:30am

Cause:
The issue was caused by the knock on effects of internal problems at one of our publishers which caused an NLA media access processing backlog that eventually led to an effective denial of service.  After resolution, given the time of the incident, it took a long time for the backlog to clear.

Solution:
In order to prevent a similar situation and impact in future, we will be deploying inbound bandwidth throttling for publisher uploads, and additional filters to check and alert us on unacceptable workflow volumes.

We apologize for the inconvenience caused. Please contact clientservices@nla.co.uk if you have further questions.

eClips Web Incident Notification: Publisher supply failure for "Independent.co.uk"

Time:    An incident was declared today at 9:10a.m

 Incident Description:

A publisher supply failure for Independent.co.uk is currently occurring. We have escalated this to the publisher and are working to resume supply as soon as possible.

Cause:  Publisher supply failure.  

Expected Resolution Time: 

The resolution time is unknown at this time.

 Next Update:

 Please expect an update by 12:00

Update on Friday, April 26, 2013 at 12:03 by eClips Team

Incident Update: The publisher supply has resumed and new article content is now available. We have a coverage gap over the 24/04 and 25/04 and are trying to address this with the publisher.

Cause: Publisher Supply failure cause unknown at this time.

Expected Resolution Time: 

The resolution time is unknown at this time.

 Next Update:

 Please expect an update by 14:00.

Update on Friday, April 26, 2013 at 15:12 by eClips Team

Incident Update: We are waiting on the publisher to determine steps to address the coverage gap.  

Cause: Publisher Supply failure cause unknown at this time.

Expected Resolution Time: 

The resolution time is unknown at this time.

 Next Update:

Please expect an update by 16:30.

Update on Friday, April 26, 2013 at 17:08 by eClips Team

Incident CLOSED

Incident Closure: The content supply for independent.co.uk has resumed. There are some coverage gaps for 24/04 & 25/04 which we are working with the Independent to recover.

Cause: Publisher Supply failure.

Resolution time: 17:08.

Brander Restart

Due to a malformed clipping file getting 'stuck' in the system, we needed to execute an emergency restart of our brander application. We restarted the service at 10:51 and clippings were unavailable for approximately 5 minutes. The service has now been restored and you shouldn't experience any other issues.