An incident was declared today at 16:30

Incident Description:

An incident has occurred which is affecting new content from thesun.co.uk loading into eClips Web. New content from thesun.co.uk has not been populated into the eClips Web service since earlier today.

Cause:

Publisher supply has been disrupted

Expected Resolution Time:

Unknown at this time. We have escalated this to News UK and are awaiting an update.

Next Update:

Please expect the next update by 18:30

An incident was declared today at 11:45

Incident Description:

An incident has occurred which is affecting Domain Name resolution. There is intermittent loss of access to our products.

Cause:

This is due to a problem at our external DNS provider, Network Solutions and is outside the NLA’s control.

Expected Resolution Time:

Unknown at this time. We have escalated this to Network Solutions & are awaiting a response. Note that users who have accessed sites in the past 24 hours should still be able to access them now.

Next Update:

Please expect the next update by 13:00

Problem:

Several pages were missing in the eClips feed for publication date 2^nd May 2013. The problem was initially discovered around 1am but due to there being less than ten missing pages at the time, it was not deemed a wider issue. It was only much later that the impact of the issue was discovered & more pages were found to be missing. An incident was declared at 9:15am & the problem was resolved by 10:30am. Missing content was then reprocessed & loaded into the eClips database with all content completed by 4pm

Cause:

A planned change took place yesterday which enabled processing of new content. Unfortunately, one of the configuration files in the change erroneously enabled a dormant part of NLA infrastructure. Despite passing through a quality checking and approval process, this subtle error went unnoticed. When the dormant system became active, it moved some content out of the production workflow, resulting in severe processing delays. The error was corrected and content was returned to the production workflow.

Solution:

In order to prevent a similar situation and impact in future, the change deployment process has been modified to include specific checks for this type of condition. Additional automated monitoring has been put in place to identify this and similar types of unexpected behaviour. Finally, the dormant parts of the system in question will be removed entirely next month.

An incident was declared today at 09:15 am

Incident Description:

An incident has occurred which affected eClips content from the first edition dated 2^nd May 2013. Content from some pages is missing from the eClips feed. This issue has mainly affected regional content and a small number of national pages.

Cause:

The cause is unknown at this time & NLA engineers are looking into the issue.

Expected Resolution Time:

Unknown at this time. Our engineers are working to resolve the issue.

Next Update:

Please expect the next update by 10:15am

 Incident Description:

A publisher supply failure for Independent.co.uk is currently occurring. We have escalated this to the publisher and are working to resume supply as soon as possible.

Cause:  Publisher supply failure.  

Expected Resolution Time: 

The resolution time is unknown at this time.

 Next Update:

 Please expect an update by 12:00

Incident Description:

A fault is causing a complete supply failure for the following News International titles:

Thesun.co.uk

Thescottishsun.co.uk

Thetimes.co.uk

No new or updated articles for these titles will be received while this issue is occurring.

Cause:  The cause of the incident is unknown at this time.

Expected Resolution Time:

The resolution time is unknown at this time.

Next Update:

Please expect an update by 16:30

Error within the main application that distributes XML files to XML clients.  The application was not working for approximately 2 and a half hours.

Root cause:

Still being investigated and has been escalated to development teams.

Action taken to prevent further incidents:

We have put internal monitors in place to alert the on-call engineer and 9* if this particular issues occurs again.  A temporary solution to fixing the problem is in place until we have a permanent solution.

The NLA’s London data centre lost connectivity to the Internet on Friday 20^th March at 18:05. Connectivity was fully restored at 19:39. All NLA services were unavailable during this period.

Cause:

The NLA’s Internet Service Provider detected a problem with one of their upstream peers which affected some downstream customers. NLA was not one of the customers affected by this problem. While the problem was being investigated, the ISP decided to failover all customers (including the NLA) to an alternate peer at 18:00 on Friday. The failover process did not complete as expected and resulted in Internet connectivity failures for some customers - including the NLA.

Solution:

Customers of the NLA’s Internet Service Provider, which were not affected by the original upstream peering issue, were then failed back to the primary peer at 19:20. Normal connectivity was restored throughout the NLA environment by 19:39. The NLA has received assurances from its Internet Service Provider that a review of the actions taken on Friday has been completed and that the procedure for peering changes has been modified to avoid a similarly impacting event in future. Operational steps associated with changes in this procedure will be scheduled for testing during a maintenance window soon.