Evening Standard Delivery Delay

Problem:

The following content has not been delivered to the NLA for production into eClips:

• Evening Standard (publication date 16.07.07)

Cause:

Associated Newspapers has informed the NLA that they are experiencing a significant IT failure which is affecting delivery of content to all 'downstream' clients. 

Solution:

The NLA production staff are working closely with Associated to ensure that this issue is resolved as quickly as possible.   Further updates will be sent when more information is available.

Update on Monday, July 16, 2007 at 12:26 by Matt Groshong

Associated Newspapers has resolved their IT issue and all pages of the Evening Standard have now been processed and are available in eClips.

We apologise for any impact this may have had on your clients.

Manila Connectivity

After further investigation our IT staff have discovered the cause of the latency which has affected our connectivity to the Manila production facility this week.  A major submarine cable that connects Japan with the U.S. has been damaged and is causing delay in a large portion of trans-pacific traffic from ASIA. 

The two providers that we use for Internet connectivity in Manila unfortunately utilise this routing which is why we experienced traffic latency earlier this week.  Fortunately, BT (the provider of the FT network infrastructure) is no longer routing traffic across this route and therefore we have seen normal performance while using the FT's network.

The cable is being worked on as a matter of urgency and is expected to be repaired on the 23rd of May.  Until this is repaired we will continue to use the FT network and expect no further disruption to the eClips service due to this issue.

For further information regarding how BT routes ASIA Internet traffic, please see this site:  http://www.bt.net/info/asia_pacific.shtml

If you require any further information or have additional questions, please don't hesitate to send us a mail.

Manila Production Facility Connectivity Issues

Problem:

The issues experienced early this morning concerning connectivity to Manila have continued throughout the day and we are now seeing latency on our backup line as well.  We feel that due to this latency all content produced in Manila may be in danger of not meeting SLA targets for delivery.

Cause:

From our investigations it would appear that there is a global problem with Internet latency between the Philippines and the USA (which is how Internet traffic is routed from ASIA to the UK).  This latency is causing the NLA traffic to be delayed and we feel puts our production of content in danger of delay.  We are working with the Manila production facility and our Internet providers but at this stage do not have an idea when these issues will be fully resolved. 

Solution:

We have implemented further redundancy with our connectivity to Manila by utilising the FT's network and our testing shows that we currently have sufficient capacity to meet deadlines.  We have also put our production facility in Chennai on 'standby' for a full Disaster Recovery fail over for all processing as a precautionary measure.

Changes:

As we know there is a potential to not meet SLA targets we would like to offer the opportunity for all of you to scan the following content this evening:

• Daily Telegraph
• Financial Times
• Guardian
• Herald
• Independent
• Lloyd's List
• Metro
• Press & Journal
• Scottish Sun *
• Scottish Times *
• Sun *
• The Scotsman
• Times *

*Note: We would ask you to review (below) clause 7.5 in the new NLA PCA licence, which we will treat as applicable to all clients; you must notify us in writing before scanning any News International Publications content

Next Update:

A service review will be conducted at 10PM to confirm status and we will send out further information by 10:30PM.  We will also send out further communication regarding this issue tomorrow.

NLA PCA Licence Excerpt:

7.5 Following expiry of the Transitional Period, in the event of Delayed Delivery you may as an interim measure for a maximum period of 24 hours from the target delivery times set out in the Service Level Schedule exercise the rights granted to you under a subsisting Electronic Licence to subject the Cuttings affected by Delayed Delivery  to Text and/or Image Scanning and deliver them pursuant to the rights granted to you under an Electronic Licence provided always that the following conditions are met:

7.5.1 where the affected Cuttings derive from publications forming part of your Database Repertoire and the Electronic Repertoire, you must notify us in writing (email being sufficient for these purposes) within 12 hours of the relevant target delivery time set out in the Service Level Schedule, specifying the title and pages of the affected Cuttings that you have  subjected to Text and/or Image Scanning  under this clause 7.5; and

7.5.2 where the affected Cuttings derive from News International Publications, as these publications form part of your Database Repertoire but not part of the Electronic Repertoire, you must notify us in writing (email being sufficient for these purposes) before exercising the rights under this clause 7.5 specifying the title and pages of the affected Cuttings that you intend to subject to Text and/or Image Scanning; and  by no later than the last day of the Viewing Period, delete from all systems and records which you own or which are under your control the images comprising those Cuttings which you subjected to Text and/or Image Scanning under this clause 7.5, and delete the associated text no later than the end of the Archive Period.

Update on Thursday, May 17, 2007 at 11:29 by Matt Groshong

As most of you will probably have noticed, there was no negative impact on NLA production last evening.  If you have not seen the statistics, please view them here.  The work performed to ensure our connectivity was sufficient for production load was very successful and we now have further confidence that our network redundancy is robust during this issue.

We continue to operate our production traffic over the FT's network but we are also in the process of providing our other network providers with further detail on the issues we are experiencing in hope of resolving this issue.   

We do not expect any further disruption to the NLA service as a result of these issues and will provide further update once this is resolved entirely.

If you have any questions or concerns please feel free to send us an email.

Content Delivery Issues, Wednesday 16th of May

At approximately 4AM this morning, we received an escalation from our production facility in Manila informing us of a potential issue with file delivery to the NLA database in London.  This issue was investigated and no cause was found.  Files were flowing into London, but at a slower than normal pace.  The result of this slow replication is that 278 post processed (files that required reprocessing due to quality issues) articles missed the 4AM deadline.  These files have now completed loading into the NLA database.

After further investigation this morning, the NLA IT team has discovered that the line that connects the Manila office to the Internet was experiencing sporadic connectivity issues.  We have now failed over to the backup connection and all content is replicating normally.  We will investigate the cause of this disruption and endeavour to prevent this from happening in the future.

We apologise for any impact this had on your business.