Manila Production Facility Connectivity Issues
Wednesday, May 16, 2007 at 15:28 Problem:
The issues experienced early this morning concerning connectivity to Manila have continued throughout the day and we are now seeing latency on our backup line as well. We feel that due to this latency all content produced in Manila may be in danger of not meeting SLA targets for delivery.
Cause:
From our investigations it would appear that there is a global problem with Internet latency between the Philippines and the USA (which is how Internet traffic is routed from ASIA to the UK). This latency is causing the NLA traffic to be delayed and we feel puts our production of content in danger of delay. We are working with the Manila production facility and our Internet providers but at this stage do not have an idea when these issues will be fully resolved.
Solution:
We have implemented further redundancy with our connectivity to Manila by utilising the FT's network and our testing shows that we currently have sufficient capacity to meet deadlines. We have also put our production facility in Chennai on 'standby' for a full Disaster Recovery fail over for all processing as a precautionary measure.
Changes:
As we know there is a potential to not meet SLA targets we would like to offer the opportunity for all of you to scan the following content this evening:
- Daily Telegraph
- Financial Times
- Guardian
- Herald
- Independent
- Lloyd's List
- Metro
- Press & Journal
- Scottish Sun *
- Scottish Times *
- Sun *
- The Scotsman
- Times *
*Note: We would ask you to review (below) clause 7.5 in the new NLA PCA licence, which we will treat as applicable to all clients; you must notify us in writing before scanning any News International Publications content
Next Update:
A service review will be conducted at 10PM to confirm status and we will send out further information by 10:30PM. We will also send out further communication regarding this issue tomorrow.
NLA PCA Licence Excerpt:
7.5.1 where the affected Cuttings derive from publications forming part of your Database Repertoire and the Electronic Repertoire, you must notify us in writing (email being sufficient for these purposes) within 12 hours of the relevant target delivery time set out in the Service Level Schedule, specifying the title and pages of the affected Cuttings that you have subjected to Text and/or Image Scanning under this clause 7.5; and
7.5.2 where the affected Cuttings derive from News International Publications, as these publications form part of your Database Repertoire but not part of the Electronic Repertoire, you must notify us in writing (email being sufficient for these purposes) before exercising the rights under this clause 7.5 specifying the title and pages of the affected Cuttings that you intend to subject to Text and/or Image Scanning; and by no later than the last day of the Viewing Period, delete from all systems and records which you own or which are under your control the images comprising those Cuttings which you subjected to Text and/or Image Scanning under this clause 7.5, and delete the associated text no later than the end of the Archive Period.
Matt Groshong
As most of you will probably have noticed, there was no negative impact on NLA production last evening. If you have not seen the statistics, please view them here. The work performed to ensure our connectivity was sufficient for production load was very successful and we now have further confidence that our network redundancy is robust during this issue.
We continue to operate our production traffic over the FT's network but we are also in the process of providing our other network providers with further detail on the issues we are experiencing in hope of resolving this issue.
We do not expect any further disruption to the NLA service as a result of these issues and will provide further update once this is resolved entirely.
If you have any questions or concerns please feel free to send us an email.