Manila Connectivity

After further investigation our IT staff have discovered the cause of the latency which has affected our connectivity to the Manila production facility this week.  A major submarine cable that connects Japan with the U.S. has been damaged and is causing delay in a large portion of trans-pacific traffic from ASIA. 

The two providers that we use for Internet connectivity in Manila unfortunately utilise this routing which is why we experienced traffic latency earlier this week.  Fortunately, BT (the provider of the FT network infrastructure) is no longer routing traffic across this route and therefore we have seen normal performance while using the FT's network.

The cable is being worked on as a matter of urgency and is expected to be repaired on the 23rd of May.  Until this is repaired we will continue to use the FT network and expect no further disruption to the eClips service due to this issue.

For further information regarding how BT routes ASIA Internet traffic, please see this site:  http://www.bt.net/info/asia_pacific.shtml

If you require any further information or have additional questions, please don't hesitate to send us a mail.

Manila Production Facility Connectivity Issues

Problem:

The issues experienced early this morning concerning connectivity to Manila have continued throughout the day and we are now seeing latency on our backup line as well.  We feel that due to this latency all content produced in Manila may be in danger of not meeting SLA targets for delivery.

Cause:

From our investigations it would appear that there is a global problem with Internet latency between the Philippines and the USA (which is how Internet traffic is routed from ASIA to the UK).  This latency is causing the NLA traffic to be delayed and we feel puts our production of content in danger of delay.  We are working with the Manila production facility and our Internet providers but at this stage do not have an idea when these issues will be fully resolved. 

Solution:

We have implemented further redundancy with our connectivity to Manila by utilising the FT's network and our testing shows that we currently have sufficient capacity to meet deadlines.  We have also put our production facility in Chennai on 'standby' for a full Disaster Recovery fail over for all processing as a precautionary measure.

Changes:

As we know there is a potential to not meet SLA targets we would like to offer the opportunity for all of you to scan the following content this evening:

• Daily Telegraph
• Financial Times
• Guardian
• Herald
• Independent
• Lloyd's List
• Metro
• Press & Journal
• Scottish Sun *
• Scottish Times *
• Sun *
• The Scotsman
• Times *

*Note: We would ask you to review (below) clause 7.5 in the new NLA PCA licence, which we will treat as applicable to all clients; you must notify us in writing before scanning any News International Publications content

Next Update:

A service review will be conducted at 10PM to confirm status and we will send out further information by 10:30PM.  We will also send out further communication regarding this issue tomorrow.

NLA PCA Licence Excerpt:

7.5 Following expiry of the Transitional Period, in the event of Delayed Delivery you may as an interim measure for a maximum period of 24 hours from the target delivery times set out in the Service Level Schedule exercise the rights granted to you under a subsisting Electronic Licence to subject the Cuttings affected by Delayed Delivery  to Text and/or Image Scanning and deliver them pursuant to the rights granted to you under an Electronic Licence provided always that the following conditions are met:

7.5.1 where the affected Cuttings derive from publications forming part of your Database Repertoire and the Electronic Repertoire, you must notify us in writing (email being sufficient for these purposes) within 12 hours of the relevant target delivery time set out in the Service Level Schedule, specifying the title and pages of the affected Cuttings that you have  subjected to Text and/or Image Scanning  under this clause 7.5; and

7.5.2 where the affected Cuttings derive from News International Publications, as these publications form part of your Database Repertoire but not part of the Electronic Repertoire, you must notify us in writing (email being sufficient for these purposes) before exercising the rights under this clause 7.5 specifying the title and pages of the affected Cuttings that you intend to subject to Text and/or Image Scanning; and  by no later than the last day of the Viewing Period, delete from all systems and records which you own or which are under your control the images comprising those Cuttings which you subjected to Text and/or Image Scanning under this clause 7.5, and delete the associated text no later than the end of the Archive Period.

Update on Thursday, May 17, 2007 at 11:29 by Matt Groshong

As most of you will probably have noticed, there was no negative impact on NLA production last evening.  If you have not seen the statistics, please view them here.  The work performed to ensure our connectivity was sufficient for production load was very successful and we now have further confidence that our network redundancy is robust during this issue.

We continue to operate our production traffic over the FT's network but we are also in the process of providing our other network providers with further detail on the issues we are experiencing in hope of resolving this issue.   

We do not expect any further disruption to the NLA service as a result of these issues and will provide further update once this is resolved entirely.

If you have any questions or concerns please feel free to send us an email.

Content Delivery Issues, Wednesday 16th of May

At approximately 4AM this morning, we received an escalation from our production facility in Manila informing us of a potential issue with file delivery to the NLA database in London.  This issue was investigated and no cause was found.  Files were flowing into London, but at a slower than normal pace.  The result of this slow replication is that 278 post processed (files that required reprocessing due to quality issues) articles missed the 4AM deadline.  These files have now completed loading into the NLA database.

After further investigation this morning, the NLA IT team has discovered that the line that connects the Manila office to the Internet was experiencing sporadic connectivity issues.  We have now failed over to the backup connection and all content is replicating normally.  We will investigate the cause of this disruption and endeavour to prevent this from happening in the future.

We apologise for any impact this had on your business.

Delivery issues – Telegraph magazines

We have recently experienced delays with delivery of the Daily Telegraph Magazine and Sunday Telegraph Stella Magazine. The matter has been escalated and the NLA Production team is in close contact with publisher counterparts.

The cause of the problem has been identified as an internal issue with Telegraph system set-up; some changes are being made this week and we expect to have secured timely delivery for the material due to be published on April 21^st and 22^nd. Further updates will be issued as appropriate.

Apologies for any negative impact this has caused.

XML delivery delay

We had some further production problems last night, caused by a failure in the software that creates the XML feeds. The result was that XML feeds delivery did not start till shortly after 3pm.

The failure was triggered by the increased volumes following addition of new regionals. Our testing over the previous week had been successful, but data volumes last night exposed a design fault. This has now been rectified.

We are acutely aware that recent service performance has been poor and are taking all steps we can to revert to our normal standards.

Regards

Andrew Hughes

MD, Digital

IT Report

Issue: DBCTRL was down between 00:01 and 03.10 FOR XML CLIENTS ONLY.

Customer Impact: YES

Cause: Unhandled timeout operation in DBCTRL program

Engineers: DC.

Outline.

DBCTRL was failing to distribute client XML. The error in the error log showed the message DBOut GetObjectsToDistribute Timeout expired. DBCTRL showed no other errors. The SQL server was checked and was found to be operating normally. Another delphi error indicated a possible sharing violation, so CIFS was cycled on FAS-B to force drop any ownership conflicts, however the error persisted.

We rolled back the version of DBCTRL that was released today. However the old version also reproduced the error . Further debug steps were taken using tools we have, however these showed nothing untoward happening on any of our systems. All NLA code was found to be operating normally, and all systems were nominal.

Looking at the OBJDISTREADY table (which logs object ids for distribution against PCA org ids), this had some 71000 rows in it. This is an abnormally high figure, so I took the following course of action:

Stop DBCTRL
Remove all INBOUND feed lines from DBCTRL (to stop new data being loaded into the DB)
Create Table DCTEMP
Move 71000 rows from OBJDISTREADY to DCTEMP
Truncate Table OBJDISTREADY
Moved rows with OBJ_ID = 34 back into OBJDISTREADY (4000 rows)
Start DBCTRL

At this point, the DBOUT error disappeared and all objects for ORG_ID 34 were distributed.

I then sequentially went down the list of ORG_IDs in the table DCTEMP (select distinct org_id ..... ) and manually pumped each ORG_IDs rows into the OBJDISTREADY table. As each one distributed successfully, I would load the next candidate ORG_ID. All files were successfully distributed.I stopped DBCTRL again, restated the inbound client feeds, and started it up. There were no further errors and inbound XML was being loaded with the appropriate outbound XML being distributed.