News International Non-delivery of Sunday's Content

Problem:

We experienced a Publisher Delivery Failure of a substantial number of books from The Sunday Times. Most of the content was recovered during the production run, but we are still attempting to recover some of the content. The books still missing are:

• Sport
• In Gear
• Appointments (Scot)
• Ecosse (Scot)
• Home (Scot)
• In Gear (Scot)
• News Review (Scot)
• Sport (Scot)

Cause:

The failure was due to an error in the publisher's FTP server.

Solution:

We work very hard with the publishers to try and reduce as many potential stumbling blocks as we can, but some things are out of our control. Our production team has been in contact with News International this morning and has been assured that NI will do their best to ensure this type of failure will not happen again.

KPI Reporting:

The Sunday Times has been removed from the monthly KPI calculations for Sunday 8th July in accordance with Section 3.2.2 of Appendix 2 of the eClips Licence.

NLA PCA Licence Excerpt:

SERVICE LEVEL SUSPENSIONS

The NLA shall not be responsible for a failure to meet any Service Level to the extent that such failure is attributable to any of the following events ("Service Level Suspensions") provided in each case that the NLA shall at all times use reasonable endeavours to meet the Service Levels and shall notify the Licensee of the occurrence of these events:

3.1 Scheduled Maintenance (except that overruns from the notified or agreed downtime period will be taken into account when measuring performance against the relevant Service Levels);

3.2 any Unavailability resulting from

3.2.1 the acts or omissions of the Licensee’s local loop provider or any national or international telecoms circuit provider;

3.2.2 the acts or omissions of a Publisher (provided always that the NLA can demonstrate it has followed such escalation procedures as may be in existence with the relevant Publisher);

3.2.3 the Licensee's applications, equipment, network or facilities;

3.2.4 the acts, omissions or wilful misconduct of the Licensee, its agents and/or employees or any Permitted User;

3.2.5 reasons of Force Majeure;

3.2.6 failure attributable to the Licensee;

3.2.6.1 not authorising software upgrades required to maintain Availability. For the avoidance of doubt the scope of any software upgrades that the NLA can require the Licensee to undertake shall not require the Licensee to change its hardware

or software operating system;

3.2.6.2 failing to install new software releases provided by the NLA free

of charge; and

3.2.7 the Licensee instructing the NLA not to work on a fault or failing to notify the NLA of a fault which has come to the Licensee's attention.

Late Delivery of Files for Monday, 11th June

Problem:

We experienced a problem writing files into client FTP folders early this morning resulting in the late delivery of a substantial number of first edition files.

Delivery statistics show completion percentages for core repertoire first edition files were

• 4.46% at 01:00
• 15.52% at 01:30
• 99.95% at 02:00

Cause:

The failure was due to an incorrectly cut list from the Guardian that should not have been included in the feed in the first place.

This file caused a failure in our file distribution software. 

Solution:

We removed the offending object from the database, and following a restart of the software, the logs indicated that everything was normal. Files were again being distributed normally.

Unfortunately, there were about 4000 XML files and about 5000 IDS and XML objects waiting to be distributed. This was occurring extremely slowly due to all the loading which was going on at the same time. We therefore stopped loading files for 30 mins while the number of files awaiting distribution decreased.

Changes:

This was essentially human error and we are currently evaluating what options we have to ensure this type of failure won't happen again. We will update this entry as soon as we have a satisfactory solution.

NLA PCA Licence Excerpt:

7.5 Following expiry of the Transitional Period, in the event of Delayed Delivery you may as an interim measure for a maximum period of 24 hours from the target delivery times set out in the Service Level Schedule exercise the rights granted to you under a subsisting Electronic Licence to subject the Cuttings affected by Delayed Delivery to Text and/or Image Scanning and deliver them pursuant to the rights granted to you under an Electronic Licence provided always that the following conditions are met:

7.5.1 where the affected Cuttings derive from publications forming part of your Database Repertoire and the Electronic Repertoire, you must notify us in writing (email being sufficient for these purposes) within 12 hours of the relevant target delivery time set out in the Service Level Schedule, specifying the title and pages of the affected Cuttings that you have subjected to Text and/or Image Scanning under this clause 7.5; and

7.5.2 where the affected Cuttings derive from News International Publications, as these publications form part of your Database Repertoire but not part of the Electronic Repertoire, you must notify us in writing (email being sufficient for these purposes) before exercising the rights under this clause 7.5 specifying the title and pages of the affected Cuttings that you intend to subject to Text and/or Image Scanning; and by no later than the last day of the Viewing Period, delete from all systems and records which you own or which are under your control the images comprising those Cuttings which you subjected to Text and/or Image Scanning under this clause 7.5, and delete the associated text no later than the end of the Archive Period.

Delivery issues – Telegraph magazines

We have recently experienced delays with delivery of the Daily Telegraph Magazine and Sunday Telegraph Stella Magazine. The matter has been escalated and the NLA Production team is in close contact with publisher counterparts.

The cause of the problem has been identified as an internal issue with Telegraph system set-up; some changes are being made this week and we expect to have secured timely delivery for the material due to be published on April 21^st and 22^nd. Further updates will be issued as appropriate.

Apologies for any negative impact this has caused.

Bugzilla Rollout

We're happy to announce the roll out of our new support ticketing system, Bugzilla. We have been using Bugzilla internally for the past 18 months to track internal development and client issues and we have been very happy with the results so far. Here’s part of the Wikipedia entry about Bugzilla.

“Bugzilla is a web-based general-purpose bug tracking tool originally developed and used by the Mozilla project. Released as open source software by Netscape Communications in 1998, Bugzilla has been adopted by a variety of organizations for use as a defect tracker for both free software and proprietary products.”

“Bugzilla's notion of a "bug" is very general, and Bugzilla uses the term "bug" to track not only software bugs but also all other user-submitted tracking tickets; for instance, Mozilla.org and the MediaWiki project use it to track feature requests as well. Bugs can be submitted by anybody, and will be assigned to a particular developer. Various status updates for each bug are allowed, together with user notes and bug examples.”

Although we are rolling out this new system in response to the new requirements placed on us by the new contract and SLA, we are very excited about the new functionality it gives us.

We can now

• Assign individual ticket numbers to each PCA request
• Track the exact date and time of PCA requests
• Accurately track and monitor the time to resolve PCA issues
• Automatically route requests to the correct person without having to run all requests through the CS team (although CS will monitor all requests)
• Reports can be created to show activity by PCA, product, type of issue, etc…

It also gives the PCAs a much better view of what’s happening.

• Web interface with simple layout to view all current (and historical) issues
• Allows PCA users to check the current status of each issue
• Allows PCAs to attach files and make additional comments
• User-defined reports can be run to track activity and ticket status

Our goal is to have all PCAs who subscribe to eClips using Bugzilla by the end of March, so someone from the Client Services team will contact you this week to schedule a training session.

Data Migration Project

Tomorrow we will begin a project to move the entire body of NLA Digital data from our current fabric attached storage system (FAS) to a new storage area network (SAN). This will involve moving approximately 2TB of data from one set of drives to another. We will be moving the data in blocks to make the work more manageable, and expect the entire project to take as much as two weeks.

The data migration has been designed to have very little or no impact to our clients. During the migration window, users may experience a slight degradation in performance as we port the existing system onto the new hardware. It's hard to quantify exactly what this degradation will be until we start the process, but we will be monitoring the system and we are prepared to abort the process if necessary.

The new SAN will provide us with not only enough expansion to accommodate growth for the next 36 months (and possibly beyond), but should also provide improved performance to our existing services.

We will post updated information to the blog as the project moves along.