NLA eClips Service Incident - Report

Problem:

Some content was loaded into the eClips database and distributed to clients later than usual on Tuesday 27^th May 2008.

Cause:

A fault with one of the external Internet links, from an eClips production site in India, resulted in corrupt files being transmitted to London. These corrupt files were not loaded into the eClips database at the first attempt.

Solution:

The problematic link was shut down and traffic from the affected production site was distributed over the remaining external links. The corrupt files were then retransmitted and any outstanding content was loaded and distributed by 5:30am.

Further investigation into the exact nature of the fault is ongoing. The implementation of additional monitoring processes will minimise the time required to react if a similar event should occur in future.

NLA eClips Service Incident: REPORT

Problem:

The eClips Service has recently experienced two service incidents which have impacted the performance of the service.  The first incident happened Friday 16th and the second occurred Monday 19th.  Both incidents occurred during the early morning which is the peak time for usage of the eClips service. 

Cause:

After further investigation and testing, NLA engineers have discovered that these incidents were caused by a recent service release which was deployed at 19:00 Thursday 15th as part of a service enhancement. 

Service testing before the release of this enhancement unfortunately did not uncover a performance issue.  This performance issue was only found when the eClips service experienced peak traffic load on Friday 16th. 

Following the first incident on the 16th, NLA engineers were unable to determine that the service enhancement had caused the incident.  However, when on 19th a similiar incident occurred at a similiar time of day, additional information was obtained that pointed to the service release as the most likely root cause of both incidents.

Solution:

As soon the service release was determined to be the cause of the incidents, NLA engineers 'rolled back' the changes (Monday 19th at aproximately 9:30).  Once the change was removed from the service, normal operation returned and this issue has not occurred again.

NLA service release testing processes have been reviewed following these incidents and new procedures are now in place to ensure that testing more accurrately reflects the level of performance releases must cope with in the live environment. 

The release of this most recent service enhancement has now been delayed until further testing and redevelopment can occur. 

NLA eClips Service Incident: CLOSURE

NLA eClips Service Incident Announcement

Time:

A service incident was declared today at 08:52.

Incident Description:

The NLA eClips service was experiencing a performance issue which was preventing some users from viewing clippings and slowing the service for most users. 

Cause:

It is believed that this issue was related to the incident that happened on Friday 16th.  Both issues are now believed to be a result of a system change made on Thursday 15th. 

Expected Resolution Time:

This incident is now closed.  All NLA eClips services are performing normally.

Next Update:

Please expect an incident report within 24-hours.

NLA eClips Service Incident: UPDATE

NLA eClips Service Incident Announcement

Time:

A service incident was declared today at 08:52.

Incident Description:

The NLA eClips service is experiencing a performance issue which is preventing some users from viewing clippings and slowing the service for most users. 

Cause:

It is believed that this issue is related to the incident that happened on Friday 16th.  Both issues are now believed to be a result of a system change made on Thursday 15th. 

Expected Resolution Time:

UPDATE:  NLA engineers are now rolling back the change that was deployed on Thursday 15th.  Services will be returned to normal just as soon as possible. 

Next Update:

Please expect the next update at 09:45.

NLA eClips Service Incident: ANNOUNCEMENT

NLA eClips Service Incident Announcement

Time:

A service incident was declared today at 08:52.

Incident Description:

The NLA eClips service is experiencing a performance issue which is preventing some users from viewing clippings and slowing the service for most users. 

Cause:

It is believed that this issue is related to the incident that happened on Friday 16th.  Both issues are now believed to be a result of a system change made on Thursday 15th. 

Expected Resolution Time:

NLA engineers are actively working on this issue to try and improve service performance and will plan a roll-back of the recent changes made as soon as it is viable.

Next Update:

Please expect the next update at 09:30.