Description of issue
All times in CEST.
In the morning of Oct 1st, 2012, severe login issues were reported in Genium INET. The system stopped processing reference data queries from the users at ca 07:00. The queries hung and received no response. The sessions querying for data eventually timed out and the sessions disconnected. These users reconnected and tried again, adding further hung queries to the system.
Due to the fact that a significant amount of users were experiencing login completion delays, TSN Nordic Operations decided to halt markets.
The NASDAQ OMX Commodities markets were in open state since 08:00 and was suspended ca 08:26.
The Swedish, Danish and Finnish fixed income markets were suspended (before open) at ca 08:27. The financial derivatives markets were suspended (before open) at ca 08:54.
After the issue was resolved, the markets were monitored until the login situation was sufficient for market opening.
The commodities markets opened 10:00, remaining markets opened 11:00.
Root cause of the incident
The root cause was an issue in the middleware that routes transactions between Genium INET processes, which caused a transaction to hang.
Plan for future prevention of the problem
On Monday night Oct 1st mitigation actions were made which included adjusting middleware parameters and increasing manual monitoring. The weekend Oct 4th-5th, the middleware flow control parameters will be reconfigured.
Finally we deeply apologize for the inconvenience the incident on October 1st has caused you.
Stockholm, 3rd Oct 2012
Gunilla Hellqvist
Vice President Nordic Operations
