Last Tuesday, Japan's Mizuho Bank, the country's third largest by market capitalization, began experiencing problems with its internal computer systems. ATMs malfunctioned and bank deposits could not be processed, which meant salaries for workers of employers which used the bank couldn't be paid. Foreign exchange transactions also stopped.
The bank's outages increased the already high stress levels of its Japanese customers, who were already trying to cope with the aftermath of the earthquake, tsunami, and word of radiation problems at a Japanese nuclear power facility.
At first blush, it was assumed that the bank's IT problems were associated with the earthquake and subsequent power outages, but the bank said they were not. However, Mizuho also admitted on Wednesday that it didn't completely understand the cause of the problems, the Wall Street Journalreported. It suspected the problems were tied to an increase in bank transactions by its customers.
On Thursday, the bank was able to restart its ATM system. By then, some 440,000 transactions valued at 570 billion yen ($7.2 billion) had not been processed, reported this Bloombergstory.
On Friday, the situation wasn't much better. The bank still did not know exactly what had caused the system malfunctions, although Mizuho believed that it was a surge in direct deposit transactions at some branch offices in Tokyo, this story at NIKKEI.com reported. The story reports that the bank stated that "about 620,000 salary payment transactions worth Y125.6 billion weren't processed Friday."
The bank also decided on Friday to bring down all of its 38,000 ATM machines for the next three days in order to restore its banking systems. About 1 million ATM users were estimated to be affected by the shut-down
On Sunday, NIKKEI.com reported that Mizuho Bank executives announced that the banking problems would persist until at least Tuesday. In the NIKKEI.com article, the bank seemed positive that its IT problems did indeed have its roots in the surge in customer direct deposits. NEKKEI.com also reported that by Sunday:
"The lender's system glitch ... [has] affected more than 1 million cash transfer orders worth about Y830 billion ($10.3 billion) so far."
"270,000 transactions that were placed at the bank earlier this week, were processed, but 890,000 transactions including salary payments have been not yet processed."
There are now stories today (here and here) at the Wall Street Journal that "... about 3,500 of its ATMs at 440 branches will be functioning from Tuesday morning for cash deposits and withdrawals ... But ATMs outside its branches will remain closed."
The WSJ also reports that Japan's Financial Services Agency is considering "disciplining" Mizuho Bank for its on-going technological problems.
The WSJ goes on to state that:
"On Sunday, Mizuho had operational-level talks with its archrivals, Sumitomo Mitsui Banking Corp. and the Bank of Tokyo-Mitsubishi UFJ, the core banking unit of Mitsubishi UFJ Financial Group, over how they could help Mizuho with transaction settlements, in case Mizuho's troubles drag on, another person familiar with the matter said. Underscoring the gravity of the situation, the FSA has asked Mizuho's rivals to help it through this crisis."
The problem with this approach is to make sure that if the two other Japanese banks process some of Mizuho's transactions, the added load doesn't take down those banking systems as an unintended consequence.
I recall hearing one Japanese resident saying during a TV interview last week that the lesson to be learned is always to have a cash reserve at home in case of an emergency.
Worth remembering, I think.
Robert N. Charette is a Contributing Editor to IEEE Spectrum and an acknowledged international authority on information technology and systems risk management. A self-described “risk ecologist,” he is interested in the intersections of business, political, technological, and societal risks. Charette is an award-winning author of multiple books and numerous articles on the subjects of risk management, project and program management, innovation, and entrepreneurship. A Life Senior Member of the IEEE, Charette was a recipient of the IEEE Computer Society’s Golden Core Award in 2008.