There was a clustering of IT-related problems, outages, and apologies last week in the government and banking spheres. However, we'll start off this week’s IT Hiccups review with a story of how a problem with a new terminal operating system at the Port of New York and New Jersey that has become a boon to the port's competitors at the Port of Virginia and elsewhere.
Port Authority of New York and New Jersey Tells Shippers to Come Back
It is hard to overestimate the importance (pdf) of an effective, fully-functioning terminal operating system (TOS) to a port’s attractiveness to shippers, and therefore, to the port’s profitability. And when a TOS doesn’t work well, it doesn’t take long for shippers to decide to go somewhere else, as the Port Authority of New York and New Jersey—the busiest on the East Coast despite still being in recovery from the ravages of Hurricane Sandy—is finding out to its dismay. The Port of Virginia, among others, is delighting in the shift.
As part of a US $3 billion investment by the Port Authority and New Jersey, Maher Terminals, one of the largest multi user container terminal operators in the world, began on 20 April to upgrade its Navis TOS over several phases at Maher’s Elizabeth, N.J., facility. The project's planners originally expected that the upgrade would be completed by 8 June. The early phases of the upgrade seemed to go well, but the final phase has proved troublesome.
According to a joint Maher-Navis statement issued on 20 June, the system’s “operations has encountered some unexpected issues” which “have led to delays.” The companies stated that they were committing “all available resources to identify and resolve” the technical issues involved, and expected the issues to be resolved shortly.
The statement also said, “With the implementation of new systems, there is always a risk of initial declines in productivity as new operating procedures and processes are streamlined into the operation.” This was a hint that the companies saw that some of the operational problems were caused by the shippers themselves, and to ensure the hint wasn’t missed, the statement added, “Noticeable improvements are already being realized as users adjust to new systems and processes.”
Apparently the “blame the customer” excuse for the port's problems didn’t go over very well, especially as the TOS-related issues continued for not days, but for several weeks. As a result, the Port of Virginia as well as other East Coast ports have seen an increase in their container traffic, a story in the Hampton Roads, Virginia, paper Daily Press reported last week. The paper said that Hapag-Lloyd, a German shipping line that operates150 ships, even went so far as to urge “its rail and local cargo customers on July 26 to seek out alternative ports” and also told its customers that the issues in NY/NJ were being made worse by a labor and trucking shortage.
The Port of Virginia successfully migrated to the Navis SPARCS N4, the same TOS that the Port Authority of New York and New Jersey is having trouble with, last year.
A story in today’s Wall Street Journal adds insights into the problems being encountered at the Port of New York and New Jersey. It says that “some trucks endured waits of an extra four to five hours for routine jobs that require one hour on a good day” and that “ships have been diverted to nearby terminals in New Jersey and Staten Island, causing more delays.” Thousands of containers worth millions of dollars are still stuck at the port, many of which contain holiday goods that are stores are anxious to receive.
One trucking company executive was quoted in the Journal as saying, “There's no word for it other than ‘hell’… I've been in business thirtysome-odd years, and this is the most stressful time I've had.”
A joint Maher-Navis statement last week said the companies had “determined that the real-time interactions between the various [TOS] system components deployed in the container yard were not operating as designed” and that they created a temporary fix: certain automated features were turned off and would be phased back in “on a controlled basis.” When they would be phased in, the companies didn't say. The companies also claimed that service had returned to “acceptable levels during the past several weeks, albeit at reduced volume.”
In the June joint-statement, the companies stated that they had expected the Port of New York and New Jersey to be returning to its previous “exceptional performance” soon. Somehow, I don’t think shippers will settle for the currently touted “acceptable” service; other Port Authorities such as those in Virginia, Baltimore, and Boston will likely keep reminding the shippers that their ports aren’t having any problems.
There is a neat real-time animation of a TOS in action at the Bremerhaven port for those interested.
Banking IT Systems Have a Bad Week
Gross understatement alert: Banking customers took it on the chin last week.
We start off with reports today that Westpac Bank customers in Australia and New Zealand are having trouble with their online and mobile banking services for a second straight week. Last Thursday and Friday, the bank apologized for intermittent connectivity problems, while today it apologized for intermittent slowdowns. The bank said last week that it didn’t know what the cause of the problems was: it hadn’t performed any updates and it wasn’t experiencing a cyber-attack, two of the usual culprits. Westpac said it would ensure that no one was out any bank fees because of the problems.
Also in Australia, the National Australia Bank apologized today to customers who were experiencing connectivity issues with its online and mobile banking services. It too denied experiencing a cyberattack, but didn’t give any further explanation for its problems. NAB, which is in the midst of a 10-year IT system modernization effort, suffered another major system issue a few weeks ago.
NAB suffered a separate black eye, but in the U.K. A week ago, customers of Clydesdale and Yorkshire banks, which are owned by NAB, discovered that they were unable to access the banks’ online services. The banks originally said it had nothing to do with their IT systems, and urged customers to check with their local ISPs. However, it soon turned out that NAB had failed to renew its domain names on time, a story at Computer World UK reported. Interestingly, the banks haven’t admitted that this oversight was the cause of their customers’ access problems, instead reiterating the previous excuse, Computer World stated.
Also in the UK, the Telegraphreported that payment processor Streamline, which the Telegraph stated “handles half of all face-to-face card transactions in Britain,” had trouble processing payments made between 26 and 28 July. As a result, an unknown number of customers may “have not received settlement for transactions processed” with their banks over those days. “We apologise for any inconvenience caused,” the processor said, although it didn’t promise to ensure any bank fees caused by the settlement issues would be covered.
Finally, in Vermont, the State Treasurer’s Office said that a processing problem at TD Bank, which is the state’s bank of record, would be delaying by a day electronic payment of retirement checks to 7059 retirees of the Vermont State Teachers’ Retirement System. TD Bank said that it would cover any late charges, overdrafts or any financial penalties assessed caused by the delay. TD Bank said it “apologizes for the inconvenience.”
US Government IT Systems Have Another Bad Week
A few weeks ago, I wrote about a number of U.S. state governments that suffered serious IT meltdowns. Last week, there was more of the same.
We start off with a mainframe system upgrade a week ago Sunday at New York State’s Department of Motor Vehicles that didn’t go as planned. A hiccup ended up taking the department’s computer system offline for several hours on Monday morning. A story at the Times Union quoted a DMV official as saying that, “Although the system was fully tested after the upgrade was completed, an issue presented itself under this morning's high transaction volumes that was not detected during testing.”
Even after the DMV’s system came back up, it was said to be operating very slowly. It wasn't until Tuesday that the system resumed normal operation.
Also on Monday, a network connectivity problem prevented the Texas Department of Motor Vehicles from registering or titling vehicles, a Star-Telegram story reported. That problem was also fixed on Tuesday.
On Tuesday, a problem with a mainframe computer in Georgia “caused the computer systems at [the Department of Driver Services], the Department of Revenue, the Department of Human Services and the Office of the Secretary of State to crash,” WGCL, CBS television's Atlanta affiliate, reported. The effects from the crash were still being felt on Wednesday. The cause of the problem was not disclosed.
Then on Thursday, the city of Alamogordo, New Mexico, said that all of the city’s e-mail and scheduled appointments between 19 and 31 July were apparently lost “while technicians were transitioning from one email system to another,” the Alamogordo Daily News reported. In addition, city employees can’t currently access their e-mail or their online calendars because of the “glitch.” The city said it has a backup system, but it also failed. There is no date being given when the situation will be rectified.
Also on Thursday, the computer system used by Nevada's Welfare System was reportedly out for the entire day. No reason was provided for the outage, other than that it was an “internal” issue, whatever that means.
Finally, on Friday, the computer system supporting all Oklahoma state agencies, including telephone services, went down because of a power failure. The system was successfully rebooted Friday night, but child support payments made by the Oklahoma Department of Human Services are going to be delayed by about a week because of the outage.
And the reason for the power outage? Apparently, “someone inadvertently hit the emergency shut off system.” Who that “someone” is has not yet been revealed.
Also of Interest…
Victoria Australia 000 Emergency System Goes Down for the Fifth Time in Six Weeks
Fat Finger Error Adds US$18 Billion to Kemper Corporation Capitalization
Glitch at M&G Accounting, UK's Biggest Bond Fund Manager, Hits 32 000 Investors
Indiana ISTEP Testing Glitches Didn’t Hurt Much (We Think)
Photo: Captain Albert E. Theberge/NOAA
Robert N. Charette is a Contributing Editor to IEEE Spectrum and an acknowledged international authority on information technology and systems risk management. A self-described “risk ecologist,” he is interested in the intersections of business, political, technological, and societal risks. Charette is an award-winning author of multiple books and numerous articles on the subjects of risk management, project and program management, innovation, and entrepreneurship. A Life Senior Member of the IEEE, Charette was a recipient of the IEEE Computer Society’s Golden Core Award in 2008.
