Bell Rings

The Stock Market Bell Rings, Computers Fail, Wall Street Cringes

By NATHANIEL POPPER

Problems with technology have at times roiled global financial markets, but the 223-year-old New York Stock Exchange has held itself up as an oasis of humans ready to step in when the computers go haywire.

On Wednesday, however, those working on the trading floor were left helpless when the computer systems at the exchange went down for nearly four hours in the middle of the day, bringing an icon of capitalism’s ceaseless energy to a costly halt.

The exchange ultimately returned to action shortly before the closing bell, and stocks continued trading throughout the day on other exchanges, like the Nasdaq and BATS Global Markets.

The disruption nonetheless rattled investors, who already had reason to be on edge, considering the Greek debt crisis and an overnight market rout in China. The benchmark Standard & Poor’s index of 500 stocks ended the day down 1.7 percent.

Wednesday provided other reminders of the fragility of automated systems that are doing jobs people once handled. An apparently unrelated technical problem grounded United Airlines flights for nearly two hours on Wednesday morning. The homepage of The Wall Street Journal was also down for part of Wednesday.

“When we traded physically we didn’t have these problems, but this is the world that we live in,” said Ted Weisberg, a trader with Seaport Securities who has been on the floor of the New York Stock Exchange for nearly 50 years.

Computer technology has revolutionized the trading of stocks in recent decades, making it faster and more efficient. New powers have emerged, including the Intercontinental Exchange, or ICE, a commodities and derivatives trading platform based in Atlanta that acquired the New York Stock Exchange for $8.2 billion in 2013.

But there have been hiccups along the way. The Nasdaq stock market went down for three hours in 2013 because of a software bug. The year before, software at Knight Capital Group went awry, leading to errant trades that resulted in losses of $440 million.

The problem Wednesday at the New York Stock Exchange is likely to revive a debate about how regulators can make the markets more resistant to computer failings.

At the same time, technology has created alternatives that allow trading in the United States to go on, even when its most prominent exchange does not. Stocks listed on the New York Stock Exchange were traded on Wednesday in the New Jersey and Chicago data centers of upstart exchanges like BATS, which is based in Lenexa, Kan.

“There are 11 exchanges in the U.S. and one of them went down — I had 10 other places to go,” said Ryan Larson, the head of stock trading at RBC Global Asset Management. “The fragmentation of the market helped us today.”

Indeed, the overall volume of stock trading on Wednesday appeared to be largely consistent with the amount of daily trading earlier in the week. Other exchanges owned by the New York Stock Exchange, like its options exchange, operated normally on Wednesday throughout the shutdown.

The problems at the New York exchange emerged soon after the opening bell rang at 9:30 Wednesday morning, when orders for several smaller stocks failed to go through, traders on the floor said.

The initial malfunctions affected only a small number of stocks, and the exchange appeared to get the problem under control, but the difficulties reappeared, on a more widespread basis, later that morning, traders said.

On a call with other members of the industry, as the problem grew worse, N.Y.S.E. employees expressed “panic” and a recognition that they had “lost control of the system,” according to a person on the call, who spoke on the condition of anonymity.

Not long after that call, at 11:32 a.m., the exchange announced that it was shutting down all trading, and said on its website that “additional information will follow as soon as possible.”

On the floor, traders said the exchange gave them no warning before making the move.

“I was about to order my lunch and when I turned around the screens all went blank,” said Peter P. Costa, a trader on the floor with Empire Executions.

In the silence that followed, traders on the floor, looking for information, flocked to a ramped area of the huge complex on Wall Street where employees of the exchange are located. The traders were told that the exchange had been updating software before the trading day began, and that the software had contained a bug that could not be fixed without shutting down the system.

The exchange’s employees on the floor proceeded to manually cancel around 700,000 trades that were in the system when it was shut down, according to Mr. Costa. Once they had finished, the exchange rebooted its system, a process that took around 45 minutes, Mr. Costa said.

Trading ultimately resumed around 3:10 p.m., less than an hour before the closing bell.

Several hours later, the exchange put out a statement with few details on what happened, other than to attribute blame to a “configuration issue.”

Throughout the day, federal regulators and law enforcement agencies monitored the situation for any sign of an outside attack and ultimately saw nothing indicating anything other than an internal software error.

President Obama was briefed on the exchange shutdown, a White House aide said, adding that officials there and at the Treasury Department were monitoring the situation.

Mary Jo White, the chairwoman of the Securities and Exchange Commission, which oversees the nation’s securities markets, said in a statement, “We are in contact with N.Y.S.E. and are closely monitoring the situation and trading in N.Y.S.E.-listed stocks.”

The New York Stock Exchange and its new Atlanta owner, the Intercontinental Exchange, are likely to face questions after the shutdown. ICE, which makes most of its money from automated derivatives trading, has been reducing staff at the New York exchange after stating a goal of cutting $450 million in expenses from the joint company.

Recent cuts have hit the exchange’s technology and communications teams, which came under pressure on Wednesday.

After the failure began, it took over half an hour for the exchange to broadcast even basic information about the shutdown, to allay concerns that it might have been caused by a malicious attack.

An ICE spokeswoman, Sara Rich, denied any suggestion the cuts had an effect. “Since the acquisition, we have been investing to build new systems and technology expertise, and the new trading platform will begin rolling out later this year,” she said.