Follow Slashdot stories on Twitter

 


Forgot your password?
Close
typodupeerror
×
IT Technology

NYSE Mayhem Traced To a Staffer Who Left a Backup System Running (bloomberg.com) 82

Posted by msmash from the turn-it-off-and-on-again dept.
An anonymous reader shares a report: More than 700 miles from Wall Street, the New York Stock Exchange's backup data center on Cermak Road in Chicago is supposed to safeguard US markets, standing by at all hours in case disaster ever strikes the world's largest venue for trading shares. When markets are closed, it participates in a well-worn routine, with NYSE staffers turning on and off systems to ensure everything works. But heading into Tuesday, an NYSE employee failed to properly shut down Cermak's disaster-recovery system -- leading to a disaster.

That human error, described by people with direct knowledge of NYSE's internal operations, is what triggered wild market swings when trading opened Tuesday morning in Manhattan. The chaos affected more than 250 companies including Wells Fargo, McDonald's, Walmart and Morgan Stanley, in some cases sending stock prices swinging by 25 percentage points in a matter of minutes. The episode has prompted the exchange to cancel thousands of trades at a cost that's still being determined. Meanwhile, market professionals and day traders are rattled and waiting for the exchange to elaborate on what it publicly called a "manual error" involving its "disaster recovery configuration."
This discussion has been archived. No new comments can be posted.

NYSE Mayhem Traced To a Staffer Who Left a Backup System Running More

NYSE Mayhem Traced To a Staffer Who Left a Backup System Running

Comments Filter:

  • there DR plan needs day to day manual work? (Score:3)

    by Joe_Dragon ( 2206452 ) on Thursday January 26, 2023 @09:02AM (#63241949)

    there DR plan needs day to day manual work?

    • Re:there DR plan needs day to day manual work? (Score:5, Insightful)

      by The-Ixian ( 168184 ) on Thursday January 26, 2023 @09:06AM (#63241963)

      This is exactly what I was thinking... Humans fail in all kinds of ways.

      I really hope that the employee doesn't take too much heat for this if it turns out to be an honest mistake.

      This sounds like a systemic problem that needs to be addressed at a much higher level.

      • Re:there DR plan needs day to day manual work? (Score:4, Insightful)

        by chromozone ( 847904 ) on Thursday January 26, 2023 @10:13AM (#63242117)

        A few years ago the NYSE vanished for a few hours and a "processor" malfunction was blamed.
        Last week air travel stopped and a "corrupt file" was blamed
        US gas pipeline (Colonial Pipeline) was crippled in 2021 and the mysterious perps were described as "dispersed" a few weeks later.
        When global; sociopaths are at work flaming wars in Europe and Asia I'm not quick to blame code and hardware quirks .

        The West is already fighting World War 3 with Russia, former White House Russia advisor Fiona Hill says
        https://www.businessinsider.co... [businessinsider.com]

        • Re:there DR plan needs day to day manual work? (Score:4, Insightful)

          by Chris Mattern ( 191822 ) on Thursday January 26, 2023 @11:01AM (#63242235)

          "When global; sociopaths are at work flaming wars in Europe and Asia I'm not quick to blame code and hardware quirks ."

          From which I can deduce that you don't work with computers professionally.

        • When global; sociopaths are at work flaming wars in Europe and Asia I'm not quick to blame code and hardware quirks .

          No, but we can blame the person demonstrating a sociopathic level of observation bias for being utterly clueless.

          Your conspiracy theories are lame. Conspiracy theories need actual benefits for the conspirators and you've demonstrated none, even if incidents such as the ones you list were restricted to times of flaming wars in Europe and Asia.

    • Comment removed (Score:5, Funny)

      by account_deleted ( 4530225 ) on Thursday January 26, 2023 @09:24AM (#63242007)
      Comment removed based on user account deletion

      • "Abby someone."

        "Abby someone... Abby who?"

        "Abby... normal."

      • Omg, they voted you +5 funny but I've had my backup guy who actually knew and understood backups get a message every day for months that the backups weren't running because the tape heads needed cleaning. He'd report every week it was ok.
        Then we needed a customer restore of course.

        I asked him later a few months after he got fired why he didn't just run the tape cleaner. We had them, he was there at least once a week anyway. All he had to do was put in the tape cleaner one time and it'd be fine for anothe

      • Re: (Score:3)

        by ceoyoyo ( 59147 )

        I worked in a research centre that decided they should have backups. So they bought a full rack sized robotic tape backup unit. Thing was delivered, but they balked at paying $50k for the software. That was okay though, since the tapes they'd ordered went mysteriously missing.

        We were touring the server room for another centre several years later and there were a stack of tapes in once corner. They said they mysteriously been delivered a few years ago but nobody knew who had ordered them.

    • Re: (Score:2)

      by gweihir ( 88907 )

      Even worse, they apparently had no fail-safes against this entirely expectable scenario where an operator makes a mistake in the manual process. Design by incompetents.

    • Some needs to buy a book on DevOps ...

    • there DR plan needs day to day manual work?

      Their

    • Cut these people some slack. I don't know the details of the NYSE systems, but it's probably more complex than two servers in your basement running a web site that no one visits.

      The error was found relatively quickly and fixed. The advantage of traditional transactions (rather than blockchain) is that they could be easily reversed . I'm guessing that the total dollars lost was zero or relatively small. Some HFT may have lost paper profits when their transactions were reversed, but nothing of value was injur

    • "there DR plan needs day to day manual work?"

      If you have billions to lose, don't trust that some minimum wage guy pushes the right buttons.
      4, 8, 15, 16, 23, 42

    • Re: (Score:2)

      by stooo ( 2202012 )

      DR cannot be automated, because disasters are random events affecting in non-previsible ways.

  • of a Tom Clancy novel?

  • Economic system (Score:3)

    by VeryFluffyBunny ( 5037285 ) on Thursday January 26, 2023 @09:14AM (#63241981)
    So... nothing in the real world actually happened & people panicked because of a computer glitch, which had real-world consequences. A bunch of gamblers freaked out & this is what it costs. Are you sure this is how you want to manage the economy?

    • It's a mistake to think the NYSE is the economy.

      • But what happens on the NYSE has consequences for the real economy, i.e. ordinary working people. Remember 2007-8?

        • Re: (Score:2)

          by ceoyoyo ( 59147 )

          2008 didn't have much to do with stock exchanges.

          About the worst thing that can happen to ordinary people when stock exchanges screw something up is that the people who entrusted their retirement fund to someone who lets a computer decide to sell their stuff take a bit of a loss (which is someone else's gain).

    • This is mostly a non-issue from an economic standpoint. Some large firms that can afford it somewhere will lose some money. Much of it will be lost to each other. Everything will be fine.

      People panicked rightly so because there was an appearance of the loss of a large sum of money. If my company stock were suddenly to get cut in half, that would have significant impact on my life plans. Sure I'd still be fine. But many people will need to significantly adjust their lives should their investments los

      • Yeah, 2007-8 didn't have any effect on the real economy, did it?
        • 2007-8 represented actual financial losses. Banks lent money against houses and they weren't going to get paid back. They issued bonds to collateralize those debts. Many were held by smaller investors. When those houses were sold, the sellers had all this extra cash to pump up the economy. Now that the mortgages were worthless and couldn't cover the value of the houses, those who had overspent in the past or those in already marginal situations had to make drastic changes due to *actual* losses. That'

  • They couldn't 'fix' opening prices for the day... (Score:5, Interesting)

    by Fly Swatter ( 30498 ) on Thursday January 26, 2023 @09:14AM (#63241983) Homepage
    So we witnessed an actual free trading market for a whole minute?

    Of the stocks I follow, several hit my low mark alerts but none hit a high price alert - hmm.. Make me wonder if the current prices are artificially propped up. :/

    • Well, no. You watched a market behave erratically when the ecosystem of facilitated trades was no longer handled appropriately by the technology. You saw a broken market, not a free one. The trades in the hopper were collectively intended for whatever you want to call the current system when it's stable.

    • So we witnessed an actual free trading market for a whole minute?

      Of the stocks I follow, several hit my low mark alerts but none hit a high price alert - hmm.. Make me wonder if the current prices are artificially propped up. :/

      You caught them, they accidentally started the whole exchange but forgot to turn on the price fixing server under Jim's desk for a minute, that explains it.

      How are you knowledgeable enough to set price alerts but also this dense? Did Robinhood do that for you?
      Did you buy anything based on those alerts? Did you sell what you consider to be inflated prices? If you figured out the big secret, you're making money right? Right?

      How about this, if you believe in conspiracy theories, do not play with your retiremen

      • Nothing to catch, it is well known the market makers are there to keep the market liquid and sort of stable. This event caused several market makers to be left out of the trading process. No conspiracy, it is just the way it is.

    • LOL "fix" opening prices. Tell us you don't have a clue about stock trading without telling us you don't have a clue about stock trading.

  • Thats funny (Score:4, Interesting)

    by Revek ( 133289 ) on Thursday January 26, 2023 @09:28AM (#63242027)
    The problem prevented two market makers from being able to setup up their orders and as a result for a very short period of time the market ran without their 'making' and stocks jumped up and down and triggered a halt. It would be interesting to see a week without their 'making' being involved.
    • It would be chaos and lots of tears.

    • So, what you're saying is: look at the swing, divide by 2, and that's how much you're typically getting screwed with every stock transaction.

    • Interesting, but relatively predictable. Might take more than a week, but the system would stabilize. The current activity is stabilized around one set of paradigms, including the activity of these market makers. In their absence, the nature of the trades would change as traders deal with the fluctuations. It's "mostly" a self-correcting system in that regard. You might not like where it stabilizes, but it "will" stabilize.

    • It would be interesting to see a week without their 'making' being involved.

      Presumably it would be chaos. These "market makers" are often doing little more than rationalising the after hours trading. We can do without them just fine. ... If it were the early 1900s.

  • Minutes later, this is done intentionally (Score:3)

    by gestalt_n_pepper ( 991155 ) on Thursday January 26, 2023 @09:31AM (#63242033)

    For profit!

    • Done intentionally for profit? A profit that doesn't exist caused by having to cancel trades and burn a lot of goodwill in the process?

      What's your idea of a "loss", winning the lottery?

  • At least not by the person blamed in the story. Whoever designed this system without any fail-safe against this entirely to be expected scenario is simply incompetent and fully responsible for the results.

    • why is the system not master / master or slave / master with an easy way to swap master / slave modes?

      Why does the DR system need to be turned on and off each day?

      • Re: (Score:3, Interesting)

        by flirek ( 1000761 )
        there is physical distance between main and backup site. that means there are physical limitations how fast transaction can be done if you require master/master configuration (you are limited by speed of signal propagation). to ensure both sides to have same data (sync replication) you need 2x time distance. that means high latency. and stock exchange wants to have low latency as much as possible. so only design is to have main / DR (DR being async copy).

        • Re: (Score:2)

          by gweihir ( 88907 )

          Still no excuse for the manual processes. And you can have a slave running a bit behind. In fact, that is the usual mode.

        • Unless you're using a satellite link or a long-distance underwater cable, round trips are generally measured in tens of milliseconds. If you don't believe me, try pinging a site that you know is on the other side of the country, such as mit.edu or jpl.nasa.gov and take a good look at the results. Yes, there are times when transit time is important, but I find it hard to believe that tracking stock transactions is one of them.
      • Because that's racist, so they had to get rid of the redundancy mechanism.

        • Re: (Score:3)

          by gweihir ( 88907 )

          Racist against whom? White slaves? Black slaves? Yellow slaves? Blue slaves? Green slaves? Because there have been slaves of all skin colors that ever were around.

          • Blue was the host, black was the master, and gray was the slave. Then we went to SATA and everyone is now equal. Happy?

      • Re: (Score:2)

        by gweihir ( 88907 )

        Exactly. Somebody messed up in system architecture and design here. And now some poor slob got hit because he made a mistake that is far too easy to make.

      • Haha, maybe because someone found that offensive: https://m.slashdot.org/story/3... [slashdot.org]

  • Let's hope for a serious fine. And start checking the backup plans of other similar institutions.

  • I'm skeptical of this explanation. Why would the NYSE allow the market to go live with a backup DR system active? Seems like there should be some circuit breakers to automate taking these systems offline before production data is introduced. I'm terrified that the reality holding together our financial markets relies on manual processes.

  • Let's say you've got a decently sturdy home - i.e. one not built in the last 20 years, but solid and dependable. You want a new toilet in the upstairs bath, so you get somebody in to do that. They don't seal the wax properly around the base, and the leaking creates swelled wood around it, as well as damage that could cause mold in the materials below. Now that sucks.

    That doesn't work, and now you've got to get some repairs completed as well as having the actual toilet fixed again. But the house is still oka

  • ...where they interview people who say, "we can't believe they bought the staffer story."

  • Meanwhile, market professionals and day traders are rattled and waiting for the exchange to elaborate on what it publicly called a "manual error" involving its "disaster recovery configuration."

    Routine scenarios such as this should be automated. No exception. Triggering it can be manual (when that makes sense!), but the process from standby to hot, and hot back to standby should not involve a human.

  • I'm almost impressed. I've made my own share of 'mess-ups', but I've never screwed something up so badly it made international news.

Slashdot Top Deals

If all else fails, lower your standards.

Close