What's Frying the Electrical Systems On BART Trains? (ieee.org) 250
Tekla Perry writes: Earlier this month, BART engineers shut down a substation in hopes that the closure would quiet the power surges that were frying the electrical propulsion equipment on BART cars -- a peak of 40 in just one day in February. The shutdown seemed to solve the problem, but BART officials weren't sure they'd really found the answer. Yesterday, the power surges popped up again, on an entirely different section of tracks, damaging 50 cars before BART closed off that section, rerouting passengers onto buses. Track inspections yesterday revealed nothing, and BART reports that it has reached out to experts around the country and asked them to fly in and help solve the mystery. Do you have a theory? Note: BART is the 5th-busiest heavy-rail rapid transit system in the U.S.
Hackers ruining our infrastructure (Score:5, Insightful)
Re: (Score:2)
I think it's the Chinese.
Re: (Score:2)
Could be; especially since a similar problem is affecting the D.C. Metro trains. Perhaps the Iranians have reverse-engineered Stuxnet and pointed it back at us?
Re: (Score:2)
I think it's the British [imdb.com]
Re:Hackers ruining our infrastructure (Score:4, Funny)
Which nation-state is sponsoring the hacking crew that will inevitably be blamed for this issue?
Well dammit we'd know already if they hadn't gone dark with all the unbreakable encryptions on their iPhones!
Re: (Score:2)
Maybe they should have used that encryption on the infrastructure instead.
Re: (Score:2)
Yes, let's encrypt all the power lines.
Re:Hackers ruining our infrastructure (Score:5, Funny)
Re: (Score:2)
I was going to suggest North Korea, but they are screwing themselves enough at the moment. Logic suggests this one will be attributed to China.
Re: (Score:2)
Re: (Score:2)
I-squared-L? (Score:5, Interesting)
could there perhaps be enough inductance in the multi-motor systems that it is generating its own connect/disconnect/connect surges? try isolating those DC motor controllers from both the track and the motors with some diode stacks and snubber caps.
Re: I-squared-L? (Score:3)
Oh come on. It couldn't possibly be that simple. I write software for a living and even I know to look for back emf first.
Re: (Score:2)
Nine out of ten times the solution is the "oh c'mon, they can't be THAT dumb!" one.
Re: I-squared-L? (Score:5, Funny)
I had a boss once who hired a consultant and was angry when the consultant told him to do the obvious thing.
"You know what a consultant is?" he groused. "Someone who borrows your watch to tell you the time."
I thought about this for a moment. "Yeah, but what's he supposed to do if you're standing there with the watch on your wrist and you don't know what time it is?"
Re: (Score:2)
And eleven out of ten times, they are in fact THAT dumb.
(And the other time it's an off-by-one erro
Comment removed (Score:4, Informative)
Re:I-squared-L? (Score:5, Insightful)
Unless they have recently upgraded a mass of cars or track, why does it suddenly start happening now ? The system has been in use and fairly reliable for a long time. I have no proof, but the suspicious borderline paranoid inside me is screaming that someone is hacking at the electrical infrastructure that feeds the Bart system, and the problem lies outside their direct observation, and is likely with PG&E's supply system to Bart. PG&E has demonstrated the disregard for maintenance, monitoring, and the security incompetence in the last few years to allow for something like this. It will likely take some outside support for Bart to prove the failures don't lie inside their infrastructure to get PG&E to even begin to look at their own systems.
Re: (Score:2)
Re: (Score:3)
Couple that with time and mechanical wear.. The bionding between rail segments is wearing out, (and/or being stolen for scrap metal value.) A few intermittant rail to rail connections based on rail movement, (as when a fully loaded train passes over), would inflict a world of hurt on the 1000volt DC electronics.
Re:I-squared-L? (Score:4, Interesting)
It is almost certainly a supply problem.
You can't defer maintenance forever (Score:5, Insightful)
BART already tweeted the reason [nytimes.com] behind the breakdowns:
From @SFBART:
BART was built to transport far fewer people, and much of our system has reached the end of its useful life. This is our reality.
BART has been continually expanding while deferring maintenance on the rest of the system, and that policy has finally come home to roost -- much of their infrastructure is over 40 years old and they can't defer maintenance forever. But by continually expanding, they've made themselves too big to fail (and they've gotten more counties on the hook to keep the service running), so they'll get bailed out one way or another.
Awe, come on now (Score:2)
I don't buy that one for a second, it makes way too much sense. I think it's people urinating on the 3rd rail, and we need another season of Mythbusters to prove it!
Re: (Score:2)
But by continually expanding
The number of people a transit system serves is determined by the choices of those people, not the administrators of the system.
Re: (Score:2)
But by continually expanding
The number of people a transit system serves is determined by the choices of those people, not the administrators of the system.
It's obviously both, but in this case THEIR wording does refer to the administrators since by "expanding" they mean "adding additional stops". Kind of a "duh" comment, really...
Re: (Score:2)
That's a bit simplistic. If you expand a system, it will automatically attract more riders if the cost to the rider remains the same. And therein lay the rub. The rational basis of the decision to expand the system is a cost/benefit one, even if the system is publicly subsidized. The rationale for public subsidies is based on there being market externalities, but once you factor in those externalities the benefits have to outweigh the costs.
That's where politics comes in. Service expansion is popular, b
Re: You can't defer maintenance forever (Score:2, Informative)
Yes, just think if actual personal automobile costrs were directly born by the users, they would run screaming.
Instead it gets widely subsidized in invisible ways.
Re: (Score:2)
Re: (Score:2)
Re: (Score:2)
Re: (Score:2)
OK, so what are those external costs? And how about the external - and direct - costs of mass transit?
It's people who cause costs. Why not save money by closing the city and forcing everyone to go elsewhere? When you understand why that would be a bad idea, you may begin to understand why simply analysing the costs of things is a poor idea.
Eventually you may begin to understand why cities work better with mass transit, despite it being a "cost". Hint: you may want to consider things that cause and reduce pr
Re: (Score:2)
FTFY
The only "force" that is involved here is the fact that federal, state, and local tax payers are forced to subsidize a transit system that most don't benefit from and will never use.
Re: (Score:2)
FTFY
I think the main problem is you're a complete and utter smeg head. There's no equivalence to those two things.
The only "force" that is involved here is the fact that federal, state, and local tax payers are forced to subsidize a transit system that most don't benefit from and will never use.
Oh I see you're one of those very silly people who think they live in an isolated island where nothing outside affects you. Even if you never use the train, it takes a lot of cars off the roads which makes the roads
Re: (Score:2)
Everyone would benefit even more if the subsidies that currently went to BART would be invested in road upgrades and new highways.
See, robbing me at gunpoint and then giving me back $5 to catch a bus home doesn't mean that I'm better off than if I hadn't been robbed at all.
You are a testament to the quality of a modern Oxford education.
Re: (Score:2)
Everyone would benefit even more if the subsidies that currently went to BART would be invested in road upgrades and new highways.
[citation needed]
And tell me, where would those highways go?
See, robbing me at gunpoint
Taxes are a thing. Don't like them? Go live in the Libertarian Paradise of the Congo.
doesn't mean that I'm better off than if I hadn't been robbed at all.
Then try moving somewhere there are no taxes. See how much better off you are. Toodles!
You are a testament to the quality of a modern Oxford
Re: (Score:2)
Plus congestion and the lost time it causes.
In that respect, public transport has a positive externality due to the fact that most people riding the subway aren't simultaneously driving a car.
Re: You can't defer maintenance forever (Score:3, Interesting)
Conservatives hate talk about externalities. Their so called free market relies on you not noticing, and God forbid holding them accountable for, pollution, congestion, oil subsidies and tax breaks, a huge military devoted to guarding overseas private assets (oil wells and shipping lanes), poisoning of water supplies, seizure of land for private gain, mandatory insurance at predatory prices (which is somehow not ok with healthcare, except for the predatory pricing part), inefficient use of resources (autom
Re: (Score:2)
That's not an "externality", since it's the drivers themselves that bear the cost of congestion.
Re: (Score:2)
An individual driver doesn't cause congestion for himself, he causes it for everybody else. And as I already pointed out, people on trains aren't causing it at all.
Re: (Score:2)
We're discussing whether drivers pay for the costs that their choice of transportation incurs. They do, as a group, even when it comes to congestion, since the cost of congestion balances out between drivers (if I slow down your commute, you slow down my commute, so I do pay this "externality").
Of course, congestion would be almost non-existent if it weren't for government mismanagement of transportation and roads
Re: (Score:2)
Re: (Score:2)
Hasn't been used in many years.
Those are not "externalities" since they are costs drivers themselves bear and pay for.
Acid rain comes overwhelmingly from power plants, not automobiles.
Personal transportation is only a small contributor to atmospheric carbon, and the difference between trains and cars is fairly small. With electric cars, cars are actually better than trains.
Re: (Score:2)
Re: (Score:3)
Re: (Score:2)
If it's anyone's choice, it comes down to the number of jobs that are created in the region. Use of highways and transit systems is an implication of that.
And it happens that we haven't yet dealt with retarded policies that don't just allow, but depend upon, continuous growth.
If this happened in your body, it would be diagnosed as cancer. Put it in a region, and we kid ourselves that it's progress.
Re: (Score:3)
Re: (Score:3)
When I could BART to work I did so because I could sit back and read instead of gritting my teeth in traffic. The trip time was about the same, but it was no longer wasted time.
Re: (Score:3)
BART is about equally priced to driving for a single person, and significantly more expensive for multiple people (i.e. versus carpool). People use it because, despite it's various frustrations, it's genuinely more convenient than driving, not because it's less expensive.
BART is the only way to get across the Bay during rush hour without waiting at least 2 hours to drive across the bridge. That's NOT an exaggeration.
Re: (Score:2, Informative)
"BART is about equally priced to driving for a single person, and significantly more expensive for multiple people (i.e. versus carpool)."
If youre driving into the city, this is not true at all unless your employer has free parking. Most of the cost of diving into the city is paying for parking, which is anywhere from $30 - $80 a ~day~.
The vast majority of BART ridership and hell-like experience is into the city in the morning and out again in the evening...if youre traveling from any other BART stop to any
Re: (Score:2)
Unless those services are priced well below actual market costs - then it's "buying" riders.
It's amazing that people (a) ignore that roads are subsidised and (b) the free market is not a magical solution which solves the preoblem of limited space in any sane way. If you don't have mass trasit, then you rely on roads and when they get full, things block up and you start to lose serious amounts of productivity. Fortunately the city administrators who you so hate have a job of keeping the city running rather t
Re:You can't defer maintenance forever (Score:5, Insightful)
BART already tweeted the reason [nytimes.com] behind the breakdowns:
That's not a reason - it's a complaint (and a beg for more money). Maybe they do need more money, but if the cause of the surges remains a mystery, then by definition they don't know the reason.
I'm sure they don't have any SCADA systems exposed to the Internet (right?), but "old stuff" is just a guess. And there are some good "old stuff" guesses elsewhere on this story.
How is BART supposed to update trains... (Score:2, Informative)
...when all their money is going to high salaries and benefits for union employees [mercurynews.com]?
Over 200 BART employees earned over $200,000 a year in total compensation...
Re:How is BART supposed to update trains... (Score:5, Insightful)
...when all their money is going to high salaries and benefits for union employees [mercurynews.com]?
Over 200 BART employees earned over $200,000 a year in total compensation...
Those look mostly like executives, which are definitely NOT Union positions. The Unions are the ones trying to get some of that executive salary down to the real workers.
Re:You can't defer maintenance forever (Score:4, Informative)
It's not just that - BART was simply never meant to be operating on the scale it does today. When BART was built, the creators envisioned a system that would serve about 100,000 people per week and choke points such as the Transbay Tube were built accordingly. Naturally, as the population increased, upgrades had to made. This worked for a while, but eventually lack of funding for serious overhauls caught up with the
constantly increasing ridership [imgur.com]. Maximum capacity is heavily influenced by the fact that sections like the tube are single line, with no easy way to expand to double or triple. BART could theoretically be a 24/7 system, but as things stand now their engineers need every minute of the nightly downtime they have to service a rapidly aging rail system.
The rails already in place are almost at capacity, with a train crossing over them every 2 minutes. With the tech booms of the last decades, there's been an even bigger spike in these numbers. Over the last decade alone, passenger alightings at some stations have more than doubled. On busy days, the BART system now serves 25 times more riders than originally envisioned. There's some money for additional trains, but that can only do so much. Eventually, we are going to need to spend money on either more parallel tracks, cars, and bigger platforms or just a new system altogether.
Their administrators are simply being realistic about the situation we're in [vox.com]
Re: (Score:2)
Bullshit. The substation that was first identified as a problem is a mere months old.
Re: (Score:2)
A 40 year old electrical system? Oh woes me!
Seriously though while you may be right electrical systems, especially the always on kind which don't cycle can last for a hell of a long time much longer than 40 years. But there are some peculiarities though. Taking maintenance into account a lot of failures are very overt and instantly show the source of the problem through arcing or flashovers in the piece of failed gear. Typically you know straight away where the source of the problem is, and the simplicity o
Re: (Score:2)
BART already tweeted the reason [nytimes.com] behind the breakdowns:
From @SFBART:
BART was built to transport far fewer people, and much of our system has reached the end of its useful life. This is our reality.
BART has been continually expanding while deferring maintenance on the rest of the system, and that policy has finally come home to roost -- much of their infrastructure is over 40 years old and they can't defer maintenance forever. But by continually expanding, they've made themselves too big to fail (and they've gotten more counties on the hook to keep the service running), so they'll get bailed out one way or another.
That's a political answer to justify asking for more money - it is not a technical answer that should be accepted on /.
Free trade (Score:2)
Damn those Chinese capacitors! Damn you to hell!
I suspect ... (Score:5, Interesting)
Throw some power quality analyzers on various sections of the track and watch the system's transient voltage response with power sources in various configurations.
Re:I suspect ... (Score:5, Funny)
Transients caused by the inherent imperfect third rail to car contacts causes a ringing (oscillation) in the system ...
Oh, sure - blame the homeless.
Re: (Score:2)
Seems most probable cause.
I don't know the technical details of BART systems sitting in India. Is it supplied by 60 Hz AC and the rectifiers are installed on the cars that run on DC motors? Usually these power electronics devices build in 1980s are robust enough. We have gained experience now and toughened the circuit in modern drives, but I doubt it be the case in 80s. Any adverse interaction with filters or may be some kind of ferroresonance can cause high voltage and damage such equipment.
vitrification of grounds and more rain (Score:5, Interesting)
So the A & B cars having inductions motors seem to be fine, while the problem seems to be confined to the C cars having the DC motor. That's one difference.
Also, what else has changed? Take a look at wunderground to see that the Bay Area is having a wet season.
Why would the C cars have been mostly fine all along and having trouble now?
So there's charge building up in the DC motors that they can't handle and that makes them blow out. The charge has nowhere to go. What controls the flow of charge? Grounding. What can go wrong with grounding? Good grounds can go bad when a lot of discharge causes the sand in the soil to vitrify (melt into glass) after discharges and lightning strikes have been shooting through it for decades. Better grounds can unexpectedly form when more highly conductive paths form up. The AC induction motors will suffer a power loss but can handle the charge jumping back & forth in unexpected ways, while the DC motors can't.
Add it all up. This has to be a grounding problem aggravated by the wet season, and an underlying assumption that once you sink a ground it's good forever. It isn't.
Re: (Score:2)
Turns out the problem is frying fuses on the A&B cars and thyristors on the C cars.
Re:vitrification of grounds and more rain (Score:4, Interesting)
This has to be a grounding problem aggravated by the wet season
Grounding problems are the cause of problems in the rail cars for many different reasons. My favourite is when an Australian train company decided to go green and implement regenerative breaking on all their equipment without any forethought. After purchasing an entire fleet of fancy new green trains they found the power goes out at the train station every time the train approaches.
But really this is not helped by the fact that ground forms an important part of the electrical path in a rail system, and not just a safety mechanism as in most other cases.
Re:vitrification of grounds and more rain (Score:4, Interesting)
Throw together something to actually log what is going on. Even something like this, with the right voltage divider plugged into a little rugged PC like this or similar would probably work and you could create the logging code with Labview.
You're in university or work in research aren't you? In the rest of the world we just hire off the shelf logging equipment specifically designed for the task and move on.
Real Time Monitoring (Score:5, Insightful)
I'm truly surprised that they don't have intensive real time monitoring with sensors through their whole system.
Proper engineering and maintenance of such a critical system demands it.
Re:Real Time Monitoring (Score:5, Insightful)
Re: (Score:2)
Re: (Score:2)
If you're surprised that BART is lacking something remotely logical and/or basically requisite you don't understand BART or the non-Tech Industry side of the Bay Area.
BART only funds repairs, no technical improvements what-so-ever. The most recent redesigns are ways to fit more standing people in each car since it raises rider count. In fact, the only technological changes I can think of having happened to BART in the 25 years I lived in the Bay Area was the upgrade to the Clipper Card system.
Bathrooms in s
Re:Real Time Monitoring (Score:4, Interesting)
I'm truly surprised that they don't have intensive real time monitoring with sensors through their whole system.
Outside of a power generation company looking at maybe one or two substations away from their plant you'll find power infrastructure monitoring to the level that could aid predictive maintenance is non-existent. Even in utilities it's non-existent. Even in companies that don't run their equipment into the ground it's non-existent.
Re: (Score:3)
Maybe its because they use DC instead of AC (Score:2)
Savages!
Back when I was a kid (Score:2)
On the other hand, that was the first time I saw Black Sabbath's Paranoid album. Didn't hear
Re: (Score:2)
Keep in mind, back then BART was brand spankin new, ultra-reliable, much cheaper than gas.
President Nixon rode it during the 1972 election.
http://blog.sfgate.com/thebigevent/2012/08/16/rail-to-the-chief-when-nixon-rode-bart/ [sfgate.com]
Re: (Score:2)
Wow. I remember reading about BART in my 6th or 7th grade social studies textbook. That was in 1976-77 or so.
Eels on a train (Score:5, Funny)
It all started 2 years ago when a student majoring in EE took an exotic canoe trip on the Amazon. One day the canoe capsized while he was studying and his book sank to the bottom. Thee eels read voraciously and learned about series parallel wiring of batteries. An idea was born.
And so now we have Electric Eels on a Train!
Re: (Score:2)
Is that better or worse than snakes on a plane?
Re: (Score:2)
Not sure, but it seems to be more expensive at any rate.
Re: (Score:3)
Explain Toles' Cartoons (Score:2)
perhaps someone is testing (Score:4, Interesting)
Thirty years too late... (Score:2)
BART is finally coming to Silicon Valley. I'm so excited. Meh...
http://www.vta.org/bart/ [vta.org]
Location for the non article readers. (Score:2)
For those outside the US... (Score:2)
Re: (Score:2)
Bay Area Rapid Transit - San Francisco's Bus and light rail system
San Francisco's transist system is called Muni. BART covers more than just San Francisco and is most common in the East Bay though it has been expanding into the South Bay over the last 10-15 years.
I've got a theory (Score:2)
It must be bunnies, bunnies, it must be bunnies! /Buffy
Poor guys (Score:2)
I wouldn't want to swap with those poor engineers working overtime to find out what's wrong here. as one myself, I know how desperate this can feel.
On the other hand, there is no better feeling than finally finding the root cause. The better the more unconnected it seems to be at first sights. That's what you got your degree for.
Homeless crazy poop (Score:2)
All downtown BART stations have to be periodically taken offline so that human excrement can be cleaned out of the escalators.
control computers probably spygrading to Windws 10 (Score:3, Funny)
Perhaps the newest computers controlling the system have forcibly "upgraded" themselves to Windows 10.
Re:control computers probably spygrading to Windws (Score:4, Interesting)
Re: (Score:2)
Re: (Score:2)
Nah, it's the aliens:
http://www.dailymail.co.uk/sci... [dailymail.co.uk]
http://www.collective-evolutio... [collective-evolution.com]
It should be a breeze to deactivate BART trains then :-)
Re: (Score:2)
I think there's a movie franchise that already explains this and it's not aliens...
This is beside the Pacific, right? This is clearly the giga-monster that is a result of all the atomic tests in the Pacific 60 years ago approaching the west coast. Good time to take a holiday in Florida, you'll only risk wet feet from the rising sea levels there.
Re: Maybe (Score:2)
Re: (Score:2)
So somebody reversed the polarity on a flux capacitor, which destabilized the warp field matrix?
Re: (Score:2)
That, and carnies.
You can't trust 'em. Nomads. Smell like cabbage. Small hands.
Small hands... short fingers [vanityfair.com].
Re:Solar flares (Score:5, Interesting)
My "go to" excuse for any electronic problems is "sunspots, or stray cosmic rays". However.....
The Sun has been very quiet for the last several weeks, and Solar Cycle 24 is on a steep downward trend. I expect that we'll begin an extended Solar Minimum by the end of 2016 which may last 3-4 years (the "average" is 2 years, and the last Solar Min was nearly 3). I also expect that the next few solar cycles will be fairly quiet. Perhaps not Maunder Minimum quiet, but probably Dalton Minimum quiet, or nearly so.
You should visit www.spaceweather.com periodically to keep up to date on this.
Also; it's going to get somewhat chilly by and by.
Re: (Score:2)
Re:Who stands to benefit? (Score:5, Interesting)
Considering that it's aged - don't rule out aging wire insulation that no longer does its job.
Re: (Score:2)
Indeed. Running power systems is not easy, and railway power is especially difficult (at least houses and substations are typically stationary). Some good research has been done on this in Europe - and I am sure experts are available for reasonable fees. (Maybe that is the actual problem?)