The Lights Have Been On At a Massachusetts School For Over a Year Because No One Can Turn Them Off (nbcnews.com) 202
An anonymous reader quotes a report from NBC News: For nearly a year and a half, a Massachusetts high school has been lit up around the clock because the district can't turn off the roughly 7,000 lights in the sprawling building. The lighting system was installed at Minnechaug Regional High School when it was built over a decade ago and was intended to save money and energy. But ever since the software that runs it failed on Aug. 24, 2021, the lights in the Springfield suburbs school have been on continuously, costing taxpayers a small fortune.
"We are very much aware this is costing taxpayers a significant amount of money," Aaron Osborne, the assistant superintendent of finance at the Hampden-Wilbraham Regional School District, told NBC News. "And we have been doing everything we can to get this problem solved." Osborne said it's difficult to say how much money it's costing because during the pandemic and in its aftermath, energy costs have fluctuated wildly. "I would say the net impact is in the thousands of dollars per month on average, but not in the tens of thousands," Osborne said. That, in part, is because the high school uses highly efficient fluorescent and LED bulbs, he said. And, when possible, teachers have manually removed bulbs from fixtures in classrooms while staffers have shut off breakers not connected to the main system to douse some of the exterior lights.
But there's hope on the horizon that the lights at Minnechaug will soon be dimmed. Paul Mustone, president of the Reflex Lighting Group, said the parts they need to replace the system at the school have finally arrived from the factory in China and they expect to do the installation over the February break. "And yes, there will be a remote override switch so this won't happen again," said Mustone, whose company has been in business for more than 40 years.
"We are very much aware this is costing taxpayers a significant amount of money," Aaron Osborne, the assistant superintendent of finance at the Hampden-Wilbraham Regional School District, told NBC News. "And we have been doing everything we can to get this problem solved." Osborne said it's difficult to say how much money it's costing because during the pandemic and in its aftermath, energy costs have fluctuated wildly. "I would say the net impact is in the thousands of dollars per month on average, but not in the tens of thousands," Osborne said. That, in part, is because the high school uses highly efficient fluorescent and LED bulbs, he said. And, when possible, teachers have manually removed bulbs from fixtures in classrooms while staffers have shut off breakers not connected to the main system to douse some of the exterior lights.
But there's hope on the horizon that the lights at Minnechaug will soon be dimmed. Paul Mustone, president of the Reflex Lighting Group, said the parts they need to replace the system at the school have finally arrived from the factory in China and they expect to do the installation over the February break. "And yes, there will be a remote override switch so this won't happen again," said Mustone, whose company has been in business for more than 40 years.
Confused? (Score:5, Insightful)
Was the school built without circuit breakers?
Re: Confused? (Score:2, Informative)
We'll, they have a few breakers it seems...
staffers have shut off breakers not connected to the main system to douse some of the exterior lights
Re: Confused? (Score:5, Insightful)
staffers have shut off breakers not connected to the main system to douse some of the exterior lights.
So why don't they use the breakers that are connected to the main system? If the clowns who installed the system did not include breakers for the lighting circuits, get some slightly more competent clowns to fit them. In my neck of the woods any local electrician could put some in in a day or two. I could do it myself if I were put to it.
there will be a remote override switch so this won't happen again
Why does it have to be remote? Put it in a frigging utility cupboard or in the basement.
Re: (Score:3)
So why don't they use the breakers that are connected to the main system?
They obviously had AI breakers installed and the AI has apparently gone rogue.
Re: Confused? (Score:5, Insightful)
Re: Confused? (Score:5, Funny)
Why does it have to be remote? Put it in a frigging utility cupboard or in the basement.
You could even make it more convenient and granular in case it happens again. For example having a per room override in a convenient location such as next to the door in the room.
Re: (Score:2)
Maybe if they turn the lights off, they won't be able to turn them back on.
Default to the off state after power-up, no control system to issue the on command.
Re: (Score:3, Insightful)
So why don't they use the breakers that are connected to the main system?
This sounds like a somewhat smart system, one that has had equipment failure and currently has no means of control. What state will they be in when you turn the system back on?
Do you know?
Are you 100% certain?
Are you willing to permanently put kids in the dark and burn your career over flicking that switch?
In my neck of the woods any local electrician could put some in in a day or two. I could do it myself if I were put to it.
I've met people like with your attitude before. I spent an entire weekend helping restart a chemical plant as a result of someone like you not at all considering the consequences of simply turning off pow
Re: Confused? (Score:4, Insightful)
For the school year your right. However July 5th 2022 I would have shut down power and started wiring up secondaries. Spent a few thousand and start bringing them back online. If they failed so be it. To control that much there has to be a relay bank. Just switch out the computer control and manually power the relays.
Problem solved temporarily. And I would charge the electricity and that repair work to the company that installed and created the software/hardware.
Schools have options for total shutdown for 2 months that businesses do not
Re: (Score:3)
I was told by a commercial electrician at a previous job that it was a bad idea to use breakers as routine switches. They are not designed or rated for the constant make/break cycles on live equipment.
He was telling me this as he was replacing breakers we routinely used to turn on/off the lights in a good sized space, and said he was in there every few years to do the same thing because for whatever reason the lights had been installed without regular switches, or they were inaccessible, or had been removed
Re:Confused? (Score:5, Insightful)
Re: (Score:2)
Or the fixtures controlled by one breaker don't map logically to one group of lights that you want to turn on/off.
Re: (Score:2)
I presume that on an intelligent lighting system which you can't control, once you flick the main breaker off the lights won't ever come on again.
Re: (Score:3)
Lighting circuits are power circuits.
While that is technically true, in facilities and maritime design conventions, "power circuits" is taken to mean "A/C power circuits other than lighting". Often, in a large enough building, there will be separate electrical panels just for lighting loads.
Re: (Score:2)
Lighting circuits are power circuits.
It is the convention in the world of wiring installations to use the term "power circuits" to refer to those which supply wall socket outlets.
Any building, even small houses, should have separate circuits for lights, power, heavy current things like cookers, and even those categories should be broken down into sub-categories like upstairs lights, downstairs lights, garage lights, upstairs power, etc etc. Each circuit should have its own fuse and/or circuit breaker - my house has about 12 circuit breaker
Re: (Score:2)
They have been using the breakers as a workaround.
Re: Confused? (Score:2)
No, they haven't - at least not comprehensively - take a look at the headline to this story, "The Lughts Have Been On ... For Over A Year"
Some teachers are pulling some breakers, but still many lights stay on.
if you flip them then the raptor fences go off lin (Score:5, Funny)
if you flip them then the raptor fences go off line
Re: (Score:2)
This is probably the real reason. There's an animated fat guy telling them they didn't say the magic word.
Re: (Score:2)
They should check with the students, maybe one of them knows Unix...
Re: (Score:2)
Re: (Score:2)
Re: (Score:2)
The issue is that more than just the lights are on the circuits. Heating, A/C, clocks, speakers.
At a *new* building, HQ for a company that rhymes with Hells Cargo, the entire HVAC system was *smart*. So freaking smart that it maintained a perfect temperature in the middle of summer ... for everywhere in this monster building, *except* the one corner of the floor where I was. There it decided to turn the heat on. Many solutions were tried by the responsible company, all failed. For about three summer mo
Re: (Score:2)
The lights are on but nobodies home.
Those that can, do. Those that can't, teach. (Score:2, Offtopic)
Remember...these are the kinds of people educating children.
Clunking switches (Score:2)
That's what you get when you go install home automation junk instead of clunking switches and copper.
Nevermind the big dark added consumption of supplying power to all those controllers 24/365....
Re: (Score:2)
I bet you're confused since you didn't even read TFS.
Re: (Score:2)
Re: (Score:2)
Oh yes, because you really want a teacher doing electrical maintenance instead of an actual electrician.
They don't need to do it [or shouldn't do it] themselves. But they should have the knowledge to know what is needed (like some additional circuit breakers) and know how to press buttons on a phone to call up an electrical contractor and not have wool pulled over their eyes enough to be charged £100,000 instead of $500.
Re: (Score:2)
If the teacher teaches electronics, electrical engineering, or electrical systems design... if they don't have one of those, maybe they should get one.
There's a saying: "Those who can't do, teach. Those who can't teach, manage."
Re: (Score:3)
The one possibility i can think of that would prevent being able to shut off the lights with the breakers is if whatever broken controller controls them won't turn them back on if it's power cycled. If there are actually controllers for each individual light, then that's the only explanation I can think of for why they can't just cut the power upstream. This whole issue where they have to get a new controller from a specific manufacturer is, of course, a great illustration of why we need true open standards
Reminds me of The Smiths (Score:2)
https://www.youtube.com/watch?... [youtube.com]
this is why smart systems are dumb (Score:5, Insightful)
this is why smart systems are dumb.
The major difference between a thing that might go wrong and a thing that cannot possibly go wrong is that when a thing that cannot possibly go wrong goes wrong, it usually turns out to be impossible to get at or repair. (Douglas Adams)
Re: (Score:2)
In general, automation is great when it works right but a royal PITA when it doesn't.
There's a saying from the 70's: "To error is human, but to really foul things up you need a computer."
Re: (Score:2)
In general, automation is great when it works right
Not even that in my experience. Decisions by automated systems always seem to be orthogonal to the decisions I would make myself.
Re: this is why smart systems are dumb (Score:2)
Re: (Score:2)
this is why smart systems are dumb.
There are much smarter than you think; "I can't turn off the lights Dave!" (AI won't be able to read Dave's lips)
Re:this is why smart systems are dumb (Score:4, Insightful)
I disagree.
Smart systems are fine, IFF there is a mechanical failsafe/override or backup. If there isn't, you get stupidity like this.
But as others have noted, any local electrician could have been hired at any time to slap in breaker boxes anywhere in the school to shut down any arbritary number of lights at the flip of the switch.
So not ONLY did they have no mechanical backup or failsafe, they didn't even bother spending a couple thousand bucks putting one in to make sure this was addressed and couldn't ever happen again.
Re: (Score:2)
There are 7K lights in that building / set of buildings.
I believe you vastly underestimate the complexity of that system.
Re: (Score:2)
Why didn't the "system" include as many as 7K switches?
They went out of their way and spent more money than if they had just put switches in each room... how smart.
Re: (Score:3)
I think the switches are there, only they are controlled remotely through a proprietary system which stopped working properly.
However, I was replying to the statement: "any local electrician could have been hired at any time to slap in breaker boxes anywhere in the school to shut down any arbritary number of lights at the flip of the switch" - which is vastly underestimating the complexity of the system. This is not a two-room apartment.
Re: (Score:2)
I'm gonna get chastised and modded down as a troll, but in my estimation 90% of everything new is shit and only 10% is actually something worthwhile and good. The 90% that are shit are reinventions of solved problems just because someone comes along and wants to spend money they're given instead of spending what's needed and giving the rest back, or because someone's inexperience/taste/stupidity leads them to dismiss a true and tried system in the name of "we can make it better". The problem is these peop
Re:this is why smart systems are dumb (Score:4, Insightful)
Smart systems are dumb because the world of tech moves way too fast for things like this. When you're installing something like a lighting system into a building, you're installing something that should last decades. The problem with these smart systems is that they are obsolete in a few years, and after 10-15 years completely antiquated. That's the problem this high school has - the system is over 10 years old, finding someone who remembers how it works is a challenge enough, and parts for it are basically unobtainium.
Any money a smart system might save in energy costs will be eaten up by the cost of ripping it out and replacing it every few years. Particularly since newer systems will undoubtedly be cloud-based, so you don't even have the option of running an unsupported but still functional system until the hardware fails.
Re: (Score:2)
I disagree.
Smart systems are fine, IFF there is a mechanical failsafe/override or backup.
Yes, there is a simple rule for installing a smart system. If it fails, the fallback system should maintain at least the functionality of the old dumb system.
This is not rocket science. Maybe the switches are a bit out-of-the-way compared to the old light switches, but they must be easily accessible.
Re: (Score:3)
The smartest smart system is a dumb system with a smart system in parallel.
You should have switches for your lights, and parallel smart switches.
If the smart system goes cuckoo then you shut it down completely and control everything with the manual switches. If you use 3-way switches then it doesn't matter if the smart switches fail "on" or "off", just toggle the dumb switch to whatever position it isn't in to change the lighting state.
remote switch (Score:5, Insightful)
"And yes, there will be a remote override switch so this won't happen again,"
Relying on remote systems is basically a guarantee that it will happen again. Where is the physical switch?
Re: (Score:2)
Where is the physical switch?
Exactly. This school's lighting system is inexcusably, someone needs to be fired and never allowed near another lighting system ever, bad. There should be physical switches to turn the lights on or off. You can have a master switch that provides power to localized switches, if you really want a single point of control/failure, but local switches should always be present.
Re:remote switch (Score:5, Interesting)
Re: (Score:2)
Similar story here. Today I went into a supply room to fetch ... well ... some supplies. Where's the light switch? There is none. Well, there's a keypad somewhere (not in the room) that you use to toggle the lights on. A map sort of shows you which button it is. Tried the one marked on the map - nada. Never did figure it out.
This system, by the way, once won an award for its design. Designed to meet the needs of bean counters. Not people who actually have to use it.
Re: remote switch (Score:2)
I work in usability and XD, and this kind of story is familiar to me.
I keep a photo gallery of elevators and parking payment machines and the like that are wallpapered with handwritten signs and tape pointing at what buttons to push to get the most basic of actions to happen, in a world where people used to know how to use these things already: push a button for your floor, put your quarters in the meter.
Meaningless icons, obscure names for what should be obvious things, buttons that do different things dep
Re: (Score:2)
Re: (Score:2)
Re: (Score:3)
Re: (Score:3, Informative)
I'll go off topic with you. Why should the seat always be down? There are many things that wives don't do where you don't really hear the husband complain like changing the HVAC filters. Can you please explain why it's much harder for the wife to put the seat down? Maybe the wives should be happy that the husband put the seat up instead of tinkling on it.
There are many things that males and females do differently. If it's important to an individual it gets done, if it's not it doesn't. I don't want to get i
Re: (Score:2)
I'll take "China" for $800, Alex.
Re: remote switch (Score:2)
Where is the physical switch?
It's a remote override switch. It's somewhere remote. So presumably they just call someone up at closing time and hope they haven't clocked off early.
Re: (Score:2)
“Remote” does not mean the same in construction that it means in IT. In this case, “remote” likely refers to “not being in the same room as the light” but is most likely still a physical switch.
Re: (Score:2)
Our local tennis club installed a "smart" lighting system: lights go on and off based on a schedule, connected to reservations, or with an app, or with sensors, etc.
But at the end of the line is a large rotary switch with a "1" and "0" setting. :-)
Guess what that does
What is the software that failed? (Score:2)
And who was the vendor that installed it? There should be accountability here somewhere. Why wouldn't the school (district) negotiate some guaranty of longevity?
Re: (Score:2)
And who was the vendor that installed it? There should be accountability here somewhere. Why wouldn't the school (district) negotiate some guaranty of longevity?
If you read the article, the company who originally installed it changed hands over the years. The school had to track them down and once there negotiate with them on getting things fixed.
As for the guaranty of longevity, you think school districts have bottomless pits of money?
Re: (Score:2)
As for the guaranty of longevity, you think school districts have bottomless pits of money?
It's a project with the depth and complexity of a high-school computer lab project. They're not exactly re-wiring a nuclear reactor here, and we're not still sitting around wondering how we could fix it. Radio Shack was slinging X10 product over two decades ago.
Light switches are even older and more reliable I hear. Would be a hell of a project for the wanna-be electrician. I hear it pays pretty well right out of high school.
Re: (Score:2)
You are assuming a sensible design. I don't think that's what's going on here. If it were a sensible design, there'd be lots of options besides unscrewing light bulbs.
I'm guessing that the system has some requirement that keeps local electricians from working on it. The needed to import special parts from China to fix the system. That's got to mean there's something really flaky in the system design.
Relays (Score:5, Informative)
There has to be relays in the system to close the various circuits in the school. They should hire actual electricians who know their stuff it sounds like. They just have to go to where each circuit's switch/relay is that powers it on and off. The 5 or 12v signals from the computer system are only powerful enough to power the switches/relays that actually turn power on or off. It shouldn't take a rocket science to replace those with manual switches.
Re:Relays (Score:5, Informative)
Re: (Score:2)
LoB
Re: (Score:2)
Seems like a safety issue. When designing fire safety systems, you have to fail safe. Lighting is rather important for safety, so it needs to fail on.
Therefore it should be possible to control it just by toggling the power to the system. Even if the control system is down, the lights should notice that there are no comms from it and turn on.
Re: (Score:2)
Sure. There are probably a bunch of them in a decent sized school, hidden up in the ceilings. So you've got to rip those out, then fish cables down through the walls and install switches. You don't want to leave half the school without light during classes, so you have to be a bit careful. Not to mention you have to figure out who's going to do it, and who's going to pay for it, which is what probably took the year.
Re: (Score:2)
They probably used a zone-controller system that had multiple zone controllers on each lighting circuit (i.e. more than one classroom per circuit breaker) and probably had the master controller fail. When the zone-controllers fail to receive a data signal from the master controller, they fail to an always-on state. Something similar can happen if something goes wrong with a data cable termination at any particular zone controller, because the zone controllers daisy-chain the data signal.
I just ripped out si
Re: (Score:2)
They should hire actual electricians who know their stuff it sounds like.
I've never once met an "electrician who knows their stuff" approach a custom made smart piece of equipment with any level of understanding of the consequences of their actions. Messing with wiring on smart systems which you have no way of controlling is *bad*. It's like trying to find out what is plugged into a UPS by switching it off level of *bad*.
It shouldn't take a rocket science to replace those with manual switches.
No, rocket science is usually quite a bit easier than bypassing a building control system. There's far fewer control elements in a rocket.
Saw this today (Score:2)
My first thought was, "But technology!"
Here's a question: what moron thought it would be a good idea to install this system and have the default option be lights on at all times?
Second question: what moron thought it would be a good idea to install this system and not have light switches?
Re:Saw this today (Score:4, Insightful)
Answer: Non-engineer PHB writing the contract and then selecting the lowest bidder.
Re:Saw this today (Score:5, Informative)
> Here's a question: what moron thought it would be a good idea to install this system and have the default option be lights on at all times?
Because lights that are always on are safer and less inconvenient than lights that are always off.
> Second question: what moron thought it would be a good idea to install this system and not have light switches?
Being somewhat familiar with similar systems (cursory research suggest the system is from Cooper Lighting), there are light switches. Problem is, the switches are low voltage digital interfaces with the control system, not local disconnects. System broke, switches don't work. The reason this is the case is you need all sorts of controls to turn the lights on and off automatically under modern energy codes, and a physical switch wouldn't provide a signal to let the system know the lights are on or off, and if the switch if off the system has no way to turn them on.
As for breakers... yes there still should be (or should have been) per-circuit overcurrent protection that should be manually operable. Article says they used this to turn off exterior lights, but I have no idea why they didn't for the rest of the lights. I can only guess there was a fear that if they shut them off by the breakers they wouldn't come on again?
=Smidge=
Your taxes at work (Score:2)
makes you feel good about paying taxes...
The last one leaving (Score:2)
Will the last one leaving please turn out the lights... errr.... no switch. Um... Will the last one leaving please unscrew all the bulbs and put them in the box next to the teachers' desk.
"Progress".
p.s., Sorry teacher, I couldn't reach the bulbs because you forgot to renew our subscription to Ladder Unfold.
Congratualtions on decision to automate (Score:2)
oh.
The lights were on... (Score:2)
The "piecemeal" approach (Score:3)
Due to the a dead server, and no communications to study, it would be difficult to reverse-engineer the commands for the control boards. You'd probably have to resort to disassembling the firmware on the control boards. Still, I can't believe there were no hackers that could be found to give that a try. Once they understood the comms you could probably slap together a control system in the scripting language of your choice.
Re: (Score:2)
the vendor put in blocks to stop hackers so that you where forced to buy all hardware from them
Re: (Score:2)
Re: The "piecemeal" approach (Score:2)
Standards change relatively quickly compared to the lifetime of a building. And even though there are standards, they tend to end after defining how to send "set variable 0005 on device 032 to value 1234" , and then whoever installed it ended up mapping variables to physical inputs and outputs, or even mapping the variables as inputs or outputs to software functions running out on field devices.
However, before you get to that point the first hurdles tend to be finding a computer with a standard com port run
Re: (Score:2)
Re: The "piecemeal" approach (Score:4, Informative)
10 years ago digital phone system vendors were demanding my employer let them install a Win2008 server connected to a public static IP without any firewall or antivirus software and with automatic updates disabled and RDC enabled.
I think you vastly overestimate the competency of companies who make their money selling installation of proprietary equipment systems rather than software services.
Says it all! (Score:2)
HOTEL CORAL ESSEX (Score:2)
HOT ORAL SEX
That, paraphrased, is what I'd do to their marquee (if they have one, and I was there) You get the idea.
Like the frosty piss in this thread.. where's the breakers? Place has got to have breakers to get past permitting and all that.
Back up parts? (Score:2)
Software is broken but parts have not come from Ch (Score:2)
This makes no sense unless the parts are mask programmed ROMs.
admins (Score:3)
Re: (Score:2)
The school administrators can't turn the lights off, but any of the middle schoolers can reprogram the building to play tetris
Well done, you have found the cause of the problem.
M5 tie-in (Score:2)
Have they let Dr. Daystrom try reasoning with it?
Solar cells (Score:2)
Cover the lights with solar cells.
Re: (Score:2)
Cover the lights with solar cells.
Mini Dyson spheres then.
light on... nobody home (Score:2)
What a learning quality building... (Score:2)
...that they don't know there's a "main switch" in the electrical panel that cuts off all the power...
Of course, in Winter, that must suck... but they have 10 other months to FIX IT.
Even if they don't want to FIX IT, just ask an electrician to put the boiler into its own fase out of the main switch and you can still have a heated building... but best to FIX IT.
Classic /. (Score:2)
These responses are so classic slashdot. I had no idea so many licensed electricians frequented /., and already knew the school electrical diagram so well they they're able to criticize teachers, school admins and support staff for not already solving this themselves.
I know my way around electrical wiring enough to make changes to my house, but I wouldn't touch a system like this where a mistake impacts hundreds or thousands of people (like, e.g. killing the heating system by flipping the wrong breaker). T
IOT at its best (Score:2)
What brand of equipment was it? (Score:3)
I was working on software for lighting systems ten-ish years ago. I certainly hope that it wasn't our brand (one of the major electrical manufacturing companies).
Ah, here's a much better link about the problem:
https://inside.lighting/news/23-01/faulty-controls-have-kept-school-lights-2021
So it was a Canadian company that had been bought out by Cooper Lighting. It's a legacy system that stopped development at about the same time it was installed, and in ten years they lost the corporate knowledge of it. There apparently was a new generation of equipment that it could be upgraded to, but ta-da!, global parts shortage!
Where I was working back then, we were in the process of deprecating our first-generation system and working on a next generation system. The old one used a protocol called LON, which was way over-designed for just building lighting, ran over 14ga twisted pair, and required a specific brand of chips you've probably never heard of. It also used old 2000s era CPUs that needed $300+ debuggers to work with them. Their now-current system uses Ethernet for high-level comms, and cat-5 serial links for switches and sensors, which should be much more future-proof to maintain, and also ISM-band radio links for parking lot stuff. And STM32 everywhere. (I haven't heard from these guys in years, but I hope that the parts shortage hasn't been too hard on them!)
The general layout of these systems is that every light has a direct run to a control box full of relays, and some sort of daisy-chain network for wall switches and sensors. The way my company's systems work, each control box has some intelligence of its own, and keeps track of its own schedules. If one box failed, it would only affect that part of the building. There is also a button next to every relay for manual override, in a part of the box that is walled off from mains power, so it's okay for the fingers of random people.
This other system sounds like it was more centralized, and the central control failed. The fun part of that is a one-per-system part is more likely to be made in low quantity and harder to replace.
Re:Sounds about right. (Score:5, Informative)
I don't know what WOK is other than something hot and oily... But are you sure about this, you read all the facts and you know indeed that the principal and teachers were not hired decades ago? Or are you just trying to make a stupid off the cuff political statement that had a typo?
Re: (Score:2)
"Negative, sir, I am a Wok cooking pan!"
Re: (Score:2)
Ah, government. If you aren't part of the solution, there's good money to be made prolonging the problem.
Why do you think that idiocy is only found in governments? You can't have met my wife's boss [non-government].
Re: (Score:2)