Control Engineering

Link This | Email this | Blog This | Comments (0)

A lesson in redundancy
April 9, 2007

Coming back to work after Easter weekend is not normally a time when one has fresh lessons on the importance of having backups for control systems, but I got a good one.

You don't have to know much religious history to be aware that Easter is one of the biggest days of the church calendar with the heaviest Sunday attendance of the year. Hence it is a day when churches tend to go overboard in music, pageantry, etc. Ours is no exception, so I began to panic when I received a phone call last Friday from our organist saying the organ had just died. I knew it would be used heavily in the service on Friday night and Sunday, so being without it would be a major problem. (I am the organ "curator.")

(Many pipe organs can be every bit as sophisticated as a DCS. Current designs are basically a highly deterministic bit-level fieldbus, where a few hundred switches, with hundreds of thousands of possible combinations, send voltages to small magnetic valves below each pipe. There can easily be 1500 valves, or more. All the information has to go from keyboard to pipe without any latency or jitter. When the organist presses a key, he or she expects to hear the right pipe or combination with absolutely no hesitation.)

Our power supply had simply quit with less than four years service. No power, no music. I had no spare. No redundant supply. The only thing that saved us was the fact that a local tuner/technician happened to have one on hand that would fill the bill, although it had a lower amperage rating. I chewed my nails waiting for a big chord to overload the system, sending a puff of smoke out of the console as everything went dead. Fortunately it held out. The supply was pretty warm after two services, but it didn't quit.

The lesson is about having backups. Nothing critical should be running without some redundancy. Of course none of this is news if you are responsible for keeping a system or plant running. This week I intend to install a second power supply that can run in parallel so we can switch when necessary. I also intend to think about other elements of our system and where we have the potential for things to go haywire. That's a good exercise from time to time for any system.

Posted by Peter Welander on April 9, 2007 | Comments (0)