Managing chaos

In our daily lives we are sometimes called upon to manage chaotic situations, as anyone with children can attest. The more children one has the more frequent and chaotic the situations are likely to be. Also the more likely that most of the screaming isn’t even real. Soon you treat the chaos like car alarms in California; they’re always going off so everybody ignores them. Of course that means that if someone really is breaking into a car no one tries to stop the thief. I even heard one talking alarm that warned, “Step away from the car,” if you got too close. Of course, that just caused people get close to the car to hear it talk.

Operators’ lives are all about waiting for the chaos to start. Of course that assumes they can tell the real alarms from the warnings, that is. In many of our facilities we have many more things that fall into the warning or even event log categories that flash and sound the horn than we have real alarms, so after a while the operator doesn’t even look at the screen to see if it’s important. They just hit the acknowledge button to silence the horn much like we do with our car alarms.

So if that’s the case why do we bother to create so many alarms? If you understand the history of DCSs, you realize that early in their history the only way to get a record of an event was to alarm it. This was a pretty good advance over older analog systems that typically had no way to print an event. In some industries this led to the use of the first-out annunciators to aid in determining what caused a trip. As control systems progressed so did their capability to host alarms and to create alarm floods because the extra alarm capability didn’t include ways to suppress alarms based on conditions. The best we could do was to assign priorities based on criticality.

But the beast had been unleashed and only recently have there been moves to rethink alarms and how we use them. Unfortunately we have become addicted to the bloody things in ways that I suspect we never anticipated. In the days of light-box annunciators, the cost per alarm and the real estate involved made us think carefully about things to be alarmed. Once alarms effectively became free we used them for everything. Our biggest addiction is to information alarms, those that tell the operator something is going on that someone else thinks he should know about but typically isn’t required to do anything or can’t do anything about. Next in line are alarms caused by an action the operator took such as a low flow alarm because he stopped a pump. During operation the low flow alarm might be a critical alarm but the operator knows he caused a low flow condition so why tell him he did. Yes we had some of those kinds of alarms in the light box days but nothing like the quantity we have now.

For a number of years now we’ve had the tools to change how we use alarms but our biggest hurdles are our addictions. Supervisors still want operators to be aware of what’s going on in their areas by pointing it out to them rather than fixing the display issues that make it hard for an operator to see what’s going on. Projects typically don’t include the time and money, mainly for ROI reasons, to do a proper alarm rationalization. The result is we’re still stuck with those because-I-did-something alarms and alarm floods when a trip occurs. Exacerbating this is the fact that our larger plants are effectively multiple plants with managers that have different operating philosophies driving different implementations of what should be a unified strategy.

So how do we fix this mess we’ve gotten ourselves into?

• Start by acknowledging the problems it’s causing and recognizing that it can be fixed. Most companies are using some combination of HAZOP, LOPA, and PHA to identify what’s really critical to safety. That can be a starting point for implementing a plant-wide alarm philosophy once one has been developed.
• Identify the bad actors and target them before anything else. Alarms that chatter because of too little deadband or from being set too close to a normal operating condition on a noisy process or even on a badly tuned loop. Look at normal shutdown periods for alarms that are caused by shutting down.
• Develop an alarm management strategy and make it someone’s job to ensure that every control system gets reviewed to comply with it. Make sure that this person will have the clout to ensure people participate and the passion to evangelize for it.
• Understand what tools for implementing the policy are available in your control system. Can it handle conditional alarming? Can it provide alarm help? Can it do alarm shelving? How easy is it to prioritize alarms? How does it capture and present non-alarm events?
• Piggyback onto projects. Have your champion ensure that any new projects have a budget for implementing the alarm rationalization. This ensures a finite scope to attack rather than just trying to do the whole plant at once.
• Don’t get discouraged if the first one is a struggle. Even when people have bought into the concept the first one or two projects will be a struggle for people because it will in some cases threaten long held concepts and in others people’s desire for control. If they have not bought in then they may be upset when you question the rationale behind an alarm that they created.
• Sell the concept that if there is no operator action required, it isn’t an alarm. This will be an ongoing battle with those addicted to informational alarms. An extension of this suggests that even if there is an operator action required, it may still be a low priority alarm based on time to respond and/or potential to do damage.
• Keep referring back to the criteria you’ve set to make sure that you’re being consistent in your alarm rankings. This is particularly important as you move from project to project or plant area to plant area because as the players change, so does their understanding of what’s written.
• Don’t be afraid to refine the philosophy. ISA 18.2 is a great starting point for working through the implementation, but you still have to address the specific peculiarities of your plant and your processes.

The good news is you can do this. The bad news is it’s a lot of work and involves a lot of people’s time. So what’s stopping you from taking these steps? Why are you still living with chaos?

This post was written by Bruce Brandt. Bruce is the Technology Leader at MAVERICK Technologies, a leading automation solutions provider offering industrial automation, strategic manufacturing, and enterprise integration services for the process industries. MAVERICK delivers expertise and consulting in a wide variety of areas including industrial automation controls, distributed control systems, manufacturing execution systems, operational strategy, business process optimization and more. 

Do you have experience and expertise with the topics mentioned in this content? You should consider contributing to our CFE Media editorial team and getting the recognition you and your company deserve. Click here to start this process.