Backup for Critical Processes

In many industries, there are processes so critical that extraordinary methods must be taken to ensure that those processes are not halted or even stalled. To facilitate this, end users in many industries implement hot standby, or redundant systems. While it might not be practical for end users to implement a hot standby solution with a very expensive distributed control system (DCS), many end ...

By Stephen L. Arnold, Telemecanique div. of Schneider Electric September 1, 2007

In many industries, there are processes so critical that extraordinary methods must be taken to ensure that those processes are not halted or even stalled. To facilitate this, end users in many industries implement hot standby, or redundant systems.

While it might not be practical for end users to implement a hot standby solution with a very expensive distributed control system (DCS), many end users, consulting engineers, system integrators, and others are now learning that PLC hot standby solutions can offer high availability in a more cost effective hardware and software combination.

Hot standby means having two sets of process controls operating simultaneously, with the contingent or back-up system able to assume control of the process immediately if the main system falters or fails. The critical function of hot standby is to prevent an unanticipated process interruption that could result in lost production time, equipment damage, and lost raw materials.

Widespread acceptance

Such systems have widespread acceptance in industries where process failure can be costly, such as petrochemicals and pharmaceuticals. There are also applications in less obvious industries, including food and beverage, water and wastewater treatment facilities, material handling, baggage handling, and certain security and control applications in marine industries.

In many process industries, if a unit stalls or fails, the nature of the ingredients coupled with the condition of the process could very well result in serious damage to expensive and highly sophisticated equipment. Results include possible loss of tens, if not hundreds, of thousands of dollars in destroyed raw product and equipment, plus lengthy downtime for repairs. In some applications, a batch could solidify, forcing workers to chip out the ruined product and thoroughly clean the internal surfaces of the vessel, resulting in further downtime and loss of profit.

How does a company determine it needs a hot standby system? The first step is to evaluate the consequences of a system failure—in both time and expense—and weigh those consequences against the cost of a redundant system.

In a food or beverage application, a process failure could result in ingredients being mixed in the wrong proportions, ruining the entire batch. If the process failed while ingredients were being added, there might be no way to determine if the right amount had been added and the batch will be wasted.

Losing control of a water or wastewater treatment plant process is not only costly, it’s potentially hazardous because untreated or partially treated sewage can be released into the environment resulting in fines and other measures taken by the U.S. Environmental Protection Agency.

A system or equipment failure in a pharmaceutical application can mean improper portions of ingredients in the manufacture of medications. When making tablets, for example, a process failure can upset the delicate balance of the ingredients and ruin an entire batch of medication.

Hot standby solutions are critical in many aspects of maritime applications. Imagine the impact of losing control in any of several critical processes on an ocean-going vessel, including power management, engine control, ballast control, tank leveling and fuel transfer. While there may be manual backup systems, losing any of these processes could be problematical at sea.

Identify the real hot standby

Not all hot standby systems are created equal. If a company decides it needs this depth of protection, it must do some research to ensure the system can perform the most critical functions properly. Here are four tests that can determine whether or not a system is truly hot standby:

Hot standby PLCs operate in parallel, connected to each other, to their I/O networks, and to their larger control systems via Ethernet.

When switchover occurs from the primary to the secondary system, does it happen within one logic cycle? A true hot standby system take system control within one logic cycle, so there is no data loss and the application is not upset or disturbed in any way.

When there is a changeover, does the secondary take control of the I/O within one logic cycle? If not, the I/O does not experience a “bump-less” transfer. This can cause valves to slam open or closed needlessly causing wear and possible damage from pressure spikes. Without smooth control in the transfer from primary to secondary, the system can actually generate more problems. When using a true hot standby system the transfer is seamless.

Is all primary system data transferred to the secondary? Any hot standby system will transfer some data, but many not transfer all of it. This can cause problems because the secondary system may not have all the most recent information to control the system properly because it may revert to old process data, setpoints, etc., when it assumes control. If the system cannot address the fault immediately, the delay could cause damage. With a true hot standby system, all data transfers to the secondary system, ensuring the most current information.

Does the hot standby system automatically manage network addresses for the primary PLC, or does the operator have to develop a scheme to swap addresses from the primary to the secondary system? With a true hot standby system, addresses switch automatically with no additional work required of the operator.

Cost investment vs. equipment failure

The case for specifying a hot standby system can be compelling in the right application. The potential for process interruptions with their resulting costs has to be weighed against the cost of installing additional equipment. Moreover, a standby system that cannot provide full functionality may be little better than no standby at all. Designs that cannot make an emergency changeover within one logic cycle or that lose current operating data can cause as much damage as stopping the process altogether.

Failure scenarios and their associated costs must be weighed against the cost of a hot standby system. With duplicate processors, control networks, I/O, and PLCs, the equipment expense can be significant, but may be lower than the expense of rebuilding or replacing failed equipment. New technologies are making the choice easier because it is now possible to provide true hot standby support using a technologically advanced PLC where it would have been necessary to use a DCS for such applications in the past.

A collaborative process

The key is to have redundancy in the system: racks for local CPUs, communications to the I/O, communication networks, and power supplies. If the application has redundant power sources, it’s not going to lose power if they are fed from separate locations. The entire system will not be lost if a network is down. The ultimate objective is to reduce the possibility of component failure.

Once a business case has been made to justify the cost of adding a standby system, process engineers and technicians who operate the critical equipment should work with a system integrator to design the system strategy. With their intimate knowledge of the operation, those engineers and technicians should be deeply involved in the process, providing the integrator with important specifications and insights into how the hot standby system must mesh with the rest of the operation.

Once in place, a properly configured hot standby system can provide a critical layer of protection for an entire plant or process unit, preserving materials, equipment, and an uninterrupted production schedule.

Author Information

Stephen L. Arnold is senior product specialist, Telemecanique Automation & Drives division of Schneider Electric North America. Reach him at stephen.arnold@us.schneider-electric.com .