Think redundantly about automation controllers without high costs

Even for smaller automation systems, designers can take advantage of automation products to incorporate practical redundancy options.

 

Redundancy Insights

• There are many practical and effective steps designers can take to improve industrial control system redundancy without excessively escalating costs.
• Implementing redundancy at one or more design levels is warranted if it can be performed at a favorable cost compared to the expense of outages, production losses, waste or other negative impacts that would result from a single point of failure.
• Applying redundancy to an industrial automation design requires a carefully balanced decision making with regard to price, performance, physical space and other factors.
• Click here for a video interview with Rin Irvin, product engineer at AutomationDirect, in which he discusses the high points of his article.

---

Redundancy and resiliency are commendable concepts, and they are mandatory for mission-critical applications like spacecraft, power generation systems and communication networks. When it comes to industrial equipment and OEM machinery, designers would also like the option to incorporate redundancy for various electrical, automation and mechanical systems. However, redundancy typically involves a high cost and increased complexity, making it impractical for all but the most crucial applications.

Fortunately, there are many practical and effective steps designers can take to improve industrial control system redundancy without escalating the cost and complexity to spaceship levels. This article identifies some industrial automation design approaches that can provide redundancy, or at least an improved degree of resiliency, in many types of applications.

Single point of failure costs

A single point of failure is any electrical, mechanical, software or other element in a system which will cause the rest of the system to stop working as intended if it fails. Single points of failure in any system are therefore undesirable as they can lead to many types of poor outcomes, ranging from minor inconveniences to situations that are far more costly and serious.

Implementing redundancy at one or more design levels is warranted if it can be performed at a favorable cost compared to the expense of outages, production losses, waste or other negative impacts. Basic redundancy options pertinent to industrial automation designs include:

• Power supply

• Sensors and instrumentation

• Programmable logic controllers (PLCs) and hardwired control

• Human-machine interfaces (HMIs), supervisory control and data acquisition (SCADA), and other user visualization/control interfaces.

Among these items, PLCs are widely considered to be the most resistant to failure, but as a clear single point of failure they are often the component where redundancy is frequently considered. This may not be the wisest or most cost-effective strategy. Regardless of the design area considered for redundancy, the amount of risk must be weighed with the cost to implement to determine where the most cost-effective benefits can be achieved.

Applying some redundancy basics

Every application is different, but as an example let’s consider a basic wastewater lift station, consisting of a below-ground “wet well” vault which receives wastewater from one or more sources, and then pumps the wastewater out to another location for further handling or treatment. Variable frequency drives (VFDs) allow the pumps to run faster or slower to maintain consistent level, or the VFDs can be bypassed to simply run the pumps in an on/off mode.

These wet wells are commonly located throughout cities, are generally unstaffed and sited in remote locations, and must operate reliably 24/7/365. Control problems can cause pumps to run constantly until they fail, or to stop running, which results in overflows. Either situation requires emergency service calls, can lead to hazards, and may even result in fines.

Redundancy is essential for ensuring continued operation, and indeed many of these systems typically have more than one pump installed to provide mechanical redundancy. But what can be done to improve automation resiliency? Here are some examples for each of the previously noted areas.

Power supply

Power failure for a control system is a top concern. If there is a widespread power failure, then larger electrically driven equipment like pumps, compressors and heaters will be unable to run and there is not much that can be done about it unless a secondary utility feed or a generator with an automatic transfer are in place to carry these loads.

But for PLCs and other automation components, which usually operate at 120V or less, there are a few cost-effective provisions designers can include. The most common is an uninterruptible power supply (UPS) or a battery control module, which use batteries to continue providing power—typically at 120V AC or 24V DC—to the equipment. This can protect automation equipment from power flickers and brownouts, and it can enable PLCs to monitor the power and send important information to the supervisory systems.

Designers can also incorporate dual power supplies, combined through a diode bridge, to maintain control power in the event of a single failure (Figure 1). To verify power supply status, a monitoring relay—or a more advanced power supply with digital communications support—should be used.

Sometimes a power failure occurs when an overcurrent protection device such as a fuse or circuit breaker trips. A newer approach is to distribute control panel power using electronic circuit breakers (ECBs) to perform the overcurrent protection device function (Figure 2). Modern ECBs provide fast-acting supplementary circuit protection, which can be remotely controlled and monitored by a PLC, and they can provide indication to HMI/SCADA systems. ECBs let users install a large 24V DC bulk power supply, then distribute it to many loads more reliably.

Sensors and instrumentation

Because of their field-mounted locations, often under adverse conditions, sensors and instrumentation are highly susceptible to malfunctions and failures from physical damage and the elements. Therefore, for the most critical measurements, designers should consider a second, or even a third sensor, using different technologies when possible.

For the wet well example, the primary level measurement may be provided by a non-contact ultrasonic level sensor. A secondary measurement technology could be implemented using a submersible hydrostatic pressure level sensor. And, as a third level of defense in case both analog transmitters fail, a pair of high-level and low-level float switches or discrete level sensing probes could be incorporated.

An important implementation note is that, if two sensors are used, the automation should be programmed to indicate a mismatch to the user, while control proceeds by using the most reasonable signal possible, or an operator-selected value. If three sensors are used, it is possible to use two-out-of-three voting logic, but operators must always be notified of signal mismatches via HMI/SCADA so they can be resolved.

 

PLCs and hardwired control

PLCs are extremely reliable devices. Certainly, they can fail on their own or be induced to fail due to electrical surges, or their associated I/O modules can fail. PLCs can even be programmed improperly in a way that causes them to stop operation. But a correctly configured and commissioned PLC can deliver years or decades of non-stop operation. Of course, it is always prudent for support/maintenance teams to maintain on-hand spares for automation components.

For PLCs, there are certain models available with redundancy capabilities, and while the complexity of these configurations has decreased over the years due to better communications protocols and other factors, it can still be relatively expensive to implement a redundant PLC in terms of upfront and ongoing costs. While redundant PLCs may make it easier for users to perform certain processor upgrades, the requirement for additional components, more space and complex configuration may not be worth the overall cost.

For smaller installations, it is often appropriate to add some basic hardwired controls in parallel with a PLC. For the wet well example, a hardwired high-low level controller could run the pumps using bypass contactors, even if the PLC or VFD fails. This running mode is not optimal, but it allows the system to keep functioning until maintenance personnel can resolve the issue.

HMIs, SCADA, and other interfaces

HMIs and SCADA are typically deployed on dedicated or PC-based platforms running visualization software and networked to local controllers and intelligent devices (Figure 3). If the networking or HMI/SCADA hardware/software fails, then operators lose the ability to monitor and control the underlying system, even if the PLC and instrumentation continue to run normally.

Networking and HMI/SCADA redundancy is a much larger topic, but for critical applications designers should consider implementing Ethernet redundant rings in conjunction with multiple HMI/SCADA devices.

However, if these measures are not practical, designers can at least include some local pushbuttons, switches, lights and other similar devices to provide a basic level of indication and control. Operators usually appreciate the availability of some bare-bones functionality in case the more capable automation elements become unavailable.

Basic automation redundancy can be practical

Applying redundancy to an industrial automation design requires a carefully balanced decision making with regard to price, performance, physical space and other factors. Fortunately, there are many products and design approaches to improve the resiliency of power, measurement, control and visualization systems. Incorporating some basics such as a UPS, redundant sensors for critical measurements and some hardwired control and indication devices is a valuable way to improve system uptime and help operators keep machines and equipment running.

Rin Irvin, product engineer, AutomationDirect. Edited by David Miller, Content Manager, Control Engineering, CFE Media and Technology, [email protected].

CONSIDER THIS

How can you improve the redundancy of your systems without unduly raising costs?