Water/wastewater: achieving the three levels of redundancy

To provide reliable water and wastewater treatment, municipalities must provide a 24-hour operation on a shoestring budget while battling constantly changing regulations, influent criteria, new security concerns and aging infrastructures. To address these challenges, many facilities are either considering an upgrade of their current automation system or installing automation for the first time.

By Grant Van Hemert, P.E., Water and Wastewater Competency Center, Schneider Electric January 1, 2009

To provide reliable water and wastewater treatment, municipalities must provide a 24-hour operation on a shoestring budget while battling constantly changing regulations, influent criteria, new security concerns and aging infrastructures. To address these challenges, many facilities are either considering an upgrade of their current automation system or installing automation for the first time. When examining an automation system’s design, the question of redundancy often arises. But what is redundancy, and does it equate to a hot standby system?

What is redundancy?

Redundancy comes in many forms and is inherent at some level in any plant design. The most basic form of redundancy requires the inclusion of a hand-off-auto switch for each component. In the automatic mode, the plant or system controller runs the process. In the hand mode this step is bypassed, resulting in continued treatment but a loss of efficiency and/or quality.

For example, in an oxidation ditch scenario, the automatic operation may stage aerators based on dissolved oxygen levels. In hand mode, it would be best to turn on all the aerators. This would ensure adequate levels of air without having to continuously monitor process conditions. However, it would also waste power.

As an alternative, sometimes a process can be bypassed. This is common in headworks, in which a bypass channel around the screen can be found. Using this bypass channel can ensure the plant operates, but it may allow particles to accumulate in the downstream clarifiers and basins.

Another form of redundancy exists when more equipment is installed than is required. For example, three pumps may be provided when only two are needed. This type of redundancy is quite common. Typically the third pump still has its own dedicated starter, variable frequency drive and control components. This extends some redundancy to the control and automation system as well.

Redundancy is also gained with multiple process trains. Sometimes each train has its own automation system, or trains have been grouped into multiple control panels. This also represents a level of redundancy.

Finally, redundancy can be designed into an automation system.

Three types of automation redundancy

Automation redundancy can be accomplished via three methods: cold, warm and hot.

Cold redundancy — Cold redundancy is for those processes where response time is of minimal concern and may require operator intervention. For example, consider two belt presses, each with a dedicated control panel. If one belt press becomes inoperable, the operator can resume operation by simply starting the other press.

In this example, the loss of one belt press may result in an accumulation of un-pressed solids, but will not likely impact the ability to meet permit. Thus, operator intervention is acceptable, and a cold redundancy design may be a good solution. However, when time is more critical, warm or hot redundancy are better approaches.

Warm redundancy — Warm redundancy is used where time is somewhat critical but a momentary outage is still acceptable. In this scenario, a momentary bump can be expected. During this bump, the valves, motors and other devices might shutdown temporarily, and the sensors may not report back to the PLC system during the bump.

An example of a warm redundancy application may be exemplified on an ATAD system. In an ATAD process, biological decomposition creates heat. Mixers, aerators and foam cutters are frequently used to distribute the heat, introduce air and reduce foam accumulation. During a momentary changeover, the mixers, aerators and foam cutters could stop working. During the changeover, the heat in the system would remain constant due to the nature of the process, and progression towards Class A would not be compromised.

However, cessation of the mixers, aerators and foam cutters for more than a few minutes would eventually cause an undesired foam build up and heat stratification, causing the process to transition from aerobic to anaerobic. Thus, in this system, the process can tolerate having the mixers, aerators and foam cutters off for a few seconds, with the understanding that they must be restored quickly and automatically to avoid risking the process integrity.

Warm redundancy systems typically have two processors connected in a primary and standby configuration. The primary processor controls the system’s inputs and outputs (I/O) while the standby processor is powered up and waits for the primary processor to stop controlling the process. When this occurs, the standby processor assumes control of the I/O and takes the designation of primary processor, allowing the offline processor to become the secondary processor, which can then be maintained without sacrificing process control.

During normal operation, the primary processor provides periodic updates to the standby processor. These updates usually occur at the end of each program scan and may only involve a portion of the data at any time. Therefore, when a changeover occurs, the standby processor can work off of incomplete data since it may take the standby processor a few program scans to catch up to where the primary was before the changeover. This can contribute to a bump in the process during the changeover.

From a hardware perspective, warm and hot redundancy systems are almost identical, and can be easily confused when looking at suppliers’ product data. Care should be taken to carefully examine the different types of systems.

Hot redundancy — Hot Redundancy is used when the process must not go down for even a brief moment under any circumstance. An example of a hot redundancy system application is a membrane or BAF application. In both instances, the process may not require the hot redundancy capabilities, but during a backwash, redundancy could be critical.

During a backwash, a valve reversal, a motor halt or re-sequencing of the valves may cause an incomplete backwash, allow contaminants to enter the clean water or may even contribute to inoperable equipment. Since none of these scenarios are tolerable, a hot redundancy configuration would be the preferred approach in this case.

Again, the hardware layout of a hot redundancy system is almost identical to a warm redundancy system. However, hot redundancy systems provide bumpless transfer of the I/O during a changeover from primary to standby. To accomplish this, communication messaging, updating the secondary processor and program execution must be properly managed.

To ensure that a hot redundancy system operates correctly, data must be transferred from the primary processor to the secondary processor with every logic cycle to ensure data integrity. Two methods exist for accomplishing the needed data transfer.

The first is to perform the transfer at the end of the program scan. Only upon completion of the transfer will the scan resume. This approach can be called “scan-and-transfer.” This technique was first employed by Modicon when it created the first redundant PLC, and it continues to be an approach taken by many suppliers today. As real world experience has shown, this method can provide accurate updating of data and good control.

However, there are criteria that should be considered when using a scan-and-transfer system. First, the true scan time of the program will be a combination of the program scan and the transfer update. Since scan time can be critical in certain applications, suppliers that support this type of transfer will indicate in their manual that the program should be designed to minimize scanning. They will offer suggestions on how to limit rung executions to only those instances when conditional logic has changed. If these suggestions are not incorporated properly, a situation could arise where a bump is experienced on the output. Any bump possibility would create a warm redundancy situation.

While the scan and transfer technique was used for years, and is a reliable and accurate method when properly applied, technological advances have made a new technique possible — a method that ensures the transfer does not depend on scan time.

This new method is referred to as “asynchronous transfer.” In an asynchronous transfer, the primary processor has two separate microprocessors embedded in its circuitry. The first microprocessor executes the program. At the end of the execution, all data is passed to the second microprocessor. This second microprocessor handles all transfer tasks while the first microprocessor executes the next program scan. Thus, one microprocessor is executing while the other is transferring data to the standby processor.

As this transfer of data from primary processor to secondary processor is asynchronous to the program scan, it now becomes possible to transfer the entire data table without affecting program execution. This eliminates any need to design the program for an optimized scan. In many cases, this benefit makes asynchronous transfer a better fit for water and wastewater treatment.

Redundancy in water and wastewater treatment is critical. However, redundancy is a philosophy, not a set solution.

Fig. 1. Cold redundancy is for those processes where response time is of minimal concern and may require operator intervention. If cessation can be tolerated until the operator can intervene, then cold is best.

Fig. 2. Warm redundancy can be used when a process bump can be tolerated. When the system can handle a slight cessation but automatic intervention is necessary, warm redundancy is the solution.

Fig. 3. For those operations that cannot experience even a momentary cessation, hot redundancy is the only option.

Author Information
Grant Van Hemert, P.E. is an application engineer at Schneider Electric’s Water and Wastewater Competency Center in Nashville, TN.