Integrating wireless with wastewater
An industrial wireless network and supervisory control and data acquisition (SCADA) system is being upgraded at the Stamford Water Pollution Control Authority (SWPCA) of Stamford, Conn., part of a five-year, $47-million capital improvement plan formed and approved by the SWPCA board. SWPCA operates a 24-million gal/day (MGD) wastewater treatment plant serving 20,000 ratepayers with a drainage area of 12,000 acres through a total of 275 miles of collection system pipelines. The SWPCA was created in 1997 by city charter and is governed by a nine-member board. Over the past two years, the board has undertaken to improve the facility and foster increased staff participation in operations policy.
The facility is being made more energy and process efficient, leading to lower operating costs. Upgrades to the treatment process allow operation conforming to new state discharge requirements. An existing SCADA system will be replaced: The distributed control units and associated communications equipment will be replaced or upgraded.
Wireless network coverage of the entire facility will be a goal during this upgrade to allow integration of wireless field devices into the process control network without expensive wiring.
One area scrutinized was the condition of the existing plant wiring and conduit system. It was discovered that some wiring and conduits were in various states of disrepair and deterioration mostly due to the aggressive environment of the plant. These conditions ranged from broken and generally deteriorated metal and PVC conduit and junction boxes to overfilled conduit and damaged wiring. The low-voltage systems, including communication and instrumentation, were most damaged.
A decision was made to explore other means of interfacing with process instrumentation to eliminate the inevitable problems associated with physical wiring, including the expense to repair or replace it. Given current capabilities of wireless technology and the cost savings that could be realized by implementing a wireless system, a proof-of-concept project was designed to demonstrate the viability of such a system in a municipal wastewater treatment environment. The system was submitted to and approved by the SWPCA board in February 2014.
Wireless system concept, design
The project is primarily a validation of concept and an application of wireless connectivity in the aggressive environment of a water pollution control plant (WPCP). The goal was to design a limited (within the fence line) IEEE 802.11 Wireless LAN (WLAN) providing coverage and communication for wireless instrumentation, general network resource access, communication, and security. Several wireless and instrumentation vendors were considered for inclusion in the project. The selection criteria were based on equipment reliability and capabilities, availability, ease of management, and conformance to open standards.
The typical water treatment plant is well suited for optimum radio propagation. Tanks for the various treatment processes are usually at ground level, or if elevated, at the same elevation; this provides large, unobstructed open spaces for line-of-sight transmission. In addition, several elevated structures provide advantageous locations for transceivers, further enhancing the favorable propagation environment. On the other hand, many structures in these facilities are constructed of reinforced concrete and can introduce attenuation and radio frequency (RF) shadows. Fortunately, these structures can be designed around, and their effect minimized by using the proper WLAN architecture.
It was of paramount importance that the system be open, that is, not proprietary. This follows the model of wired networks (IEEE 802.3 Ethernet), the widely used standard for wired networks. No proprietary cable, software, or interface is required for an addition or expansion of the system; it is essentially "plug and play." The growth and perfection of the IEEE 802.11 WLAN standards makes plant-wide wireless a natural choice for future expansion.
At the instrument level, however, there are no set standards, and the various low-level communication protocols being used are vying for supremacy. The quasi-proprietary nature of these protocols typically requires specific cabling and cabling standards, separate interfaces and drivers, specialized testing equipment, and trained designers and technicians. These all add cost to a project before, during, and after implementation.
The WLAN system is a mesh network that does not require a central controller to monitor and control the activity of connected wireless access points (APs). A controller can lead to throughput and performance issues on some networks. The elimination of a controller also removed the possibility of a single point of failure and loss of the network. A cloud-based utility provides network management from anywhere. The APs form a cooperative mesh network, allowing multiple and redundant transmission paths. Losing an AP will not take down the network. Links are dynamically re-routed to ensure communication is uninterrupted. Such an architecture is very well suited to integration with a distributed control system. The WLAN conforms to IEEE 802.11 WiFi, an open standard.
At implementation, there were no instruments with IEEE 802.11 communications. To interface field instruments, a cabinet for analog signal wiring and conversion to IEEE 802.11 wireless signals was required; the time and cost essentially removed any benefit of wireless.
Another wireless system uses smart gateways and modules for instrument-level wireless communications. HART (Highway Addressable Remote Transducer) is a well-known wired instrument-level communication scheme. HART allows the remote management of field devices; small transceivers are used in conjunction with HART-capable instruments, resulting in WiHART. This system operates under the IEEE 802.15 standard (Personal Area Networks), forming a separate mesh network among instruments.
The system was chosen to allow easy integration of existing instruments into the new wireless system, requiring minimal wiring and involvement of the plant electrician. Transceivers are installed in series with the existing current loop and scavenge power from that loop. The instrument-level mesh architecture actively repeats signals from any device without line of sight to the gateway. This capability fits neatly with the overall system concept.
It should be noted that most, if not all, instrumentation is proprietary to varying extents, even with use of standards. Within each instrument or device are physical or electrical characteristics unique to the device’s function. The project goal was to use open protocols as far as practical to allow for easily expanding the plant network using standard equipment and interfaces. As WLANs become even more widespread, IEEE 802.11 interfaces will become standard features; new instrumentation or mobile devices will easily associate to the network and require minimal, if any, configuration.
Cost analysis, efficiencies
Cost, particularly in a municipal environment, is a major factor in determining what is included in a capital improvement. The savings identified for this simple link, if multiplied by hundreds, possibly thousands of signals, would be hard to ignore; these savings could be applied to other improvements or simply put into reserves. Either way, it is a benefit to the rate or taxpayers. Data acquisition is just one reason for exploring use of wireless technology in this environment. It is envisioned that the final installed system will facilitate more and better security options, integration into computerized maintenance management systems (CMMS), low-, or no-cost intra-plant communication, and plant-wide operator mobility. All of these possible uses contribute to greater efficiency and reduced operating costs. Replacement of failing wiring systems and upgrades to other systems could easily be integrated into a project budget using the savings realized from using a wireless communication system.
The primary treatment odor control system (OCS) selected as the test bed was a convenient 330 ft from the control building, offering an apples-to-apples comparison between the two technologies (one Ethernet segment is limited to 328 ft or 100 m). A wired system was designed: a ¾-in. conduit in a trench carrying one CAT5 cable. An estimate was done for a trench dug under concrete, macadam and grass with restoration; labor to pull, terminate and test; and finally, design, management, and IT involvement. As can be seen in the accompanying chart, the cost was slightly more than $30,000.
These figures are for the cost of the link and not for connected instruments or translational bridge. To demonstrate the efficacy of a wireless system, however, it was necessary to have data to transmit; this necessitated the inclusion of wireless instrumentation, which would be an additional cost, separate from the cost of the network. Both the wireless network and instrumentation equipment were generously provided on an evaluation basis by the vendors involved. SWPCA later purchased the wireless APs for inclusion into a planned expansion into the plant-wide wireless network.
The wireless system required no trench, and labor was limited to installation of the APs and minimal IT involvement. With everything included—equipment, design, management, IT and installation-the cost for the same link topped out at $5,420. Compare this to the $30,200 required for a hard link. The savings for this simple link approaches $25,000, or a savings of about 83%. Multiply this by four hardwired links, and it is apparent that savings add up very quickly. The comparison of time required for installation was also striking: an estimated two weeks to completion of an operating system for the wired system versus two days for wireless. In addition, the wired system is capable of communication between two points; the wireless system is capable of communication with multiple instruments and mobile clients. As would be expected, the capabilities of a wireless system for the relatively small capital expenditure required impressed the SWPCA staff and board.
Another SWPCA concern was network security. All wireless systems operate in an "unbounded" medium as opposed to wired systems, and can theoretically be intercepted and compromised for nefarious purposes, or "hacked." The fear was that someone would or could maliciously break into the network to commit mayhem, like turning pumps off or closing valves. While this is certainly a concern, it is unlikely. Unlike banks, businesses, or research facilities, a typical water treatment facility is not known to produce or contain anything of great value. It would take not insignificant technical and financial resources to hack into this network design; the returns are simply not sufficient to justify the effort.
This being said, there is always the problem of malicious interference, with or without wireless, by those with a grudge or too much time on their hands. SWPCA is not at the point where wireless control of equipment is being considered, though there is an expectation that this will eventually happen. The network is presently limited to data acquisition, which is a substantial function of the system. Control will remain hardwired to local control panels that are themselves hardwired into the Ethernet-based plant SCADA network, a high-speed Ethernet fiber ring. While jamming of RF signals is possible and relatively inexpensive to accomplish, it requires the perpetrator to be very close to an AP or device, obligating the miscreant to commit criminal trespass and violate several federal laws prohibiting jamming.
Physical damage, such as malicious vandalism, is the most likely scenario for taking out a wireless system. This problem can be mitigated somewhat by proper installation practice and security measures such as cameras and access control. Most outdoor APs are well built and can take a lot of abuse; placing them out of reach or providing surveillance will substantially reduce opportunities for damaging the system.
The IEEE 802.15 and the IEEE 802.11 systems adhere to Wireless Protected Access 2 (WPA-2), which is based on the Advanced Encryption Standard (AES) using a 128-bit encryption key; this key is practically impossible for a casual hacker to compromise, requiring computing resources that will not fit in a typical vehicle, such as those used by the U.S. National Security Agency (NSA).
A more likely attack route would be through the plant SCADA network. Separation from the outside world and other measures decrease risk there.
The wireless link, as compared to a hardwired link, was much less labor-intensive to install. The link required two outdoor APs that were rated IP68; one AP is mounted on the outside wall of the control building, while the other AP is mounted on an access platform on the OCS scrubber; both are approximately 12 ft above grade. The APs are PoE (power over Ethernet) capable, so only one wire, the CAT5 cable, was fished through the building wall, providing data and power. At the OCS, local utility power was used with a POE injector to provide power to the AP.
Five transmitters were installed at the primary OCS. Sump level, pH/ORP (oxidation reduction potential), and differential pressure (DP) measurements across the scrubber bed provided live signals for validation. Two other transmitters were mounted at two locations in the area to provide additional mesh points. Installation was straightforward, requiring no special tools or equipment, and was accomplished in less than three hours by the plant electrician. There is no hardwired connection to the sensor network at the scrubber location; the AP relies completely on wireless for communication. The scrubber AP is primarily responsible for communication to scrubber instrumentation.
Once powered, the APs formed a mesh network and began to exchange data and control traffic. Each AP has a key that allows the APs to automatically configure into a mesh network. At least one member of the network must be a wired portal into the network. Without a hard connection, it would be difficult to enable and maintain secure access to wired network resources.
Within minutes, by using the wireless network online management interface, the management dashboard was populated with the two wireless network access points. When the gateway/hotspot was configured and powered up, it showed up in the client listings on the management interface. The network installation and setup, including solving the protocol translation issue, had taken less than two days. A comparable wired installation and setup would have taken two weeks.
The APs are dual band: 2.4 GHz ISM band, and 5 GHz UNII band, using an unlicensed spectrum. To achieve reliability and connectivity goals, the design called for local connectivity (instrumentation and mobile clients) to be accomplished on the industrial, scientific, and medical (ISM) channels on both APs, while the backhaul from the OCS to the control building would be accomplished using the Unlicensed National Information Infrastructure (UNII) channels. This was done for several reasons: First, the UNII band isn’t as widely used in wireless equipment so the UNII band is a much quieter RF environment. This approach largely removes the probability of interference to surrounding homes and businesses, which are likely to be using the ISM band for WLAN connectivity. The UNII band would allow more reliable communication with lower probability of interference from outside RF sources.
The project became more interesting when effecting communication between IEEE 802.15 devices and the IEEE 802.11 APs, from WiHART to WiFi. Data from the instruments needed to be converted from the IEEE 802.15 WiHART protocol to an IEEE 802.11 WiFi stream to transmit process data to the control room via the wireless backhaul. The instrument-level transceivers communicate with an intelligent IEEE 802.15 gateway, which converts the data to IEEE 802.3 Ethernet, then over a wired network connection to an IEEE 802.11n industrial WiFi hotspot for communication with the scrubber AP.
A "translational bridge" was assembled between the two systems, allowing the scrubber AP to send data to the control room wirelessly. The data, shown in simple form on the gateway’s webpage, is updated regularly. Data was received without errors and agreed with process data as shown on existing instrumentation.
Ordinarily, the smart gateway could simply be interfaced with the existing plant network backbone and integrated into an existing SCADA system. This system is used widely in process monitoring and control in manufacturing and refineries because of the mesh architecture and considerable range. However, it is a network specialized for the transmission of only one protocol, WiHART, using the IEEE 802.15 standard protocols.
If all instrumentation uses the same protocol and transmission standard, the resulting network can be very robust. In this case, it was important to maintain and use the WLAN 802.11 network as the primary network for the additional uses aside from data acquisition.
Wireless performance, coverage
Another concern during the design phase was RF spillover into the surrounding neighborhood and interference with the WLANs in surrounding homes and businesses, particularly on the ISM band. The facility is approximately 800 by 1,200 ft. The APs are roughly in the center of that space; two APs are effectively providing WiFi coverage to about 60% of the plant area. A predictive study was done prior to implementation, and as the image shows, did not identify any areas of concern. The signal drops off, conveniently, at the fence line. Coverage conforms well to that predicted; rough field surveys were made using the native RSSI indicators of mobile phones and tablets of different manufacture, and all generally agreed with the predictive map.
After the instrumentation segment of the system had been proven and data proven as reliable and accurate, several foremen were given access to the network and asked to, in effect, "try to break it." From initial apprehension, the operators came to appreciate the mobile capabilities of the system. The system is reported to be reliable and available in most areas of operation, and performance is solid. Video performance (instructional and informational videos) is solid and continuous throughout the coverage area. Remote access of the smart gateway and OCS instruments was easily accomplished on available mobile devices. Internet access, even at the fringes of coverage, remains error-free and reliable.
While the system is capable of throughput of 650 MBPS, the data being monitored is very low bandwidth. As a general rule of thumb, the faster a process variable changes, the higher the bandwidth required. OCS process variables change slowly, and changes are frequently measured in hours. Hence, the system is largely underutilized, allowing network communication resources to be used for other tasks.
Preliminary assessments, after six months of operation, are that the system has exceeded expectations for reliability and performance. Process data from the OCS was accurate and reliable. Plant area coverage was surprisingly robust and predictable. Barring unexpected equipment failure, the system is expected to be available and error-free. System acceptance by plant staff and their willingness to use the network allow for real-world testing of system capabilities. No significant difficulties or deficiencies were experienced or noted that would recommend against expanding the network and its utilization.
Future wireless expansion, savings
One use being explored is voice over IP (VOIP) for use as intra-plant communications. "Convergence" technology allows the use of communication equipment that will use the plant WLAN for communication within the range of the WLAN network and switch to the common carrier when outside of that coverage. VOIP has the capability of eliminating several costly voice communication services at minimal or no cost to implement when a wireless network with capacity is available.
CMMS integration will allow mobile, plant-wide access to inventory, predictive and preventive maintenance schedules, work orders, and timesheets, and access to online operations and maintenance (O&M) and repair data at the equipment location. The benefits from an operator being able to access required information for the repair or adjustment of equipment without leaving the equipment location should not be underestimated. Eliminating lost time creates greater efficiency and cost savings. Asset tracking and real-time inventory control are benefits that also will translate into long-term savings.
Security, in the form of wireless IP cameras and other IEEE 802.11 capable security devices, is another valuable WLAN use. Wireless cameras require only a source of power, available at any light pole, power outlet, or by using photovoltaic (PV) solar and batteries. Wireless cameras can be placed practically anywhere without the need for running costly and vulnerable signal wiring. DVRs can be placed anywhere as a result, greatly enhancing surveillance capabilities and protecting security data.
Flexibility in instrument placement will be another benefit. Instead of hardwiring an instrument to an area, an instrument could, conceivably, be moved to the most advantageous area should process changes occur. Process control personnel also could experiment with the placement of instrumentation to achieve maximum accuracy or benefit. All that is required is a power source, which is always available in a well-designed plant. Hardwiring, alternatively, would be costly and time-consuming, and introduces errors if not installed properly. Wireless, by comparison, requires device placement, power-up, and association with the wireless network, typically done in an hour or less.
Other considerations of recommending wireless integration were the issues of reliability and minimal downtime. When a wiring system is damaged, or is being replaced or repaired, days or weeks could be spent on that task. Real-time data is lost during that period, as operators are required to spend valuable time making rounds and noting data. Data loss makes a very expensive SCADA system very irrelevant. Wireless downtime is minimal; if a wireless AP fails, replacement can be done in less than a morning and the system will be back on line, possibly without a noticeable interruption.
The present SCADA upgrade will integrate the wireless network into its design, resulting in a more extensive system at substantially reduced cost. This will be accomplished by expanding the wireless system by at least two additional APs. Using directional antennas and adjusting transmit power, it is anticipated that up to 95% of the plant area can be covered without interference to neighboring homes and businesses. Much of the short-term cost savings will result from reducing the electrical portion of future upgrades to process instrumentation and communication.
Those involved in the project view wireless technology as the future of data communication; the goal is to "future-proof" the facility to allow the SWPCA to be well positioned to take advantage of future improvements in technology.
Efficiency, flexibility, reliability
A wireless network implementation is not only about labor and cost savings; it is also about efficiency, utility, flexibility, and reliability. Wireless technology is widely accepted, even expected, by most people. Ten years ago, smartphones were a high-tech toy; now these wireless and mobile computer terminals are widely used, and for many, an essential part of their lives. The technology has become more robust and reliable, which are requirements for the exacting nature and science of water treatment.
Integration of wireless networks into a plant environment will undergo the same trials that programmable logic controllers (PLCs) and SCADA systems endured a generation ago. After initial trepidation and resistance, these enhancements to the treatment process are now taken for granted—and many operators wonder how we ever got along without them. So it will be for wireless technology. While this industry is somewhat late to consider or implement wireless technology, it has allowed the technology to mature into a secure, predictable, and valuable tool.
Those involved with the SWPCA project believe wireless technology is the future of industrial communications and must be considered as a viable and cost-effective option in current and future plant designs or upgrades. Done right, a wireless implementation can future-proof the plant by allowing seamless integration with inevitable improvements with minimal capital investments. Integrating wireless technology into plant control networks and systems will result in reduced capital expenditures and operating costs, resulting in lower user rates, a worthy goal and a clear benefit to the ratepayers served by these facilities.
– Daniel Capano, firstname.lastname@example.org, is president of Diversified Technical Services Inc. of Stamford, Conn., and serves as vice chairman of the Stamford Water Pollution Control Authority; edited by Mark T. Hoske, content manager, Control Engineering, email@example.com.
- Wastewater utility Stamford Water Pollution Control Authority installed two wireless networks for SCADA system communications.
- Wireless integration reduced capital expenditures and operating costs, resulting in lower user rates.
- Less installation time and greater flexibility were among added benefits of using wireless in this application, integrating wireless with Ethernet.
How could wireless networking enhance Ethernet communications at your facility?
On March 10, a webcast will discuss this SWPCA project, wireless and Ethernet communications. See "Integrating wireless into an industrial Ethernet application" at www.controleng.com/webcasts; it will be archived thereafter.
– More about the author: Daniel Capano is president of Diversified Technical Services Inc. of Stamford, Conn. He serves as vice chairman of the Stamford Water Pollution Control Authority. Capano consults to several architectural firms on matters involving instrumentation and control systems and their associated communications systems. He writes the Industrial Wireless Tutorials blog for Control Engineering magazine. Link to the blogs below or with linked above.
– Project details: Linked below, learn more about the SWPCA project in a related article that highlights some of the products that were used as well as products slated for upgrades.