Complex upgrades demand advanced expertise

Partial upgrades to existing automation systems can be difficult, requiring close cooperation between the system integrator and plant personnel.

10/03/2018


Figure 1: Water and wastewater plants often are difficult to upgrade because of aging control systems, limited budgets, and requirements for continuous uptime. Courtesy: Tesco Controls Inc.When it comes to upgrading automation systems, some of the most difficult projects are partial upgrades where some components must be replaced while others must be left in place, often for budgetary reasons. For many of these projects, extended downtime is not an option, increasing the degree of difficulty. Water and wastewater plants often are particularly susceptible to these problems because of aging control systems, limited budgets, and requirements for continuous uptime (see Figure 1).

Projects involving partial upgrades require a close partnership between the end user and the system integrator throughout the project, from design to startup. Inevitably, issues with existing hardware and software, and the integration between old and new, are found after the contract is signed. Only a true partnering arrangement, where both parties are committed to each other for the long term, can allow each of these issues to be addressed with little or no impact on the project cost, schedule, and performance.

The two water/wastewater projects described in this article fit that description. Each was completed successfully through a partnership between a municipality and Tesco Controls Inc., a system integrator specializing in the water/wastewater industry.

Project 1: Replacing, converting, and upgrading

A ground-water treatment plant was running two different Wonderware InTouch (v. 9.5) supervisory control and data acquisition (SCADA) applications on obsolete Microsoft Windows XP computers. Each was communicating to in-plant Square D SyMax programmable logic controllers (PLCs) over a legacy serial SyNet communication protocol and media. Several remote SyMax PLCs also were connected to the SCADA system over a serial radio link.

Figure 2: Upgrading only certain components of an existing automation system while keeping components running was particularly challenging at this ground water treatment plant as shown in this system architecture diagram. Courtesy: Tesco Controls Inc.The municipality's ultimate goal was to replace all legacy PLCs at both in-plant and remote sites, but the project was budgeted for a phased approach, with only three PLCs initially slated for replacement. In the first phase, one in-plant PLC and two remote PLCs would be replaced with a hot-standby Schneider Modicon M580 PLC and Schneider Modicon M340 PLCs, respectively. The remaining plant and remote PLCs would be replaced in future phases. However, the municipality needed to keep the other existing in-plant and remote SyMax PLCs communicating and operating during the upgrade process (see Figure 2).

The second goal was to upgrade the SCADA software and hardware to Wonderware InTouch for System Platform 2017, running on a virtualized infrastructure. This entailed migrating existing InTouch screens communicating to the nonupgraded plant and remote PLCs to the System Platform application, while developing new screens following updated end user standards.

The main challenges of this project included:

  • Migrating a critical process in an operational water treatment plant
  • Minimizing downtime to keep the plant online and producing water
  • Maintaining visibility and alarming at the SCADA system through the transition
  • Minimizing the amount of plant staff needed to run the plant manually during cutovers
  • Performing cutovers within 8-hour staff availability windows
  • Performing a phased integration where only part of the plant would be upgraded
  • Maintaining the legacy serial radio network communicating to remote sites
  • Performing the migration with incomplete documentation of the existing control system, including missing details of the existing PLC software and panel wiring.

These challenges required Tesco to:

  • Determine the best approach for a legacy migration from Square D SyMax PLCs to new Schneider M580 and M340 PLCs
  • Reverse-engineer the existing SyMax PLC code to be reprogrammed in Schneider Unity software for the new Schneider PLCs
  • Port over legacy components until a future phase of the project
  • Upgrade the SCADA system hardware and software.

Replacing PLCs and converting protocols. Since the existing in-plant SyMax PLCs had to remain online, Tesco used a Niobrara R&D Modbus serial-to-Ethernet bridge module to bridge the existing SyNet network to the new Modbus TCP network as seamlessly as possible. This is shown in the upper part of the partial upgrade system architecture diagram in Figure 2. Bridging the new and old networks with the serial-to-Ethernet bridge module allowed the existing SyMax PLCs to establish peer-to-peer communication with the new M580 PLC while maintaining communications with the SCADA system.

To facilitate the roadmap for the future project phases, Tesco coordinated with the municipality to install an additional serial-to-Ethernet bridge module at each upgraded remote site to convert the serial radio communication protocol to Modbus TCP, which was required by the upgraded remote PLCs. This was critical to keeping costs down for the current project until the district could install a new radio system in future phases. This serial-to-Ethernet bridge solution successfully maintained a high standard of reliability for the current communications infrastructure. 

SCADA upgrade. Since the plant could not be taken offline, the cutover process for the SCADA upgrade was planned carefully and coordinated between the contractor and the operations group, with the end user's project manager taking a central role in synchronizing the different elements of the upgrade. Strong communication between these teams was paramount to the success of this project. The cutover was workshopped over several months with all involved parties into a three-part process.

Part one of the cutover involved all equipment out of service, so that the running process would not be interrupted. This also allowed the integration team to check its cutover plan against the reality of challenges that could develop in the field, such as wiring not being landed where indicated on as-built project drawings, and wiring not connected to field assets outside of the panels being modified. It was a necessary safeguard that allowed the team to test the approach. The success of this part of the cutover familiarized the team with the reality of the field conditions.

Part two of the cutover involved equipment in service but in a standby state. If a piece of equipment was in use and encountered unexpected downtime, the cutover standby equipment could be brought into immediate use. At this juncture, communication between the old SyNet network and the new Modbus TCP network became necessary because of the requirement of peer-to-peer communication between legacy and new PLCs. Additionally, the new SCADA system was communicating with the new in-plant PLC and tested against all field inputs and outputs.

Part three of the cutover involved the remaining in-service and live equipment. With parts one and two successfully completed, the project teams were familiar with and confident in the cutover approach. Operations first changed the plant over to use redundant equipment, made ready in part two of the cutover, where applicable. The system being cutover was divided into two trains (Train 1 and Train 2), only one of which was in service at a given time. This allowed the team to place one train in service, then cut over the train not in use. The newly upgraded train was then placed in service, and the remaining train was cut over, completing the SCADA upgrade.

The new Wonderware System Platform 2017 software was installed on an HP Server PC running Microsoft/Hyper-V Failover High-Availability Clusters with off-premises cluster redundancy connected by a wide area network (WAN). All the Wonderware System Platform virtual machines (VMs) were hosted within this cluster.

The SCADA system now includes an operator interface that allows monitoring and control of plant and remote assets with reporting, trending, alarm generation, event recording, and other features. All of these features are delivered to desktop through Microsoft Remote Desktop Services. 

Project 1 results. The purpose of this project was to upgrade plant operations by improving PLC maintainability and SCADA robustness, while keeping the existing plant operations intact. The former SCADA system relied on long-outdated technology. Additionally, the old computers were equipped with SyNet proprietary daughter cards that plugged directly into the SyNet network via the Windows XP computer motherboard.

Through this project, these outdated and unsupported physical-world liabilities were replaced with a new, virtualized SCADA system, subject to backups of all virtual machines as simple data that can be copied off to a storage device. This significantly improved the end user's disaster recovery capability. The obsolete Windows XP computers were replaced with Microsoft Hyper-V cluster-based Microsoft 2016 Datacenter Edition virtual images.

With a SCADA system that is modern and up-to-date, the end user can stop worrying about where to procure the parts to support a generation of obsolete operating systems and control software. There is an expected future cost savings in operator and maintenance outlays as the new architecture alleviates the end user's focus from maintaining their tools to producing and delivering high quality water to their consumers. 


<< First < Previous Page 1 Page 2 Next > Last >>

Engineers' Choice Awards
The Engineers' Choice Awards highlight some of the best new control, instrumentation and automation products as chosen by Control Engineering subscribers.
System Integrator Giants
The System Integrator Giants program lists the top 100 system integrators among companies listed in CFE Media's Global System Integrator Database.
System Integrator of the Year
Each year, a panel of Control Engineering and Plant Engineering editors and industry expert judges select the System Integrator of the Year Award winners in three categories.
Design of Safe and Reliable Hydraulic Systems for Subsea Applications
This eGuide explains how the operation of hydraulic systems for subsea applications requires the user to consider additional aspects because of the unique conditions that apply to the setting
How to Maximize Factory Automation Efficiency with Low Cost Machine Vision
This eGuide illustrates solutions, applications and benefits of machine vision systems.
Wireless Reliability in Harsh Environments
Learn how to increase device reliability in harsh environments and decrease unplanned system downtime.
October 2018
HMI hardware evolution, Data acquisition strategies, Matching motors and drives, Machine vision advice
September 2018
Optimize controls via cloud software, ladder logic simulation, industrial wireless best practices
August 2018
Augmented reality and virtual reality education, autotuning PID control, cybersecurity advice, educating engineers
Edge Computing
This article collection contains several articles on how today's technologies heap benefits onto an edge-computing architecture such as faster computing, better networking, more memory, smarter analytics, cloud-based intelligence, and lower costs.
Data Center Design
Data centers, data closets, edge and cloud computing, co-location facilities, and similar topics are among the fastest-changing in the industry.
IIoT: Machines, Equipment, & Asset Management
Articles in this digital report highlight technologies that enable Industrial Internet of Things, IIoT-related products and strategies.
SIDB

Find and connect with the most suitable service provider for your unique application. Start searching the Global System Integrator Database Now!

October 2018
2018 Product of the Year; Subsurface data methodologies; Digital twins; Well lifecycle data
August 2018
SCADA standardization, capital expenditures, data-driven drilling and execution
June 2018
Machine learning, produced water benefits, progressive cavity pumps
John O. Ayuk, PE, CFSE, PMP, CAP
Automation Engineer; Wood Group
Doug Baker
System Integrator; Cross Integrated Systems Group
Jose S. Vasquez, Jr.
Jose S. Vasquez, Jr.
Fire & Life Safety Engineer; Technip USA Inc.
Data Centers: Impacts of Climate and Cooling Technology
This course focuses on climate analysis, appropriateness of cooling system selection, and combining cooling systems.
Safety First: Arc Flash 101
This course will help identify and reveal electrical hazards and identify the solutions to implementing and maintaining a safe work environment.
Critical Power: Hospital Electrical Systems
This course explains how maintaining power and communication systems through emergency power-generation systems is critical.
Engineers' Choice Awards
The Engineers' Choice Awards highlight some of the best new control, instrumentation and automation products as chosen by Control Engineering subscribers.
System Integrator Giants
The System Integrator Giants program lists the top 100 system integrators among companies listed in CFE Media's Global System Integrator Database.
System Integrator of the Year
Each year, a panel of Control Engineering and Plant Engineering editors and industry expert judges select the System Integrator of the Year Award winners in three categories.
Design of Safe and Reliable Hydraulic Systems for Subsea Applications
This eGuide explains how the operation of hydraulic systems for subsea applications requires the user to consider additional aspects because of the unique conditions that apply to the setting
How to Maximize Factory Automation Efficiency with Low Cost Machine Vision
This eGuide illustrates solutions, applications and benefits of machine vision systems.
Wireless Reliability in Harsh Environments
Learn how to increase device reliability in harsh environments and decrease unplanned system downtime.
October 2018
HMI hardware evolution, Data acquisition strategies, Matching motors and drives, Machine vision advice
September 2018
Optimize controls via cloud software, ladder logic simulation, industrial wireless best practices
August 2018
Augmented reality and virtual reality education, autotuning PID control, cybersecurity advice, educating engineers
Edge Computing
This article collection contains several articles on how today's technologies heap benefits onto an edge-computing architecture such as faster computing, better networking, more memory, smarter analytics, cloud-based intelligence, and lower costs.
Data Center Design
Data centers, data closets, edge and cloud computing, co-location facilities, and similar topics are among the fastest-changing in the industry.
IIoT: Machines, Equipment, & Asset Management
Articles in this digital report highlight technologies that enable Industrial Internet of Things, IIoT-related products and strategies.
SIDB

Find and connect with the most suitable service provider for your unique application. Start searching the Global System Integrator Database Now!

October 2018
2018 Product of the Year; Subsurface data methodologies; Digital twins; Well lifecycle data
August 2018
SCADA standardization, capital expenditures, data-driven drilling and execution
June 2018
Machine learning, produced water benefits, progressive cavity pumps
John O. Ayuk, PE, CFSE, PMP, CAP
Automation Engineer; Wood Group
Doug Baker
System Integrator; Cross Integrated Systems Group
Jose S. Vasquez, Jr.
Jose S. Vasquez, Jr.
Fire & Life Safety Engineer; Technip USA Inc.
Data Centers: Impacts of Climate and Cooling Technology
This course focuses on climate analysis, appropriateness of cooling system selection, and combining cooling systems.
Safety First: Arc Flash 101
This course will help identify and reveal electrical hazards and identify the solutions to implementing and maintaining a safe work environment.
Critical Power: Hospital Electrical Systems
This course explains how maintaining power and communication systems through emergency power-generation systems is critical.
Engineers' Choice Awards
The Engineers' Choice Awards highlight some of the best new control, instrumentation and automation products as chosen by Control Engineering subscribers.
System Integrator Giants
The System Integrator Giants program lists the top 100 system integrators among companies listed in CFE Media's Global System Integrator Database.
System Integrator of the Year
Each year, a panel of Control Engineering and Plant Engineering editors and industry expert judges select the System Integrator of the Year Award winners in three categories.
Design of Safe and Reliable Hydraulic Systems for Subsea Applications
This eGuide explains how the operation of hydraulic systems for subsea applications requires the user to consider additional aspects because of the unique conditions that apply to the setting
How to Maximize Factory Automation Efficiency with Low Cost Machine Vision
This eGuide illustrates solutions, applications and benefits of machine vision systems.
Wireless Reliability in Harsh Environments
Learn how to increase device reliability in harsh environments and decrease unplanned system downtime.
October 2018
HMI hardware evolution, Data acquisition strategies, Matching motors and drives, Machine vision advice
September 2018
Optimize controls via cloud software, ladder logic simulation, industrial wireless best practices
August 2018
Augmented reality and virtual reality education, autotuning PID control, cybersecurity advice, educating engineers
Edge Computing
This article collection contains several articles on how today's technologies heap benefits onto an edge-computing architecture such as faster computing, better networking, more memory, smarter analytics, cloud-based intelligence, and lower costs.
Data Center Design
Data centers, data closets, edge and cloud computing, co-location facilities, and similar topics are among the fastest-changing in the industry.
IIoT: Machines, Equipment, & Asset Management
Articles in this digital report highlight technologies that enable Industrial Internet of Things, IIoT-related products and strategies.
SIDB

Find and connect with the most suitable service provider for your unique application. Start searching the Global System Integrator Database Now!

October 2018
2018 Product of the Year; Subsurface data methodologies; Digital twins; Well lifecycle data
August 2018
SCADA standardization, capital expenditures, data-driven drilling and execution
June 2018
Machine learning, produced water benefits, progressive cavity pumps
John O. Ayuk, PE, CFSE, PMP, CAP
Automation Engineer; Wood Group
Doug Baker
System Integrator; Cross Integrated Systems Group
Jose S. Vasquez, Jr.
Jose S. Vasquez, Jr.
Fire & Life Safety Engineer; Technip USA Inc.
Data Centers: Impacts of Climate and Cooling Technology
This course focuses on climate analysis, appropriateness of cooling system selection, and combining cooling systems.
Safety First: Arc Flash 101
This course will help identify and reveal electrical hazards and identify the solutions to implementing and maintaining a safe work environment.
Critical Power: Hospital Electrical Systems
This course explains how maintaining power and communication systems through emergency power-generation systems is critical.
click me