Moving SCADA to the data center
Historically, human-machine interface (HMI) and supervisory control and data acquisition (SCADA) systems were not included in the scope of the charter for information technology (IT). Process engineering or the plant maintenance organization were normally tasked with the responsibility for the SCADA/HMI systems. The responsibility included the selection and management of the computing platform and the industrial control network (ICN). In many cases, a lone electrician was responsible for the entire system, including the workstation, the ICN, and the programmable logic controller (PLC). In this era of SCADA/HMI, the ICN was rarely interconnected with the IT enterprise network or the Internet. Remote access typically was done using phone modems connected directly to the SCADA host workstation.
Bringing IT and SCADA in sync
In the second half of the 1990s, two trends brought IT and SCADA closer together. The first was the Y2K scare that allegedly affected all enterprise computing systems. This included computing systems found in production management software, such as SCADA and HMI. IT departments included production systems in the scope of their audits, looking at production systems to ensure that they were ready to handle the new century date formats.
This alone did not lead to the integration of production systems and IT architecture. For the most part, these systems still weren’t connected to the rest of the corporate IT networks. Given the air gap security combined with the unique requirements for managing traffic on the ICN, centralized IT network management tools simply could not be used effectively.
A second trend that coincided with the Y2K focus was the deployment of enterprise resource planning (ERP) systems in many organizations. ERP’s data demands and uncertain connection to the shop floor meant that either a parallel system had to be deployed or SCADA would become the bridge. As a result, SCADA and other production systems such as manufacturing execution systems were used for tracking production and feeding ERP’s shop floor data appetite by connecting to both IT networks and industrial control networks.
During the past 15 years, we have seen continued integration of IT and production systems. SCADA systems are now for the most part connected to the enterprise networks. Remote access usually is provided via virtual private networks (VPNs) so the SCADA system can be reached from anywhere on the Internet.
There has been some organizational friction in bringing these two very different cultures together. At the top level, the IT department generally looks at its systems as dynamic and having a focus on scalability, performance, and cybersecurity. It may be more willing to apply patches and updates, as long as it knows it has the ability to roll back to previous configurations if problems occur. The focus of the industrial network is to deliver extremely high reliability and safety. Due to the critical nature of real-time data, even short term disruptions can impact production rates and the safety of the workers and equipment that rely on it. For this reason, software patches and firmware upgrades are more thoroughly tested before they are applied to production systems.
SCADA’s place in the control center versus data center
Today, systems are more under the protection and management of IT and its practices. This is particularly true for cybersecurity and virtualization, which are becoming widely adopted. This leads to the question of whether the SCADA host computer should be removed from the control room or the shop floor and moved into the data center.
PcVue Solutions was recently asked to provide guidance to a medium-sized organization that was considering this issue. The company has a large warehouse that houses materials that must be controlled in strict environmental conditions. To automatically maintain the integrity of the warehouse environment, the company purchased a system from a vendor that uses our SCADA platform for the HMI. The vendor delivered the HMI on a single-user workstation connected via an industrial network to several PLCs in the warehouse that controlled the heating and cooling equipment along with other environmental control equipment. The workstation was physically located in a desk in the warehouse.
Management was concerned that the workstation was exposed to physical damage from normal warehouse material handling activities such as forklift movements (see Figure 1). They also were concerned about the time required to respond to either an accident of this sort or even normal maintenance issues with the workstation, given its physical location. For these reasons, the company wanted to move the workstation from the warehouse floor to the enterprise data center.
Newer protocols offer possibilities
This was possible because the PLC communication was an Internet Protocol (IP), which has generally replaced the older serial protocols in modern ICNs. The IP created the possibility to have a VPN established from the data center to allow the HMI to connect to the PLCs on the ICN.
To provide the warehouse operator access to the graphical user interface (GUI) of the HMI, the IT department preferred to use Microsoft Remote Desktop Applications (RDA). RDA is a subset of the widely used Remote Desktop Services (RDS). RDA allows the HMI GUI to appear on the operator’s normal desktop as an icon without having the RDS requirement to overlay the entire desktop. As a standard Microsoft solution, the use of RDS or RDA is transparent to Microsoft-compliant SCADA software platforms.
The IT department also wanted to host the HMI workstation on a virtual machine (VM) in the data center rather than have it on a dedicated workstation, as was the case when it was in the warehouse. With a standard Microsoft application, running on a VM is transparent, as designed. The only issue that was out of the ordinary for IT was the need to map a universal serial bus (USB) port to the host. Use of a USB dongle for licensing is a common practice found in SCADA and HMI platforms, but less common for enterprise software. In this case, it was a simply a mapping of the USB port to the VM hosting the SCADA. A network USB device may be required in more complicated multi-station networks.
Success in the data center
The move from the warehouse floor to the data center was judged a success from many perspectives (see Figure 2). Some of the many benefits include increased transparency of the technology, tightened cybersecurity, and improvement in system reliability including a major reduction to disaster recovery estimates. Both IT and the warehouse management organization judged the performance to be equal to what they had when the workstation was located in the warehouse.
The program had another effect. The facilities maintenance team realized it could improve performance if it has ready access to the environmental monitoring information. Subsequently to roll-out of the virtualized workstation, the maintenance department requested separate and unique access to the SCADA/HMI system.
The project originally had been licensed and configured as a standalone system supporting a single user at a time. A new user profile for maintenance could be set up under the existing license and the RDA connection could be shared with the maintenance organization. With the new architecture, anyone using the system could install RDA. After it is installed, it is simply a matter of opening the RDA icon on the desktop, logging into a session and the user is in a window that behaves exactly as it did on the dedicated workstation. The downside was when the operations were logged in, the maintenance people would get a "connection refused error" when they tried to log in and vice versa when maintenance was logged in, operations could not access the GUI.
To provide for multiple users to access a standalone HMI, it needed to be reconfigured to a SCADA system. Some modifications were required to the license and the configuration of the SCADA project. A standalone HMI license can be converted to a multiuser client server SCADA license in most systems with various degrees of difficulty. In this case, it was a matter of the exchange of a license file.
There also are configuration changes required to reflect the new users in the system. For example, the configuration to ensure that each user is sharing the same communication channel to the PLC and not impacting the overall level of traffic on the ICN with parallel communications requests is required.
With these architectural and configuration changes in place, it was now possible to expand the number of concurrent users. The customer considered how many users would need to access the system simultaneously. In this case, the answer was three: the on-shift operator, the maintenance manager, and the plant manager. There are multiple people in the operations and maintenance department, but for sizing the platform, it was decided that it would support a maximum of three users simultaneously.
Virtual machines were provisioned for the additional client workstations supporting the architecture. The VM hosting the SCADA server software also was designated as the license manager. The network license enables the client station VMs to be created on-demand rather than consuming resources during idle times. Client access licenses were acquired from Microsoft for the new RDA sessions required for this project.
The system now is a multiuser system residing in the IT department’s data center. There have been no issues about performance or accessibility of the system from the warehouse operations end users or the new maintenance department and management end users.
Bearing in mind that the environmental controls that are being monitored and applied here do not change rapidly, the migration to the data center was well advised. If any additional latency was introduced by the more complex ICN routing, it was not noticeable. More importantly, the risk to losing the SCADA server from physical damage from a forklift strike was eliminated.
Disaster recovery plan
The IT department did call one last time when it was reviewing its overall disaster recovery plans. Protection and recovery of historical alarm and event database information was discussed, which it maintains in a Microsoft SQL Server database. Second, consideration was given as to how to best protect the project configuration directory and the platform installation directory. Finally, a review was done of the steps to put it all together into a clear plan of action in the case of an unexpected loss of the server hosting the virtual machines or even the data center itself.
The disaster recovery discussion was done entirely within the comfort zone of the IT professionals. They know the cost of downtime and a result, they are used to thinking about minimizing the time required for recovering servers and databases.
The acceptance of SCADA and HMI in the data center already has occurred. As virtualization becomes the standard deployment of computing in an enterprise setting, we will see continued growth of the percentage of customers choosing this approach. Cybersecurity concerns also will push the SCADA center into the safety of the data center. For most SCADA and many HMI applications of the future, this likely will be the standard deployment strategy.
Edward Nugent is the business development director for PcVue Inc., a global independent SCADA/HMI provider in Woburn, Mass. He has 24 years of experience with SCADA development and implementation.
– See other articles from the supplement below.