Can your communication networks support an automation upgrade?
Before you deploy a new HMI, install new software to support mobile devices, or add a loop tuning utility—make sure your networks can handle the increased traffic without bogging down.
Companies considering process plant upgrades know they have to work within the physical constraints of existing equipment. Pipes, pumps, and valves have limits to the volume of fluids they can handle. Electrical switchgear and wiring has a maximum current carrying capacity. If it's necessary to move beyond those limits, then upgrades or replacement of items that create the bottlenecks will be necessary.
The same applies in automation systems. A PLC or other controller has a maximum number of I/O connections, which is easy to see, but other limits aren't so obvious. A pressure drop in piping may signal excess volume in a process, but what are the signs of an overloaded control network? The early signs are more subtle.
When a new plant is designed, engineers typically build a level of excess capacity into all the systems to cover themselves and make sure the owner isn't disappointed by some unanticipated choke point. This applies throughout a facility, typically including the digital networks that facilitate communication among field devices, controllers, HMIs, and other process-related computing systems.
Those networks, like the process piping and electrical wiring, were designed to handle anticipated volume plus some extra. The problem is that designers 20, 10, or even 5 years ago didn't anticipate all the things users have added to their plant networks. Back in 2009, who would have thought they might be using an iPad to remotely access a process unit?
Little by little, that initial extra network capacity gets used up, although you might not realize it. If your network can handle the existing traffic, you may not know it's close to its limits until one more new thing begins to bog the system down. It may be a major upgrade like a new HMI, but even small additions of things like software utilities can make a difference. The problem with most companies is they don't know how close to they edge they are in terms of network capacity.
How plants get in trouble
While there are charts that tell you exactly how much liquid you can expect to put through a pipe or the ampacity of a given size of wire, network loading isn't as clear-cut. Measuring the performance of your network isn't easy, and there are few guidelines to suggest how much traffic is practical.
We often talk about automation system controllers being "overloaded," but this usually has more to do with moving data among controllers than with inadequate brute computing power. It is rare to outstrip a processor's capability to make loop calculations, but common for controllers to slow down when they are asked to continuously communicate large amounts of data.
Consequently, some customers without existing network problems simply launch into an automation system upgrade assuming they won't have problems. More sensible individuals may realize there needs to be some sort of network traffic analysis prior to the project, but this is often something of an afterthought.
As an automation solutions provider, we often see customers discover there are few tools available to measure existing and projected new network traffic. Control system vendors include diagnostic software for existing systems but they don't have capabilities to predict what network loading will look like for a system after modifications are carried out. It seems inordinately difficult to get a simple answer to the question, "Can our networks handle this new load?"
As legacy systems reach their end of life, it often becomes a viable alternative to replace the HMI and front-end components, but leave the controllers in place. As vendors jockey for additional installed base, Vendor A convinces the user that its HMI can work seamlessly on an automation system from Vendor B. Maybe it can, or at least it did on Vendor A's test bench. There may even be a number of successful projects where that combination has been used, and the salesperson will cite these as reasons why there is no need to worry.
The salesperson may be absolutely correct as some cross-platform combinations work very well, but the most meaningful question is whether the combination will work in this specific situation, and there are lots of possible variables in play that are site-specific.
If the salesperson responsible for the project truly understands what is involved and is giving the customer proper guidance, there is a strong possibility for success. However, as independent automation solution providers, we have seen more situations where the choice is made and approved without adequate research. When the purchase orders are issued and problems begin to emerge, our services are called upon to solve the compatibility issues.
Fortunately, there are no infamous case histories where such a cross-platform DCS upgrade caused an entire system to crash when it was first turned on, but there are many where a project in its late stage suddenly had to be delayed for a few weeks to solve some network loading issue. As with any project problem, addressing these types of issues early is much less expensive and time-consuming than late fixes.
Tools and guidelines
While there are many tools for evaluating network health in office and commercial IT systems, there are far fewer for looking at process automation system networks, but there are some choices. In some cases, normal IT tools can be used, but judgment is required to interpret the evaluation results.
A study often shows the number of packets moving around and how easily they move, with a specific examination as to the number of collisions. Most automation systems have some sort of internal diagnostic utility that monitors how quickly inquiries get answered, which is related to network loading. When the HMI needs information to refresh the operator screen, does it come through on the first try, or does it get delayed? Does the HMI have to retry often?
Most systems can count situations where information is held up. This is good to know, but it might not help much when trying to figure out how much more bandwidth is available. Most systems can also give you an idea of controller loading, and as mentioned earlier this typically relates more to movement of information than numbers of calculations. This is useful when you are trying to determine how tasks are spread out around your system. You may find certain controllers carrying a disproportionate portion of the load, and communication can be improved by reassigning some tasks to less loaded parts of the automation system.
These tools are certainly useful as far as they go, but they aren't very predictive of success as you consider adding new things to your networks. Of course, if you find that your networks have bandwidth to burn, you don't need to worry, but this isn't often the case.
Bandwidth costs money, and probably cost more back when most plants were built. There have been few technology developments over the years that have caused bandwidth requirements to go down and many with the opposite affect, so more companies are closer to the edge than they realize.