Migrating industrial networks

Sooner or later, that legacy plant-floor network must be upgraded. When that time comes, control engineers should use network migration best practices.
By David McCarthy, TriCore Inc. November 28, 2015

While your trusted plant-floor network may have served you well these many years, there will come a time when you need to think about upgrading. Perhaps the benefits of a modern industrial network are just too great to ignore, or the scarcity of parts to keep things up and running is becoming a real challenge. Whatever the reason, the benefits of upgrading can be significant. So can the challenges. No worries, others have been there before you. Here is what you need to know.

Then and now

Not so long ago, all industrial networks were mainly proprietary creatures. Control devices were largely hardwired. Information crawled along at relatively low speeds, and network topologies came in a variety of configurations. Plant-floor data storage and analysis was relatively limited. Network cabling media was largely copper and, more often than not, installed by an electrician versus a network cabling firm. It was not uncommon to have multiple networks comingled on the same plant floor.

Figure 1: Industrial Ethernet cable and connector types include Category 6 cables with RJ45 ends (a), M5 (b), serial cable (c), and BNC (d). Courtesy: TriCore Inc.Figure 1: Industrial Ethernet cable and connector types include Category 6 cables with RJ45 ends (a), M5 (b), serial cable (c), and BNC (d). Courtesy: TriCore Inc.

Figure 1: Industrial Ethernet cable and connector types include Category 6 cables with RJ45 ends (a), M5 (b), serial cable (c), and BNC (d). Courtesy: TriCore Inc.Figure 1: Industrial Ethernet cable and connector types include Category 6 cables with RJ45 ends (a), M5 (b), serial cable (c), and BNC (d). Courtesy: TriCore Inc.

There were many proprietary industrial communication networks in this earlier era including DH1, DH, and DH+; Modbus RTU; TIWAY; and many more. Each controller manufacturer also had at least one proprietary device-control I/O network, sometimes more, depending on the variety of makes and models they offered. Networks were largely deterministic at the time, particularly I/O networks.

Today, plant-floor devices continue to get smarter, providing remote configuration, calibration, and diagnostic capabilities only dreamed of a short time ago. The cost of both bandwidth and storage continue to plummet, allowing for plant-floor data warehousing and analytics on a scale not previously possible. Industrial Ethernet in its various forms is beginning to dominate manufacturing networks globally as the solution platform of choice. It’s a communications network, it’s a fieldbus, and it’s highly interoperable.

Information analysis from the plant floor and across the enterprise is becoming an essential and standard part of most modern control systems. While the benefits of this information explosion can result in significant boosts to the efficiency and quality of a manufacturing process, it comes at a cost. There are new challenges with the physical media installation, network equipment configuration, system security, and more that you are likely to encounter when upgrading to a modern industrial network platform. Let’s take a closer look at some of the issues you might encounter along the way and how best to handle them. 

Plan to succeed

A solid network migration plan is the foundation for a successful outcome. It begins with a good understanding of where you are and where you want to be. Start with a comprehensive system architecture drawing of your new industrial network. It should include all devices and switches connected to the network, references to their name and physical location, cable type references, and intended IP addresses listed throughout.

Next up is documenting your existing network. You may need to do a bit of detective work to identify all connected devices, ascertain their logical network address, and their physical locations. You also have to document the physical cable type and wiring configuration on the network. You may encounter trunkline/dropline configurations, daisy chained wiring, or other configurations. Each of these legacy networks has very specific cabling, shielding, and connection best practices. This information is generally readily available online or from the manufacturer.

Some of these older networks are very sensitive to the amount of information moving across them and the amount of devices connecting to them. If you are phasing in installation and you need to connect to an existing network, simply adding one more connection may bring the entire network to a halt. You will need to take a close look at both node count and traffic on these older networks before attempting to integrate with them. Trapping and counting communication error codes in your controller can give an early warning if there might be a problem in this regard and help you plan accordingly.

Speaking of error codes, there is another issue that may occasionally surface. It occurs when a modern processor or I/O scanner is upgraded and the legacy network remains otherwise intact. With improvements in technology over the years, these newer devices may be more sensitive to network communication failures than their older counterparts. Errors that were previously too transient or invisible to the older processors just might be picked up by their modern cousins. If you find yourself in this position, you may need to kick back to the older processor as you work your way through the issue. A thorough review of the best practices of your legacy platform is generally the best path toward identifying the source of the problem.

After you have documented your new network, thoroughly researched your existing one, and vetted any potential points of failure, you can put together your detailed plan on how to migrate from one platform to the other. Now let’s take a closer look at some of the requirements of a modern industrial network.

Basic cable

Like all networks, your industrial Ethernet is only as good as its cabling (see Figure 1). Industrial applications are often electrically noisy places. They are subject to high electromagnetic interference (EMI), wide temperature ranges, dust, humidity, and a host of other factors. The ANSI/TIA-1005 standard states that Category 6 or better cabling should be used for hosts or devices that are exposed to an industrial environment. Category 6 cable is good for up to 1 GB at 100 meters (328 ft) and 10 GB at 55 meters (180 ft). Category 6e cable can support up to 10 GB at 100 meters (328 ft).

Category 6/6e cable is generally less susceptible to cross talk and external EMI noise. Versions are available that are less susceptible to physical deterioration in the harsher industrial environments. Make sure that the RJ45 ends and jacks are also rated for Category 6. For the best results, use premade patch cables for short runs, with factory installed connectors. For long runs you will need to install jacks.

Shielded Ethernet cable may perform better in high EMI environments if run outside of conduit. You need to install this properly, or it will create more problems than it solves. The key to the use of a shielded cable is in proper grounding.

A single ground reference is essential. Multiple ground connections can cause what is referred to as ground loops, where the difference in voltage potential at the ground connections can induce noise on the cable. This can wreak havoc on your network. To get this right, use a grounded RJ45 connector on one end only of the cable. On the other end use a nonconductive RJ45 connector to eliminate the possibility of ground loops.

If your Ethernet cable crosses power lines, do so at right angles. Separate parallel Ethernet and power cables by at least 8 to 12 in. with more distance for higher voltages and longer parallel runs. If the Ethernet cable is in a metal pathway or conduit, each section of the pathway or conduit must be bonded to the adjacent section such that it has electrical continuity along it entire path.

In general, route Ethernet cables away from equipment that generates EMI. This includes things like motors, motor control equipment, lighting, and power conductors. Within panels, separate Ethernet cables from conductors by at least 2 in. When routing away from EMI sources within a panel, follow the recommend bend radius for the cable.

Which switch?

Figure 2: A managed switch routes multicast traffic only to those ports associated with members of a multicast group. An unmanaged switch treats multicast data the same as broadcast data, sending it everywhere. Courtesy: TriCore Inc.All of your new cabling will connect to the electronic heart of your industrial network. This can be in the form of one or more network hubs or switches. While hubs may be fine in certain office settings, never, ever use them in an industrial setting. They are nothing more than multiport repeaters, broadcasting all traffic everywhere. In the front office, this may cause your e-mail or browser to become sluggish. While this may not be a big deal there, it can be disastrous on your plant floor.

That leaves your choice to managed and unmanaged switches. While managed switches are generally preferable, they are also more expensive than unmanaged switches. Let’s take a closer look at how a switch operates and compare the managed and unmanaged varieties.

There are three types of traffic on an industrial Ethernet network: unicast traffic routes from one point to another point; multicast traffic routes from one point to many points; and broadcast traffic routes from one point to all points. Every device on your network has a unique identifier, referred to as a media access control (MAC) address. This is the key to the much more discriminating behavior of a switch compared to a hub.

When a switch first powers up, it initially behaves like a hub broadcasting all traffic everywhere. As devices pass information between ports on a switch, it watches this traffic and figures out which MAC address is associated with which port. It places this information in a MAC address table. When it figures out the MAC address of a device connected to a particular port, it will watch for information intended for that MAC address and transmit such information only to the port associated with that address.

After a switch has built its MAC address table, managed and unmanaged switches treat unicast and broadcast traffic identically. Generally, you want to keep your broadcast traffic under 100 broadcasts/sec, at a bandwidth of 100 Mbits. A little bit of broadcasting is an integral part of any network. Examples include devices like print servers, announcing themselves periodically to the network, or a computer first booting up requesting an IP address.

A bit of snooping

One of the primary differences between a managed and unmanaged switch is in how they treat multicast traffic. Multicast traffic typically comes from smart devices on plant-floor process networks in a connection-oriented producer/consumer-based technology. In this context, a connection is simply a relationship between two or more nodes across a network.

EtherNet/IP, managed by ODVA, is an application-layer communication protocol that uses this technology. EtherNet/IP is based on the common industrial protocol standard. Typical examples of items you might find in a multicast group include flowmeters, smart sensors, scales, and more. Each of these items produces process data and consumes configuration data.

A device needs to be a member of a multicast group to receive group data. All members of the group receive data. You do not need to be a member of a group to send data to the group. The main problem with multicast traffic in a producer/consumer model is that traffic grows exponentially with the number of hosts. This is where the managed switch comes in.

A managed switch has the ability to turn on Internet group management protocol (IGMP) snooping. Here’s how it works. When enabled, IGMP snooping sends out broadcast traffic to determine the members of any multicast groups. Using this information, combined with the MAC address table, allows a managed switch to route multicast traffic only to those ports associated with members of a multicast group. An unmanaged switch treats multicast data the same as broadcast data and sends it everywhere (see Figure 2).

If your network is using producer/consumer technology, or otherwise has multicast traffic, a managed switch is an absolute must and worth the premium you will pay for it. 

Security in the modern world

Your existing control systems may contain older hardware and software components that were never designed with robust security in mind. To take advantage of all the benefits of the information explosion on the plant floor, you will need to take a hard look at how to secure your system.

There are three main aspects to securing a control system:

  1. Network layer
  2. Server and workstation computer hardware layer
  3. Application layer.

When upgrading your existing network, the challenge is determining all of the devices on the network and which ones have an open connection to the outside world. It can take some work to physically locate each device, determine its current configuration, and make modifications to secure it without disrupting other communications in the system.

An issue with existing server and workstation hardware is installing operating system patches, antivirus programs, or other configuration changes without disrupting communications and operations in other areas of the control system. It is not unusual for such updates and patches to induce problems on the application layer.

Application layer security has its own unique set of challenges. When securing an existing HMI system, the technology platform may be very primitive with regard to security capabilities. In addition, there may be basic people processes that must be enhanced. Many unsecured systems do not require operators to login/logout, resulting in everybody having the ability to do everything. One of the first steps in securing any system is to document and provide the correct level of security for users based on their needs without leaving the system wide open. This usually means that operators, supervisors, maintenance workers, etc., all have different security privileges by class and unique login credentials per person.

One last item to note when securing any system is physical security. Restricting physical access to certain key components, such as application and data servers, should always be part of any overall security strategy.

Keep the following in mind when putting your security strategy together:

  • Understand the risks. Your system must be evaluated to determine what could go wrong, where the potential vulnerabilities are, and what the consequences are should a vulnerability be exploited. The probability and severity of such occurrences should be blended to create an overall risk factor. Focus your resources on the areas with highest overall risk factors.
  • Define the security zones in your system. Each zone is comprised of logical and physical components that share common security requirements and have clearly defined zone boundaries. Within each zone, you will likely deal with security of the network, hardware, and application layers.
  • Identify the data that moves between zones.
  • Determine your potential points of physical and logical entry for each zone and secure them. This likely entails not just technology, but people practices, such as prohibiting thumb drives and removable drives, signed acknowledgement of policies by outside contractors, and more.
  • Audit the results. Security is an ongoing process, not just a passive feature of your system. You need both the technological tools and the people processes working together to ensure your security scheme is robust and sustainable over the lifecycle of your system.

Mission accomplished

After you finally get things up and running, you will need the tools to keep an eye on things to ensure everything remains running smoothly. Rogue Wi-Fi access points causing duplicate IP collisions, network cameras inducing broadcast storms, and a host of other items can create headaches on your network.

There are many products available to help manage things. For a low-level look at things, packet-analysis products provide visibility into the exact nature of the traffic on a particular network node. When a broader view is required, there are many network-traffic-monitoring solutions on the market. These products show the devices that might be generating excessive traffic or broadcast storms, which devices are requesting large files, and any devices having connection problems or sluggish response.

Follow these guidelines to reap the benefits that a modern industrial network can bring. Begin the process with a detailed migration plan. Lay the best foundation by following all cabling best practices. Follow that up with the right selection and configuration of your network hardware. Finish it all with a well-executed security and monitoring scheme, and you can rest easy that your industrial network will function safely and effectively for years to come.

David McCarthy is president and CEO of TriCore Inc., a national systems integration firm based in Racine, Wis., with offices in Santa Fe Springs, Calif., and opening soon in Indianapolis, Ind. Before he founded TriCore in 1991, McCarthy served in various capacities at Alfa Laval/Tetra Pak, including manager of engineering for its U.S.-based food engineering company. McCarthy, who has more than 30 years of experience in automation, is a computer scientist from Rochester Institute of Technology.

This article appears in the Applied Automation supplement for Control Engineering 
and Plant Engineering

– See other articles from the supplement below.