Cover story: Troubleshooting Industrial Ethernet

When John Lennon famously complained in song that “…it ain’t easy, you know how hard it can be,” he wasn’t talking about troubleshooting industrial Ethernet automation networks, but he could have been. As industrial automation systems gravitate more to industrial Ethernet, it is important to look at the tools and techniques available to troubleshoot problems on these networks.

The Control Engineering Network Diagnostics Survey asks readers about their experiences setting up and maintaining automation networks. What protocol standards do they use? How well do these networks hold up? What tools do they have available, and what tools are needed? If you buy or specify industrial network diagnostic devices or services, go to www.controleng.com under the events and awards menu and select Network Diagnostics Survey to make your opinions known. (Hurry, the survey will be open only through month’s end, or until we have a statistically valid sample.)

Cover Story summary: Industrial Network Diagnostics

-Industrial network troubleshooting help for Ethernet, others

– Take a network diagnostic survey-Survey is closed. For all surveys we have made public, and research from others, click here.

-Hardware, software, diagnostic tools

-Ethernet webcast; photo gallery

-Podcasts – additional network tips, advice

-More than a dozen links to additional resources, information

“Over 80% of the calls that come into our facility,” says Mara White, market development manager for Fluke Networks, “involve physical infrastructure problems, and I’ve heard the same thing from automation companies.”

Checklist

10 considerations for top network health and performance
1. Use the appropriate test equipment to collect information from the network
2. Check for return loss
3. Ensure the cable stays dry
4. Consider insertion losses
5. Higher temperature can decrease reliability
6. Installation materials may appear to affect cabling performance
7. Look at dc resistance
8. Check the fiber connectors
9. Clean carefully
10. Be careful not to add loss during network design

Source: “10 network troubleshooting tips from Fluke Networks,” Control Engineering, Mark T. Hoske, March 19, 2010.

Marty Jansons, network consultant at Siemens Industry, sees Internet protocol (IP) addressing faults as one of the biggest causes of network trouble. “If the IP addresses aren’t correct, or if someone puts in an IP address that may be already assigned to another node, those are the kinds of things that give you heartache.”

Michael Frayne, manager for network interfaces products at Molex, divides network infrastructure problems into three main issue categori

• Media (and related connections)
• Network configuration
• Faulty and/or nonconforming devices at nodes and in links.

“Problems related to application/network configuration are often harder to find,” Frayne says, “and may require the use of both a protocol-layer (SNMP, etc.) as well as a device-specific tool to get to the root cause.

Hardware or software? It ain’t easy

“How do you know if you are chasing down a hardware or software fault?” Frayne continues. “For example, media problems may require the use of a physical layer diagnostic tool that measures aspects like the signal quality on the media. Protocol-layer and/or device-specific diagnostics tools can be used to determine if a device has faulted due to an application condition or a serious malfunction.”

Table of Critical Ethernet Characteristics

1	Network layout	Do not take office topologies to the plant floor, but implement plant/machine topologies with Ethernet. Office Ethernet infrastructures are typically based on commercial-grade products meant to be located in a temperature controlled environment, and switches designed around a large star topology. Industrial Ethernet architectures, on the other hand, take into account different conditions and additional feature requirements, such as high-speed redundancy.
2	Protocols	The critical thing to know about Ethernet is that it’s just the wire. You need an industrial protocol to run automation applications.
3	Throughput	It’s not how fast the network is, but how quickly and predictably the data gets to where it’s needed.
4	Network configuration	It’s not just how easy it is to setup, but how little programming you have to do to make it run.
5	Planning ahead	It’s not only important to support today’s applications. You need the same Ethernet to support all your applications in the future.
6	Legacy systems	The critical thing about Industrial Ethernet is not just if it talks Ethernet, but how it integrates with already implemented networks and machines from different vendors.
7	Cost	The critical thing about the cost of an Industrial Ethernet network is not the cost of the components, but the cost of engineering, installation, and maintenance.

Source: “7 critical things to know about Ethernet,” Jeremy Bryant, Siemens Energy & Automation Inc., Control Engineering, Jan. 4, 2008

“You may have to use a combination [of hardware and software tools],” Jansons points out. He recommends starting with a network analysis tool, which involves monitoring network traffic with an auxiliary PC.

“You see a troubleshooting sequence,” White concludes. “As soon as you can isolate where there may be a problem, and isolate it to a certain area, [and down] to a certain machine. Any of the physical cabling instruments is going to [finish the job].”

Maintenance and post-failure diagnostic tools include handheld testers, in-line devices, laptop PC-based components, monitoring software, and other more application-specific accessories.

Rather than requiring an operator to interpret a signal based on experience, some newer devices extract key measurements from the complex network activity. This advanced capability is often accomplished by combining a protocol analysis and an electrical signal analyzer in one handheld tool.

Reflections of trouble

Another tool for troubleshooting physical-layer components is the time domain reflectometer (TDR). TDRs send pulses over network media and measure reflections to pinpoint any problems, such as shorts, opens, mis-wired connectors, excessive cable lengths, or incorrect cable types.

TDRs use the media’s impedance characteristics to quantitatively determine the degree (via the strength of reflections) and position (via the time the reflection reaches the TDR) of any significant faults.

A fault, such as a crushed cable, will manifest itself as a localized change in the cable’s surge impedance. When an electrical pulse reaches such a surge-impedance discontinuity, it reflects some of its energy back the way it came—toward the TDR. The sharper the discontinuity, the more strongly the signal reflects. The farther the discontinuity is from the TDR, the longer it will take for the reflection to reach the TDR. Knowing the cable’s nominal surge impedance, and the speed at which signals traverse the cable, the TDR can calculate the seriousness of the fault as well as its exact position.

It is very common requirement for new Ethernet media installations to be 100% tested and certified by the installer using a TDR. Media testers, such as cable analyzers, are available for industrial control networks and have significant value when used during installation to verify correct installation.

Network interface cards

Besides the usual handheld devices, network diagnostics can also be done with PCs and laptop computers combined with network interface cards (NICs) featuring built-in diagnostic capabilities and related software. “Usually,” Jansons reports, “the switches themselves already have built-in features that allow you to not only recognize these problems, but also, using open protocols, such as SNMP (simple network management protocol), can be leveraged to help you with troubleshooting not only at the device level, but at the port level.”

So, troubleshooting industrial Ethernet automation networks is a complex task not easily boiled down to a step-by-step method you can apply indiscriminately. It is more like a toolbox, whose tools you have to select ad hoc depending on the particular situation. Control Engineering articles advise on what tools to put into the toolbox, and when to pull them out for use.

Design, operations, repair

“Network diagnostics is really one of three phases involved in successful deployment of industrial networks,” Nick Jones, Woodhead Connectivity (now part of Molex), said in “Network diagnostic tools aided by intelligent design, maintenance,” December 2002 issue. “The first phase is proper design and installation, including appropriate diagnostic capabilities. The second is network operations, which includes periodic check-ups and proactive maintenance to ensure network health. The third phase is post-failure repair, which is when most traditional diagnostics are performed.”

“Industrial network selection advice from Rockwell Automation,” an Aug. 4, 2009, article, listed 10 characteristics that engineers must keep in mind when using industrial Ethernet to control factory automation systems. Steve Zuponcic, manager of IA Solutions and commercial engineering at Rockwell Automation, provided the list; See the table, 10 considerations for top network health and performance.

For more info, see:

www.flukenetworks.com

www.molex.com

www.moxa.com

www.rockwellautomation.com

www.sea.siemens.com

More network diagnostic resources

Below is a list of references, articles, and resources to date from Control Engineering:

• Podcast: Hardware and software to troubleshoot industrial Ethernet – Siemens Industry, Control Engineering
• Podcast: Troubleshooting and tools for industrial networks, Ethernet-Fluke Networks, Control Engineering

• Take the network diagnostics survey: If you buy or specify industrial network diagnostic devices or services, take Network Diagnostics Survey to make your opinions known. Survey will be open through month’s end or until a statistically valid sample is gathered.
• Webcast on industrial Ethernet at www.controleng.com/webcasts
• 10 network troubleshooting tips from Fluke Networks
• CC-Link IE extends 1 Gigabit industrial Ethernet to field devices, with copper
• ProSoft Technology 802.11n Industrial Hotspot Radios resolve tough wireless problems
• Industrial network selection advice from Rockwell Automation
• Network diagnostics: New system troubleshoots FOUNDATION fieldbus
• Network diagnostic tools aided by intelligent design, maintenance
• New tools improve network diagnostics, maintenance
• High Performance Industrial Networks – research
• Plug into Industrial Ethernet Protocols – research
• Wireless: No doubts about IEEE 802.11n advantages, say Rockwell Automation, Cisco
• Molex: New DeviceNet diagnostics, add-on profiles, integration help, proactive DeviceNet
• 7 critical things to know about Ethernet
• Flash video on network diagnostic software from Moxa

PHOTO GALLERY – Network diagnostics

Additional images related to network diagnostics follow.

Do you have experience and expertise with the topics mentioned in this content? You should consider contributing to our CFE Media editorial team and getting the recognition you and your company deserve. Click here to start this process.