Cover story: Troubleshooting Industrial Ethernet

It’s a problem when gremlins invade factory automation systems: Is the network at fault? As industrial automation systems gravitate more to industrial Ethernet, tools and techniques have improved to troubleshoot network problems. See the advice below; take an online network diagnostics survey. Hear podcast network troubleshooting interviews.

10/13/2010


Control Engineering October 2010 coverWhen John Lennon famously complained in song that “…it ain’t easy, you know how hard it can be,” he wasn’t talking about troubleshooting industrial Ethernet automation networks, but he could have been. As industrial automation systems gravitate more to industrial Ethernet, it is important to look at the tools and techniques available to troubleshoot problems on these networks.

The Control Engineering Network Diagnostics Survey asks readers about their experiences setting up and maintaining automation networks. What protocol standards do they use? How well do these networks hold up? What tools do they have available, and what tools are needed? If you buy or specify industrial network diagnostic devices or services, go to www.controleng.com under the events and awards menu and select Network Diagnostics Survey to make your opinions known. (Hurry, the survey will be open only through month’s end, or until we have a statistically valid sample.)

Cover Story summary: Industrial Network Diagnostics

-Industrial network troubleshooting help for Ethernet, others

 

- Take a network diagnostic survey-Survey is closed. For all surveys we have made public, and research from others, click here.

 

-Hardware, software, diagnostic tools

 

-Ethernet webcast; photo gallery

 

-Podcasts - additional network tips, advice

 

-More than a dozen links to additional resources, information

 

 

“Over 80% of the calls that come into our facility,” says Mara White, market development manager for Fluke Networks, “involve physical infrastructure problems, and I’ve heard the same thing from automation companies.”

Checklist

10 considerations for top network health and performance

 

1. Use the appropriate test equipment to collect information from the network

 

2. Check for return loss

 

3. Ensure the cable stays dry

 

4. Consider insertion losses

 

5. Higher temperature can decrease reliability

 

6. Installation materials may appear to affect cabling performance

 

7. Look at dc resistance

 

8. Check the fiber connectors

 

9. Clean carefully

 

10. Be careful not to add loss during network design

 

Source: 10 network troubleshooting tips from Fluke Networks,” Control Engineering, Mark T. Hoske, March 19, 2010.

 

Handheld portable network analyzers simplify troubleshooting automated testers. Source: Fluke NetworksMarty Jansons, network consultant at Siemens Industry, sees Internet protocol (IP) addressing faults as one of the biggest causes of network trouble. “If the IP addresses aren’t correct, or if someone puts in an IP address that may be already assigned to another node, those are the kinds of things that give you heartache.”

Michael Frayne, manager for network interfaces products at Molex, divides network infrastructure problems into three main issue categori

  • Media (and related connections)
  • Network configuration
  • Faulty and/or nonconforming devices at nodes and in links.

“Problems related to application/network configuration are often harder to find,” Frayne says, “and may require the use of both a protocol-layer (SNMP, etc.) as well as a device-specific tool to get to the root cause.

Hardware or software? It ain’t easy

“How do you know if you are chasing down a hardware or software fault?” Frayne continues. “For example, media problems may require the use of a physical layer diagnostic tool that measures aspects like the signal quality on the media. Protocol-layer and/or device-specific diagnostics tools can be used to determine if a device has faulted due to an application condition or a serious malfunction.”

 

Table of Critical Ethernet Characteristics

1

 

Network layout

 

Do not take office topologies to the plant floor, but implement plant/machine topologies with Ethernet. Office Ethernet infrastructures are typically based on commercial-grade products meant to be located in a temperature controlled environment, and switches designed around a large star topology. Industrial Ethernet architectures, on the other hand, take into account different conditions and additional feature requirements, such as high-speed redundancy.

 

2

 

Protocols

 

The critical thing to know about Ethernet is that it’s just the wire. You need an industrial protocol to run automation applications.

 

3

 

Throughput

 

It’s not how fast the network is, but how quickly and predictably the data gets to where it’s needed.

 

4

 

Network configuration

 

It’s not just how easy it is to setup, but how little programming you have to do to make it run.

 

5

 

Planning ahead

 

It’s not only important to support today’s applications. You need the same Ethernet to support all your applications in the future.

 

6

 

Legacy systems

 

The critical thing about Industrial Ethernet is not just if it talks Ethernet, but how it integrates with already implemented networks and machines from different vendors.

 

7

 

Cost

 

The critical thing about the cost of an Industrial Ethernet network is not the cost of the components, but the cost of engineering, installation, and maintenance.

 

Source: “7 critical things to know about Ethernet,” Jeremy Bryant, Siemens Energy & Automation Inc., Control Engineering, Jan. 4, 2008

 

This is a good waveform for a CAN-based network. Source: Molex“You may have to use a combination [of hardware and software tools],” Jansons points out. He recommends starting with a network analysis tool, which involves monitoring network traffic with an auxiliary PC.

“You see a troubleshooting sequence,” White concludes. “As soon as you can isolate where there may be a problem, and isolate it to a certain area, [and down] to a certain machine. Any of the physical cabling instruments is going to [finish the job].”

This CAN network waveform shows a short. Source: MolexMaintenance and post-failure diagnostic tools include handheld testers, in-line devices, laptop PC-based components, monitoring software, and other more application-specific accessories.

Rather than requiring an operator to interpret a signal based on experience, some newer devices extract key measurements from the complex network activity. This advanced capability is often accomplished by combining a protocol analysis and an electrical signal analyzer in one handheld tool.

Reflections of trouble

Software can monitor network faults such as loss of power, broken cables, or traffic overloads. Source: MoxaAnother tool for troubleshooting physical-layer components is the time domain reflectometer (TDR). TDRs send pulses over network media and measure reflections to pinpoint any problems, such as shorts, opens, mis-wired connectors, excessive cable lengths, or incorrect cable types.

TDRs use the media’s impedance characteristics to quantitatively determine the degree (via the strength of reflections) and position (via the time the reflection reaches the TDR) of any significant faults.

After an industrial network fault a message can be sent via text message, email, and / or a local alarm. Source: MoxaA fault, such as a crushed cable, will manifest itself as a localized change in the cable’s surge impedance. When an electrical pulse reaches such a surge-impedance discontinuity, it reflects some of its energy back the way it came—toward the TDR. The sharper the discontinuity, the more strongly the signal reflects. The farther the discontinuity is from the TDR, the longer it will take for the reflection to reach the TDR. Knowing the cable’s nominal surge impedance, and the speed at which signals traverse the cable, the TDR can calculate the seriousness of the fault as well as its exact position.

It is very common requirement for new Ethernet media installations to be 100% tested and certified by the installer using a TDR. Media testers, such as cable analyzers, are available for industrial control networks and have significant value when used during installation to verify correct installation.

Network interface cards

Network devices also work inline, designed to connect and monitor, sending alerts as needed. Source Molex.Besides the usual handheld devices, network diagnostics can also be done with PCs and laptop computers combined with network interface cards (NICs) featuring built-in diagnostic capabilities and related software. “Usually,” Jansons reports, “the switches themselves already have built-in features that allow you to not only recognize these problems, but also, using open protocols, such as SNMP (simple network management protocol), can be leveraged to help you with troubleshooting not only at the device level, but at the port level.”

So, troubleshooting industrial Ethernet automation networks is a complex task not easily boiled down to a step-by-step method you can apply indiscriminately. It is more like a toolbox, whose tools you have to select ad hoc depending on the particular situation. Control Engineering articles advise on what tools to put into the toolbox, and when to pull them out for use.

Design, operations, repair

“Network diagnostics is really one of three phases involved in successful deployment of industrial networks,” Nick Jones, Woodhead Connectivity (now part of Molex), said in “Network diagnostic tools aided by intelligent design, maintenance,” December 2002 issue. “The first phase is proper design and installation, including appropriate diagnostic capabilities. The second is network operations, which includes periodic check-ups and proactive maintenance to ensure network health. The third phase is post-failure repair, which is when most traditional diagnostics are performed.”

“Industrial network selection advice from Rockwell Automation,” an Aug. 4, 2009, article, listed 10 characteristics that engineers must keep in mind when using industrial Ethernet to control factory automation systems. Steve Zuponcic, manager of IA Solutions and commercial engineering at Rockwell Automation, provided the list; See the table, 10 considerations for top network health and performance.

For more info, see:

www.flukenetworks.com

www.molex.com

www.moxa.com

www.rockwellautomation.com

www.sea.siemens.com

More network diagnostic resources

Below is a list of references, articles, and resources to date from Control Engineering:

PHOTO GALLERY - Network diagnostics

Additional images related to network diagnostics follow.

Technician uses a handheld portable network analyzer to troubleshoot automated controls on a packaging line. Source: Fluke Networks

Cable analyzers are programmed to perform multiple tests needed to troubleshoot fiber optic and twisted-pair copper Ethernet links.  Source: Fluke Networks



No comments
The Engineers' Choice Awards highlight some of the best new control, instrumentation and automation products as chosen by...
Each year, a panel of Control Engineering editors and industry expert judges select the System Integrator of the Year Award winners.
Control Engineering Leaders Under 40 identifies and gives recognition to young engineers who...
Learn more about methods used to ensure that the integration between the safety system and the process control...
Adding industrial toughness and reliability to Ethernet eGuide
Technological advances like multiple-in-multiple-out (MIMO) transmitting and receiving
Virtualization advice: 4 ways splitting servers can help manufacturing; Efficient motion controls; Fill the brain drain; Learn from the HART Plant of the Year
Two sides to process safety: Combining human and technical factors in your program; Preparing HMI graphics for migrations; Mechatronics and safety; Engineers' Choice Awards
Detecting security breaches: Forensic invenstigations depend on knowing your networks inside and out; Wireless workers; Opening robotic control; Product exclusive: Robust encoders
The Ask Control Engineering blog covers all aspects of automation, including motors, drives, sensors, motion control, machine control, and embedded systems.
Join this ongoing discussion of machine guarding topics, including solutions assessments, regulatory compliance, gap analysis...
News and comments from Control Engineering process industries editor, Peter Welander.
IMS Research, recently acquired by IHS Inc., is a leading independent supplier of market research and consultancy to the global electronics industry.
This is a blog from the trenches – written by engineers who are implementing and upgrading control systems every day across every industry.
Anthony Baker is a fictitious aggregation of experts from Callisto Integration, providing manufacturing consulting and systems integration.
Integrator Guide

Integrator Guide

Search the online Automation Integrator Guide
 

Create New Listing

Visit the System Integrators page to view past winners of Control Engineering's System Integrator of the Year Award and learn how to enter the competition. You will also find more information on system integrators and Control System Integrators Association.

Case Study Database

Case Study Database

Get more exposure for your case study by uploading it to the Control Engineering case study database, where end-users can identify relevant solutions and explore what the experts are doing to effectively implement a variety of technology and productivity related projects.

These case studies provide examples of how knowledgeable solution providers have used technology, processes and people to create effective and successful implementations in real-world situations. Case studies can be completed by filling out a simple online form where you can outline the project title, abstract, and full story in 1500 words or less; upload photos, videos and a logo.

Click here to visit the Case Study Database and upload your case study.