Expanded online version: Network diagnostic tools aided by intelligent design, maintenance

By Control Engineering Staff January 2, 2003

Back to main article

SIDEBARS: Balance alleviates network demons | Laundry list for diagnosing networks | IT managers shine at proactive network maintenance

Network diagnostics begins with defining, knowing and understanding your network. On the plant floor, there are generally two kinds of networks. The first are information networks that are designed to move large amounts of data, almost exclusively via Ethernet, but for which timely delivery of an individual message isn’t an especially high priority. The second are control networks and fieldbuses that generally move smaller volumes of data and place a lot more emphasis on quality and/or timeliness of delivery of each individual message or piece of information.

Network diagnostics is really one of three phases involved in successful deployment of industrial networks. The first phase is proper design and installation, including appropriate diagnostic capabilities. The second is network operations, which includes periodic check-ups and proactive maintenance to ensure network health. The third phase is post-failure repair, which is when most traditional diagnostics are performed. Problems that lead to failure can include catastrophic breakdowns, when a network ceases to function, or slowly escalating problems, when network performance degrades over time.

Plant-floor engineers traditionally focus on rapid identification and repairing of problems, which is limited to the third phase. However, information technology (IT) managers know that by focusing on the first two phases, they can dramatically reduce problems in that third phase.

Maintenance and post-failure diagnostic tools include handheld meters, in-line devices, laptop PC-based components, monitoring software and other more application-specific accessories.

Handhelds, other tools

Though they appear to have evolved from traditional voltmeters and multi-meters, handheld network diagnostic tools have more in common with advanced test equipment. They still gather electrical signals, but, rather than requiring an operator to interpret a signal based on experience, these newer devices extract key voltage measurements from the complex network activity. This advanced capability is often accomplished by combining a protocol analyzer and an electrical signal analyzer in one handheld tool.

The IT world has its own array of specialized tools. Network sniffers are devices that are permanently connected to networks to monitor them and provide diagnostic data. They too monitor a network’s wires; examine the protocols that are running; and help determine if the network is healthy.

Another tool familiar to IT professionals is the time domain reflectometer (TDR). TDRs send pulses over network media and measuring media and measure reflections to pinpoint any problems, such as shorts, opens, mis-wired connectors, excessive cable lengths or incorrect cable types. It is very common requirement for new Ethernet media installations to be 100% tested and certified by the installer using a TDR. Media testers are also available for industrial control networks and have significant value when used during installation to verify correct installation. The downside is that active testers such as these must be used on an inactive network, often with nodes disconnected.

Besides the usual handheld devices, network diagnostics can also be done with PCs and laptop computers combined with network interface cards (NICs) featuring built-in diagnostic capabilities and related software.

Because different diagnostic tools measure different phenomena, engineers often have to use more than one. For example, using a TDR on cables before powering up can help confirm the physical layer. Next, protocol and electrical analyzers can check the network’s electrical characteristics and performance, which includes network error rates and bandwidth utilization. Numerous tools are available, check with the protocol organization that applies to your network, whether its DeviceNet, Profibus, Foundation fieldbus or another protocol, for appropriate diagnostic tools and methods.

Baselines, benchmarks

Besides gathering and analyzing data, network diagnostic tools also enable you to compare current conditions against historical precedent to detect deterioration. Baselines can either be recorded at installation, documenting the as-built characteristics of a network, or when a network is being certified or commissioned.

In fact, being diligent at the installation phase can not only prevent problems later, it can also aid proactive maintenance efforts using existing diagnostic equipment. Many users tend to buy and connect components; turn on their networks; and then forget about them-until they fail. However, carefully planning and designing a network-essentially fulfilling the needs of its installation, operations/maintenance and repair phases preemptively-is more efficient than trying to find and fix problems after they happen.

Nick Jones, staff researcher, Woodhead Connectivity (Waterloo, Ontario, Canada) and chief technical officer, Open DeviceNet Vendors Association (ODVA, Boca Raton, FL)

Balance alleviates network demons

Networks specifications generally strike a balance between several competing requirements, including data rate, distance and noise immunity. Improving one characteristic of a network often results in degradation of another.

Noise immunity

Data rate

Distance

Other network features, such as media access method-deciding who gets to talk next on a shared media-and integrated power supply in the cable, can have side effects. These also have to be considered when defining physical-layer specifications and guidelines.

Proper installation strikes the right balance in a network deployment to combat potentially demonic problems from hampering network performance. These include:

Bus errors, often intermittent and usually caused by physical layer problems, can be tracked down using diagnostic tools to check error rates for each node.

Marginal media, also includes a range of obvious and subtle problems, such as loose, corroded or improperly installed connections or cables; faulty device circuitry; and topology-related problems.

Noise, or signal interference that is caused by external influences, which are either electric, electromagnetic or magnetic.

Source: Control Engineering with data from ”DeviceNet Physical Layer: An Insider’s View” and ”Three Hidden Demons in Your Network” by Nick Jones, staff researcher, Woodhead Connectivity (Waterloo, Ontario, Canada), and available at www.myNetAlert.com www.mysst.com

Laundry list for diagnosing networks

To proactively diagnose industrial networks and fieldbuses, users should perform the following tasks:

1) Secure the installation guide for the specific type of network you’re installing from your vendor or the applicable network organization;

2) Read it and understand it;

3) Make sure your sub-contractor completes steps 1) and 2), and then installs the network exactly as specified; follows its rules precisely; and doesn’t try to bend them;

4) Verify. Use network analysis and diagnostic tools to make sure your network’s characteristics are within acceptable limits.

5) Baseline. When verifying initial measurements, record them all. Seek software-based and online tools for automating this process.

6) Monitor. Perform periodic check-ups of your network, and compare its performance to the baseline.

7) If and/or when a problem or failure occurs, use diagnostic tools and baseline data to complete repairs more quickly and efficiently.

IT managers shine at proactive network maintenance

Because they usually concentrate on network uptime during all three phases-design/installation, operation/maintenance and repair-of the typical network lifecycle, information technology (IT) managers often do a better job of keeping networks running than their counterparts in control and automation.

IT managers focus on average delivery times and mean throughput, though they generally don’t experience catastrophic outcomes if a single message is delayed. Meanwhile, plant floor/industrial engineers have historically had to consider worst-case scenarios and design their networks to meet criteria for performing in these conditions. There’s increasing overlap between these two groups, though still not much understanding of each other. Both move information, but their goals are different.

Large IT departments and internet service providers (ISPs) have networks that can’t be allowed to go down, and there is certainly one thing the control and automation staffers can learn from them: It’s better to focus on maintaining healthy networks rather than repairing faulty ones.

I think the reason automation users tend to avoid proactive measures comes down to financial organization. For instance, organizational groups responsible for capital budgets are usually responsible for designing and installing new systems. However, after installation, these systems are turned over to a second group to operate and maintain.

Unfortunately, these two groups have goals that conflict. The first group wants to minimize installed costs, while the second seeks to minimize operating costs. Because the two groups operate independently, the first group is unlikely to design and install features that increase operating uptime, or reduce fault diagnosis time, if those features also increase the installed cost.

In my experience automation users are enthusiastic about 24/7 diagnostic products and would love to add these capabilities, but they can’t because they aren’t part of the original specification and the resulting increased cost isn’t in their group’s budget. It’s these organizationally split capital expenditure budgets and runtime maintenance budgets that form a major obstacle to optimizing return on investment (ROI) and total cost of ownership (TCO).

In fact, it’s often practical to increase ROI and reduce TCO by adding less than 10% to the installed cost on many projects. This can eliminate significant downtime at a much lower cost than simple reliance on post-failure diagnostic capability. The trick is to get all the groups, including upper management and the financial staff, to focus on TCO.

-Nick Jones, Woodhead Connectivity, Waterloo, Ontario