Synchronizing industrial Ethernet networks

Automation engineers can develop architectures that meet the demands of their applications by understanding the differences between distributed clocks and the IEEE 1588 precision time protocol.
By Joey Stubbs, PE, EtherCAT Technology Group June 3, 2014

Figure 1: Whereas IEEE 1588 requires special and complex microcontroller-based hardware even for the most simple of digital I/O devices, EtherCAT DC devices can be implemented without microcontroller support, resulting in lower device and system costs. CoMany machine building and manufacturing engineers have gotten used to industrial Ethernet as a way of life. While they’re by no means extinct, traditional fieldbuses with custom hardware and cabling are in the rearview mirror for many enterprises. The question today is no longer whether to use industrial Ethernet; the real question is how to take full advantage of industrial Ethernet and how to ensure best practices are in place.

The general benefits of using one of the many industrial Ethernet protocols for machine networking are numerous, such as for I/O, the motion bus, and safety in many cases. But some protocols have the potential not just to greatly increase the speed of cycle times, but also to generate significant improvements in manufacturing precision, accuracy of processes, and system diagnostics.

The importance of synchronization

Synchronization, for example, is an area where the more advanced industrial Ethernet protocols can make an immediate mark. By implementing synchronization, a consistent time base can be created across applications for any number of spatially separated industrial Ethernet-connected devices and machine sections, e.g., for applications with multiple motion axes to achieve a path through space of a robotic arm, or those involving measurement technology for diagnostics of bearing wear. Synchronization is a key element in modern automation systems to ensure that both simple and complex devices are always synchronized to each other, to external applications, and to events in a reliable, repeatable manner. Device-level digital communication systems (fieldbuses) give the developers and end users of devices the opportunity to network these devices together, with industrial Ethernet being the most recent and most widely accepted entry into this market. However, what technologies are available today for synchronizing a network of Ethernet-connected devices, and how can this synchronization be used to the advantage of the automation engineer?

First, how should the term "synchronized system" be defined? Is it simply defined by how deterministically the frames are sent and received? Is it that the devices know what time the frame arrives and is sent? Also, how does one tackle commanding an I/O to turn on, or a motion move to begin at a future time? What about latching in the time of an external event? There are many factors relevant to the internal workings of the end devices that determine how well the device can interact with the environment, such as reacting to a time-valued command, or sensing when an external value has reached some limit. The answers will be different for each user. However, most will be more concerned with the need to ensure the input or output signal to the wire (or when motion begins, or when ∆T parameters are collected) is as controlled as possible, and with the least amount of jitter possible. Jitter is the variation from a perfect synchronization-or as good as it gets. Other concerns may include the complexity involved to implement and manage such a system.

This article examines two different approaches to industrial Ethernet network synchronization (see Figure 1). The first is IEEE 1588, which is used in many different fieldbus protocols, Internet applications, and switched network topologies. The second approach relates to the concept of distributed clocks as used by EtherCAT, an open real-time Ethernet network. These different approaches, with pros and cons, will be covered briefly.

IEEE 1588-PTP

Figure 2: This drawing shows synchronization messages sent between the BMC and slave clocks in an IEEE 1588 system. Courtesy: EtherCAT Technology GroupIEEE 1588 precision time protocol (PTP) offers a solution for implementing network-wide synchronization down to sub-microsecond levels across a variety of transmission media, including over Ethernet. This standard has been adopted by several of the industrial Ethernet fieldbus protocol providers as the means of achieving temporal synchronization in their respective technologies (see Figure 2).

The IEEE 1588-PTP standard does not specify some key parameters for devices, such as the type and frequency of device oscillators. Therefore, there can be various qualities of clocks in a given system, some better than others. A slower clock will have less time resolution and, therefore, will be less accurate with its time and timestamping. Because of this, the IEEE 1588 network has to determine the best master clock (BMC) to use in the network, a negotiation algorithm that involves communicating device parameters to each other and determining the best device to use as the reference clock. Obviously, if all the devices have lower-quality clocks, then the overall BMC may still not offer the performance the user desires. If the network becomes larger, spanning through several subnets may be required, and special switches with IEEE clocks built in will be required, each becoming the BMC of each subnet.

After determining the BMC, the time delay between the BMC and the other IEEE 1588 devices must be determined and periodic timing corrections to the BMC must be issued. Because in IEEE 1588 this has to occur in a switched network environment, the routing delays depend highly on the topology, and each additional switch not only adds delay, but additional devices on the network also increase network traffic, which means that traffic becomes more jittery and susceptible to being queued in a network switch, causing a considerable amount of jitter overall.

The actual timestamps that are accessed and read can vary somewhat in an IEEE 1588 system. The so-called asynchronous timestamp reads occur from one of two places: either a specialized Ethernet transceiver chip, or a specialized media access control-both accessed from software on the host microcontroller via an interrupt from either of these devices. Synchronous timestamp reads occur when the timestamp is inserted into the message as it enters the device. This requires additional specialized hardware, as the timestamp addition changes the cyclic redundancy code of the Ethernet frame, which must be recalculated and changed. The advantage here is that the timestamp and data match. Both of these approaches require a microcontroller (as well as software, RAM, etc.) to be designed into every slave device, which may be overkill for simple devices.

Distributed clocks push precision to the nanosecond level

Figure 3: This drawing shows how propagation delay between the reference clock and other DC devices is calculated. EtherCAT devices with and without DC functionality can be mixed freely with no impact to the synchronization quality of the network. CourtesAs another way to implement synchronization, EtherCAT is an example of an industrial Ethernet system that uses distributed clocks (DCs). These DCs are built into the associated EtherCAT slave controllers (ESCs), so there is no external circuitry, or special infrastructure required for implementing DCs. ESCs are the IC chips that implement the EtherCAT protocol in hardware. Devices with and without DC functionality can be mixed freely in the same network with no impact to the synchronization quality of the network. The EtherCAT specification dictates the frequency and quality of the DC device’s oscillator. Therefore, there is no need for a BMC determination to be made. Any DC device can be the reference master clock. By operating principle, the reference clock is always the first EtherCAT slave that has DC functionality enabled. The advantage of this is that there is no negotiation required, and all slave devices have the same oscillator quality. Furthermore, the method of distributing the time from the reference clock is extremely efficient and elegant (see Figure 3).

Because of the processing-on-the-fly operating principle of EtherCAT, every frame is effectively routed in a cut-through fashion through every slave of an EtherCAT network (up to 65,535 slaves). Regardless of which slave is, or is not, being addressed, every frame goes through each slave device in the same path every time (there is no active routing of frames). This results in the same timing throughout the network. Each slave can predictably calculate how long it takes for data to pass between the forward direction (Tx) and returning direction (Rx), and the master also knows the exact EtherCAT network topology. This is important because the master can easily calculate the propagation delay between any two points in the network with these sets of data, such as between its reference clock (always the first DC-enabled device in the network) and each additional DC-enabled slave. This calculation needs to be done only once for a given network, and is completely topology-independent.

After calculating the propagation delay from the reference clock to an individual slave, this value is given to the slave, and each slave receives its unique propagation delay value. This is done so that each slave can set and maintain its local clock to that of the reference clock. For drift compensation, the master simply adds a very small instruction (16 bytes) to every cyclic frame, which grabs the time from the reference clock and distributes it to all other slaves. Each slave will compare the sum of the received time value from the reference clock, plus the propagation delay to its own local clock value and determine if it is running fast or slow. Compensation is done simply by adjusting how much time is counted for each pulse of its local oscillator, thereby closing a phase loop to the reference clock.

The performance of DCs is independent of the network topology, number of slave devices, and jitter of the frames coming from the master. Real-world jitter values of less than ±100 nanoseconds are regularly achieved. The main factor in tightening the phase lock loop is simply the need to issue the time distribution command more often. Because neither the jitter of the network nor the timekeeping of the individual slave devices is impacted by any jitter of the Ethernet frames, the EtherCAT network doesn’t require any special master card to facilitate jitter-free frames. As long as the data are received by the farthest slave prior to the network-wide time interrupts to use it, there are no problems with frame jitter. All EtherCAT masters calculate how far in advance they need to begin sending out the frame based on any NIC jitter, any delay and jitter in the network, as well as the length of the frame itself.

Additionally, the DC unit that facilitates this synchronization doesn’t require the additional complexity or cost of a microcontroller. So, even low-cost digital I/O can be constructed of only the EtherCAT slave controller chip, an EEPROM, and the driver circuitry for the input/output signals. Yet this can deliver output signals that can be commanded down to the nanosecond, or latch in time values on rising or falling signal edges (configurable) with the same nanosecond resolution, all without the addition of a microcontroller.

Use of time by field devices

Figure 4: This graph shows how oversampling can enable extremely short measurements. Courtesy: EtherCAT Technology GroupHaving network-wide synchronized clocks is nice, but they are ultimately useless unless users can implement relevant features and benefits with them. Getting the slave device to behave in relation to the time is the ultimate goal of a synchronized system. Here, the differences between IEEE 1588 and DCs are quite apparent.

IEEE 1588 devices are typically somewhat complex because a microcontroller is required regardless of whether the core function of the device is a simple one (such as a digital I/O) or not. However, the devices will typically be implemented within the slave with the microcontroller running some kind of software, which reads directly from time registers to facilitate any and all functions, from simple digital signals to complex motion profiles.

On the other hand, the internal DC unit inside the EtherCAT slave controller is tied directly to the internal frame transceivers, the logical processing unit, an interrupt request to an optional microprocessor, and to a set of input and output pins. These input (latch) and output (sync) pins can be used directly as I/O signals for simple digital devices, or can be used with a microcontroller to achieve interrupts that can enable more advanced functionality.

At the basic level of device implementation, the sync pins can be used as outputs for digital signals. Associated with each sync pin is a time value register. When a value is set to this register, the sync pin will fire automatically with nanosecond resolution after the local time reaches this preset. In this way, a very inexpensive digital output module can be constructed, because there is no accompanying cost of a microcontroller, RAM, or software stack. These devices will have very low jitter, and can be synchronized to other network-wide DC devices in the network.

Again, the corollary to the sync pins are the input latch pins. These pins can be used as input pins in a simple I/O device, because a time value will latch into their associated registers when the configured rising or falling edge is detected. Both the sync and latch pins give functionality that is not tied to the scanning of a traditional PLC, but allows input capture or output command to occur at any point in time with nanosecond resolution.

When precise input time can be paired with precise output response, the user can measure the exact time that an event occurred and command an exact time to react to this event. This can include, for example, reacting to an alarm in a manufacturing line and calculating the future time required to avoid the rejection of product that is not at risk, but reliably eliminating the product that is at risk.

Oversampling can be accomplished through the use of subordinated interrupts, which will allow a device to take many samples of a signal at a rate that can be a multiple of the controller scan rate. This permits the capturing of an event (or even multiple events) that would otherwise be invisible to a traditional PLC, because their duration is short enough to fit between PLC scans. Similarly, oversampling digital output devices can generate pulse trains that would be impossible to detect with simple I/O, which at the maximum would only be able to create square waves of half the scan frequency (see Figure 4).

Using oversampling with analog capture ability enables analog modules to capture or create waveforms, which can add great functionality to a high-speed process. Continuously running oversampling modules can be used for adding condition monitoring and preventive maintenance functionality to an automation process, where fast Fourier transform (FFT) algorithms can self-diagnose motors, bearings, or gears for wear and tear. All this is possible while providing the control of the process and equipment itself.

The preceding paragraphs are not meant to imply that DCs and IEEE 1588 are necessarily rival synchronization schemes, nor that they are incompatible. The key point is that the internal synchronization methodology inside EtherCAT is based on DCs because of the aforementioned benefits of simplicity, cost efficiency, and flexibility in design. Also, DCs are already built into the EtherCAT slave controllers, so there are no additional hardware requirements. As a matter of fact, there are several sources of EtherCAT-to-IEEE 1588 boundary clocks that allow the bridging of time values from one to another. These are used for synchronizing an EtherCAT network to an exterior IEEE 1588 timing source, such as from a grandmaster clock, or another fieldbus system that uses IEEE 1588.

Conclusion

Both IEEE 1588 and DCs offer the automation engineer the ability to implement a network of highly synchronized devices spread across a large area and long network distances. Whereas IEEE 1588 offers a viable solution for Internet- and switch-based protocols, EtherCAT uses the more streamlined, bandwidth-efficient DC solution, which also ensures very low jitter. Whereas IEEE 1588 requires special and complex microcontroller-based hardware even for the most simple of digital I/O devices, EtherCAT DC devices can be implemented without microcontroller support, resulting in lower device and system costs. These two synchronization schemes can still be used together, bridged via boundary devices. This allows the network time to be shared between an EtherCAT network and an IEEE 1588-based system. Between these two synchronization schemes, automation engineers can develop bottom-up or top-down architectures to meet the demands of their application or their customers.

About the author

Joey Stubbs is the North American representative of the EtherCAT Technology Group. He is a registered professional engineer in control systems and holds a BS in electrical engineering from the University of South Carolina, as well as several other technical degrees. He has nearly 30 years of industrial experience in implementing industrial Ethernet technologies, automation projects, motion control, nuclear power, and power distribution.

This article appears in the Applied Automation supplement for Control Engineering and Plant Engineering