Understanding fiber-optic network technology for SCADA
Supervisory control and data acquisition (SCADA) networks have undergone significant changes since the start of the decade as companies move to modernize their systems, improve security, and reduce networking costs. But as these networks need more bandwidth to support security, Internet of Things (IoT) sensor data, and other application data traffic, many organizations are considering moving to fiber-optic networks for their next-generation SCADA networks.
While several other contributors to this magazine have covered issues related to fiber-optic cables and optics (see, for example, "Fiber optics: A backbone for advanced building design" in sister publication Consulting-Specifying Engineer), there is still a need to know more about network technology choices for building out these new networks.
One of the biggest changes in SCADA networking has been the evolution from serial networking protocols to the Internet protocol (IP), the ubiquitous networking technology for the rest of enterprise networks. Adopting IP for SCADA networks means equipment costs can be less expensive, and bandwidth can scale up to 10 Gbps for end stations and up to 100G bps for backbone networks.
Moving to IP also has provided a means to better integrate SCADA-based operational transformation (OT) systems with an organization’s information technology (IT) systems to set the stage for IoT, facilitate better and lower-cost tracking of network conditions, and provide access to better data analysis tools for operational data management.
Still, not every organization has moved to IP, and even some of those that have done so have not made a wholesale shift. Some are still cautious about the ability of a packet-oriented technology like Ethernet to replace deterministic serial networks that are based on time-division multiplexor (TDM) technology with guaranteed data delivery. Most SCADA applications have real-time data needs that require 99.999% network reliability and low end-to-end delay.
But only recently, with the introduction of ITU G.8032 (an International Telecom Union standard) has Ethernet had a standard for 50 ms failover that delivers levels of reliability that match serial protocols. Many industrial network managers are just now becoming comfortable with this technology.
Video, IoT are increasing bandwidth requirements
Another network evolution is the need for SCADA backbone networks to support other data flows from equipment in the substation. Surveillance video data transport is an immediate need. There is a growing demand for better physical security at substations, which typically means increased video surveillance-leveraging the SCADA backbone network to backhaul the video data to a central location for monitoring.
In the not too distant future, these networks will require greater capacity to support sensors that monitor a wide range of industrial processes as part of the emerging IoT applications. Some pundits are predicting that networks could support thousands of sensors that will be monitoring processes and equipment and reporting back into a central console over the network.
Not only is IoT driving more bandwidth, it is also driving a consolidation of OT and IT systems to better process valuable production data and enable faster decision making based on this information.
Moving to fiber optics
Even though all fiber-optic networks provide bandwidth, transmission distance, security, and electromagnetic interference (EMI) advantages, it is important to choose the right network technology as well.
Network technology decisions can provide added reliability, manageability, and redundancy. However, each network technology implementation will have its own cost considerations involving the total capital costs required to build the network in addition to ongoing operating expenses.
Networks can be built in a wide variety of topologies, but the two main topologies that should be considered for a fiber-optic SCADA network are:
- Ring: A ring topology is a network in which each network node (i.e. remote facility) is connected to its adjacent nodes in a logical ring fashion so that data travels around the ring until it reaches its destination. Ring networks are the easiest to build and to scale-a new node must simply connect to its peers in any part of the ring-as long as the distance that packets travel around the circumference of the ring is within the latency requirements of the network protocol. Cable redundancy must be built into a ring network to protect against node failures or cable breaks.
- Mesh: In a fully connected mesh network, each node is directly connected to every other node, and data can be routed to any node on the network with very low delay. Mesh networks are complex and costly to scale though, because each new node added to the network requires a quadratic increase in connections for every node in the network.
There are three main network technologies to consider in building a fiber-optic SCADA network.
Learn more about the three technologies for building a fiber-optic SCADA network.
First network technology: Coarse wave division multiplexing
With coarse wave division multiplexing (CWDM), up to 16 wavelengths of light are transmitted across a pair of fiber cables; each wavelength is an independent data channel for a separate flow of data up to 10 Gbps. CWDM networks operate in a ring topology.
CWDM does provide flexibility and bandwidth, but perhaps the biggest advantage of the technology for SCADA networks is its simplicity. CWDM is a passive technology that can support transport of any protocol over the link, as long as it is at a specific wavelength (i.e. serial bit streams over fiber at 1570 nm, alongside 10 Gbps Ethernet at 1590 nm).
This allows network managers to build a backbone that can be upgraded as the network evolves. If a new network type is supported, 10 Gb Ethernet for example, then any open channel can be configured for this data. This is because the multiplexor simply refracts light at any network speed, regardless of the protocol being deployed.
One downside to CWDM networks is the "inefficiency" of the solution. Like any circuit-oriented technology, backbone channels are tied up, meaning that if they are not being used, their bandwidth is not available for other networks that may be flooded with packets. This is in comparison to packet-switched networks where there are no guaranteed channels, but the full bandwidth of the link is available to the transmitting network.
Second network technology: Multiprotocol label switching
Multiprotocol label switching (MPLS) is a technology that encapsulates data into special packets that include a "label" that is used to switch the packet to its destination.
MPLS is an open standard via the Internet Engineering Task Force (IETF). It has been rapidly adopted by almost every major telecommunications service provider as a platform for supporting thousands of customers over a common infrastructure. It is heavily used in service provider and enterprise networks and is a compelling choice for SCADA networks.
In MPLS, when the packet enters a network, it is assigned a route called a forwarding equivalence class (FEC). Each router knows that packet’s FEC thanks to its label-a bit sequence that identifies the FEC. Not only does the FEC indicate the path through the network, it also tells the router how to handle the data flow. The FEC appended to video data packets, for example, will map that data flow to a low-latency path.
Because of its design, MPLS can transport many different types of payloads. In a SCADA application, this could include serial bit streams, IP packets, video data streams, and others. This flexibility makes MPLS a viable option for building a modern network that also supports legacy data formats.
There are a number of factors that make MPLS a great choice for a SCADA network. It’s a mature and reliable technology that is proven in large-scale networks. It offers a flexible network architecture that supports the connection of remote substations either in redundant network rings or in a linear structure where data flows in a line between each substation.
However, most MPLS networking equipment is considered carrier grade for use in telecommunications networks. This is good because carrier grade equipment has increased reliability, but it also means that MPLS networks will be more expensive to build and to operate.
Third network technology: EtherNet/IP backbone network
A third option is to create a routed EtherNet/IP backbone protocol. Perhaps the biggest advantage of Ethernet is its flexibility and versatility. Ethernet has a very wide bandwidth range with standards ranging from 10 Mbps to 100 Gbps. Ethernet can be easily deployed over either copper or fiber-optic media.
Ethernet can be deployed in a mesh or ring topology, and many Ethernet products have been hardened as well to provide the ruggedness and wide operating temperature range (-40 to 75 C/-40 to 167 F) required for remote locations or outdoor use.
Most data protocols can be packetized for an Ethernet network, and with support for 50 ms Ethernet failover capabilities and quality of service functionality, Ethernet brings high-quality wide area network (WAN) features into what’s normally considered a local area network (LAN) technology.
Ethernet packets can be routed based on their IP addresses (layer 3 address), which for SCADA networks is the key to a well-controlled network. IP protocols work by broadcasting data packets to all stations within a broadcast domain. This requires each device on the network to examine that data packet and discard or accept it as appropriate. Larger broadcast domains mean each device has to process more packets. Creating a reasonable layer 3 broadcast domain can help to limit this network flooding and boost network capacity.
The Power over Ethernet (PoE) standard is another advantage of leveraging an Ethernet network. PoE allows power to be delivered over the same Cat 5 or Cat 6 copper cable that transmits Ethernet data. This means that devices such as IP cameras, gas analyzers, and embedded computers can be conveniently located without the cost of installing an additional power spur.
There are two PoE standards: IEEE 802.3af delivers up to 15.4 W (good for a VoIP phone or WiFi access point); and the newer Power over Ethernet Plus (PoE +) standard (IEEE 802.3at) delivers up to 25.5 W in usable power while also remaining backward compatible with the current standard.
It should be said, however, that any SCADA network that deploys Ethernet as an access network can benefit from PoE or PoE+ even if another type of backbone network technology is selected.
With the steady demand for increased bandwidth and speeds, fiber-optic networks are a clear choice for SCADA network and remote communications needs. This overview of network technologies is a good starting point for providing the information needed to deploy the network that is the best fit for your organization.
Tony LeFebvre, director, product management, Transition Networks. Edited by Chris Vavra, production editor, CFE Media, Control Engineering, email@example.com.
- Fiber-optic networks are a good choice for SCADA networks and remote communication needs because of the demand for increased bandwidth and speed.
- One of the biggest changes in SCADA networking has been the evolution from serial networking protocols to Internet protocol (IP).
- There are three main network technologies to consider in building a fiber-optic SCADA network: Coarse wave division multiplexing, Multiprotocol label switching, and EtherNet/IP backbone networking.
What other networks could be used in building a fiber-optic SCADA network?
See related stories about the IoT and SCADA networks linked below.