Ethernet hardware webcast questions and answers
Some of the questions raised during the Control Engineering webcast on Ethernet hardware were answered during the live question-and-answer session (available for viewing as part of the 1-hour webcast). Questions among Control Engineering registrants’ included Ethernet network design, Ethernet switches, Ethernet protocols, network troubleshooting, network security, and Power over Ethernet-related topics. Kurt Forster, an industrial network expert with Autopro Automation Consultants Ltd., provided advice on Ethernet hardware for the webcast and answered additional questions that didn’t fit into the 1-hour webcast, below. Registrants to the Control Engineering RCEP-accredited webcast are eligible for a professional development hour (PDH) after viewing and passing a quiz. See the Ethernet Hardware webcast here.
Audience question: What is the recommended separation between industrial and IT networks?
Answer from Kurt Forster: There are many different ways to separate the industrial networks from the enterprise. These are the most common:
1. Full air gap is a total segregation between the two infrastructures with no possible connectivity or direct data transfer between the two.
2. On-command air gap. This is the same as No. 1; however, when asked to do so and permission is granted, a cable between the two infrastructures would be connected and enabled for an amount of time. This would then be disconnected once the session was finished.
3. Single firewall pass-through is when you have a firewall in between the two infrastructures, and a select set of clients are allowed through.
4. Single firewall and an automation demilitarized zone (DMZ). It’s the same as No. 3, without a pass-through. All data ends in a DMZ zone. (Firewall and DMZ are owned and controlled by the integrated control systems – information systems (ICS-IS) team
5. Double firewall shared DMZ is where one firewall on the enterprise connects to an ICS switch. From that ICS switch the automation firewall also would connect into it. (The DMZ space would be shared between information technology/information systems (IT/IS) and ICS-IT. Most servers and computers would be dual-homed, or it would be a shared IP range.)
6. ICS-IS firewall to ICS/IS boundary router with a DMZ coming off of the firewall would run from the boundary firewall into an IT/IS firewall with a DMZ coming off of the firewall.
Recommendation: It depends on who administrates the infrastructures above. However, I feel that No. 6 is the best and recommended solution as there are clear defined DMZs from both sides of the boundary router.
Question: What are the recommended ways of connecting industrial Ethernet to legacy networks?
Answer: This depends on what you call a legacy network and which legacy network is deployed. However, if we are talking about a token ring network or a control network, then normally you would have devices with the protocol network interface card (NIC), for Modbus, etc., in a PC and a second Ethernet NIC in the PC. This PC would normally be a historian, which would be able to push the data to the new historian or supervisory control and data acquisition (SCADA) server. This question is too broad to recommend one clear solution. However, if you build the new network as a ISA95 Purdue level and connect the legacy network via a dual homing at ISA Level 2, then this would be the best solution without more details.
Q: When doing a SCADA system installation using an Ethernet-type network connection, are there differences in an intranet- or Internet-type installation and setup? What are the advantages or disadvantages of each?
A: An intranet installation would be done on a server over the internal network with the installation media actually in the server itself. Internet installation would be done between a computer on an intranet and a device at a remote location connecting to the network via the Internet through a firewall. This could also mean that the installation is done on a cloud server on the Internet.
I am going to presume that you are asking about an intranet installation and a cloud Installation. The advantage of an intranet installation is that you are responsible for the hardware and applications being installed on the hardware. You are also responsible for the security and the patching levels of the devices. This may not sound like an advantage, but some applications are not designed to be installed on virtual or hardware that is not supported. Certain drivers are required to allow alarming with network failures. These drivers may not work on cloud servers.
The benefit of using cloud servers for applications that can run on the cloud is that you never need to worry about hardware upgrades, and if done correctly in a "high availability" or "fault tolerant" mode, you should never lose connection to the servers.
Q: What are the advantages of using SCADA systems, and what is the best type of physical hardware to use for maximum redundancy (minimizes downtime) in the event of power interruption or natural disaster?
A: The advantages of using SCADA can be found around the Internet, but in short, in most systems, it allows the monitoring, control-system administration, data collection, and historization to be done from a central location. When this is done from two separated locations in a fault-tolerant and high- availability design, a disaster could happen and could be controlled from a separate building (sometimes called a war room or standby control room) while the main control building is being evacuated.
Q: Besides security, what other advantages does Ethernet provide over intranet-type hardware?
A: Intranet is just a term for a type of network architecture and whether it crosses communication boundaries between business zones, such as intranet, extranet, and Internet. Ethernet devices are used in all of these designs.
Q: What are the advantages and advancements, current and future, that we should know about in the Ethernet hardware?
A: Ethernet hardware runs all infrastructures from remote and local closed networks to cloud systems, so it is important to understand the different types of Ethernet devices for the business zones in which you will work, such as enterprise, manufacturing, production, control, and automation. There are devices like switches that are used in all of the zones mentioned, but know which switch to use for its zone application is important. Because the industrial sector is normally 10 years behind the IT/IS sector, Ethernet devices that have trended and proven to be successful over the past 10 years slowly are getting introduced into new designs being deployed today. Often systems designed and applied in the industrial market are done in 15-year lifecycles. The technology must be proven, reliable, and maintainable for this period.
Q: What do I need to know about Ethernet hardware as a data center engineer / operator engineer HVAC?
A: Industrial and enterprise hardware are interchangeable, but you need to ensure you have the right devices for the right environment. Enterprise standard devices are not very useful when being deployed in an IP67 or NEMA 4a enclosure without vents in extreme cold or heat. It also is not a good idea to deploy industrial devices that are DIN-rail mounted, 24 V powered devices in a data center where 120 V uninterruptible power supply (UPS) use is the norm.
Q: An RJ45 connector is not very industrialized. We use pre-connectorized industrial Ethernet cables with M-12 connectors. Aren’t field installed RJ-45 connectors a significant source of problems?
A: I recommend that all field standard devices follow one of the following standards, depending on the application:
- IEC 61076-2-100 recommended connector (M12-4 D-coding connector)
- IEC 61076-3-106 (LC Connector) ANSI/TIA/EIA-604-10A (Fiber)
Also see the following resources for more information:
- TIA-TR-42.9 Industrial Cabling Working Group
- ANSI/TIA-1005 Industrial Cabling Standard.
Q: How do you get the simple network management protocol (SNMP) data collected by network management software to the plant floor human machine interface (HMI) displays that maintenance personnel use?
A: This all depends on how the network architecture has been made. If it is an ISA 95-based architecture, you may find that you have remote locations that have Level 3 administration workstations in maintenance offices on the workshop floor. You may also have Level 2 engineering workstations that maintenance teams use. Answers differ depending on the network design and the policies and procedures for allowing non-operators or control engineers to gain access. [Increasingly, cybersecurity measures include role-based access to various levels.]
Q: What about the selection of an Ethernet backbone?
A: The backbone can be anything. If it is SCADA and not control or safety related, then wireless may be used, but it is normally Fiber (multimode or single mode) in a redundant routed cables.
Q: It is possible to share a site acceptance test (SAT) document?
A: No. This document is developed for and specific to each customer’s application.
Q: Infrastructure devices seem to be the most important ones from a selection standpoint. I have found that you can’t mix and match between vendors because their features generally aren’t interoperable. Can you really expect to use a multi-vendor infrastructure? (The most we have gotten away with is one industrial switch supplier that interfaces to the IT equipment.) If you use one industrial Ethernet switch supplier, then you better choose the right one.
A: I totally agree that all networks should be sourced by one vendor, but it is not always possible. Engineering, procurement, and construction (EPCs) providers; industrial automation (IA) firms; and main automation contractors (MACs) may all have their own standards of devices and will not listen to anyone on what they will supply. This is why it is important right at the very beginning of a project to get written into any agreement or contract what the devices are and who the suppliers should be.
Q: Please discuss pros and cons with using more than one manufacturer for subnets or devices.
- Some hardware programmable logic controllers (PLCs) and remote terminal units (RTUs) only allow configuration of their IP devices but not the subnets.
- Some devices only have fixed fiber ports and do not support single mode fiber.
- Some devices only have a very limited management console or functionality.
The ability to make a cheaper network.
Q: What are Power over Ethernet and Power over Ethernet Plus, what’s the difference, and how can they be used?
A: Power over Ethernet allows devices to be powered via the Ethernet cables that are connected to them. This is the IEEE 802.3af standard. Normally, a PoE switch or PoE power device would be located in a cabinet or an enclosure. One end of the Cat5 / Cat6 cable is connected to the POE switch or power device, and the other end of the cable would connect either into the end devices that would accept PoE or into another power device that extracts the power out of the cable and puts it into a jack plug that would go into the normal power socket of the end device. PoE-Plus is the IEEE 802.3at standard. The basic difference is that PoE is 15.4 W/port and PoE-Plus is 30 W/port.
Q: Please discuss routers versus Layer 3 switches.
A: In today’s networks there is not much of a difference between the two for the amount of traffic that is being transported around the infrastructure. Routers are great to use for just routing where you don’t have a change of placing in a device on the port and it being accepted into the network as a device. With high-end routers there are more functionality like virtual private network (VPN) and some routing protocols. However, in an industrial network, Layer 3 switches are a great lower-cost alternative. I would never use a Layer 3 switch to be a boundary router.
Q: Are a few parts of an Ethernet project usually overlooked?
A: I would look into:
- GPS NTP Time Servers with multiport that can service multiple subnets without routing of the protocol.
- Control firewalls that are specifically designed to protect the PLC, RTU, and controllers.
- NTP digital Ethernet clocks in the control and operating rooms give the times from the ICS network. This helps when there are incidents; people can look at and record time stamps accurate to the SCADA or control network.
- Industrial DIN-rail mounted 24 V firewalls (not control firewalls). If possible, keep all fiber the same (single mode or multimode, but not both).
- Stay away from OM3 fiber as it is not required for control and SCADA Networks. (Only use OM3 when connecting blade servers and storage devices.)
– Edited by Mark T. Hoske, content manager, Control Engineering, firstname.lastname@example.org.