IT vs. OT: Bridging the divide
You’re a networking person who works in your plant in operations technology (OT), supporting the technology that keeps manufacturing going. An e-mail arrives with a message that strikes terror: Your corporate IT department has been assigned the task of updating networks and implementing new cyber security measures in the plant, and you are to cooperate. In other words, IT is moving into the plant. Is this necessarily bad news? It probably isn’t good news, but the question is, why does the thought of combining IT and OT normally draw strong reactions?
“When you take people with an IT background and bring them into an industrial control system environment, there’s a lack of understanding from operations why they’re there and there is a lack of understanding of the specific controls environment needs from IT,” says Tim Conway, technical director, ICS and SCADA for the SANS Institute. He points out that typically IT professionals are trained and driven to perform a task: “They work on a box, a VM (virtual machine), a storage area network, or a firewall. They don’t realize that they’re a part of a larger control system operation, and how the things that they do can impact others.”
Conway’s experience came from many years working in networking engineering and management at a mid-sized electric utility. He’s seen how difficult it can be to develop IT personnel to realize the larger context: “If they’re network guys, they see how a change affects their networks and the inter-dependent IT system functions, meaning active directory or workstation authentication, or monitoring and alerting, and all the other IT functions. But they don’t think systemically from an operations perspective. For example, the impact out to the breaker in the substation if the communication path is lost. I compare the development challenge to what we do with our safety programs where we ask people to think about safety from the perspective of their work product. They have to think about how their actions can impact their own safety, impact the safety of the equipment and operation, and the safety of others. We ask them to all walk through the process and say, ‘Here’s what I work on, and here’s how it can impact the safety of the people in the field.’ The same applies to networks they support and the control systems that rely upon them.”
Needs of industrial networks
Younger IT people likely find a walk through a manufacturing facility to be like a trip to a museum. Engineers used to working with the latest technologies probably find most of the equipment running a process unit quaint, but they have to understand that industrial users are seldom impressed with the newest technologies since a technology is only a means to an end. If it works, who cares how old it is?
“That’s one of the biggest issues,” says Perry Tobin, senior consultant for Matrix Technologies. “IT people are typically young and don’t have 5 or 10 years working in a manufacturing environment and understanding the legacy issues. The IT person comes down and sees Windows 2000 machines that are deployed and will be there for two or three more years, and says, ‘Oh my, we need to get rid of that.’ But you say, ‘No, you can’t just change that machine out. There’s licensing, there are issues with Rockwell, Siemens, and some of the older software that won’t run on a new platform.’
“They’re all about upgrading, bigger, faster. IT people are not impressed with longevity. They’re appalled at how long it’s been static. It hasn’t had an upgrade, it hasn’t had updated firmware. They don’t realize that if something has been running without a reboot for seven years, don’t touch it. OT people tend to be in the same position longer.”
IT people also find themselves largely stripped of their skills and tools when they move into the plant. The techniques that they use routinely to solve problems and secure communication may simply not be available. Conway explains, “IT security people who look at a traditional plant control system, would want to engage a standard security package; switchport security, intrusion detection on the backplane of the VLANs, and SNMP rollups, for example. In many cases, the system vendors would simply say ‘You can’t do it. These switches have custom code and are built for a certain scan rate, certain throughput, and if you screw with that, we can’t ensure the availability and integrity of the controller talking through the switch to the workstation.’ This is a challenging response to IT security personnel who want to provide security defenses, but it needs to be understood and evaluated because a secure system that does not perform its functions as engineered or perform them safely, would not be desirable for anyone. There are approaches working with all stakeholders to achieve a balance.
Dealing with the unknown
When IT people have to take on a problem-solving task in the plant, they often discover many kinds of devices and communication approaches that are much different than they’re used to. Hunting for creative solutions can go in new directions if an engineer has to work with manufacturing to find ways to communicate with a system or piece of equipment to collect performance data. Kevin Price, senior product manager of Infor EAM, has seen many situations where a reliability engineer has to work with IT to extract data from an individual machine or system for performance analysis. As he describes the situation, “The reliability engineers are trying to reach a specific OEE (overall equipment effectiveness) rating. In order to do that, they need to understand how the asset is running from a quality perspective and an availability perspective.
“To do that, they need to be able to monitor it. To do that, they need a meter that can talk to that piece of equipment, whether analog, digital, or a system. All these tell, in real time or batch, the health of that asset. You have to work with IT in order to do those integrations and pull it to a system like ours. Our connection to the system, from an IT perspective, is at that integration layer. Now that we’re moving from analog to digital with some of these controls and systems, it’s becoming more open and the data more readily available. It’s more accessible to the average IT resource. But if you look at some of the systems that were installed in the 1990s, they’re proprietary, they’re analog, they’ve never been rebooted, and they’re running like a champ. The problem is the IT person can’t get any data out of it. So the reliability engineer gets frustrated because he can’t understand how that equipment could be improved because nobody knows how to talk to it.”
Developing an inferiority complex
In most situations, OT is in a weaker position in the corporate pecking order since there are typically fewer of them and they are more isolated at the device level end of the systems. Corporate IT people are better organized and connected. The corporate culture can leave OT feeling like a second banana and forced to do what those up the chain dictate.
Tobin says it doesn’t have to be that way. He suggests, “When everybody gets together and thinks long term, it definitely builds a much better relationship than if somebody says, ‘We’ve been tasked with putting a new network in the plant over the next six months, and here’s what you’re going to get.’ It’s the knowledge of OT understanding more what IT wants to do, it’s the understanding of IT knowing what OT needs, and somebody to coordinate that. There’s an education side to it. Companies that are willing to invest the time and money to bring people together to get that dialog going are the ones that are successful and don’t have a lot of animosity between the two. The right technology has to be there and it’s going to change, but the corporate culture and the communication between IT and OT are the key things to making any success between the two.”
Twilight of OT?
Individuals working in OT also need a different kind of motivation or they may feel they are stuck in a career dead end. IT skills can be applied in all sorts of industries so an engineer may move from banking to retailing with relative ease. Such is not the case for people coming from OT. “If engineers spend time learning the operations of a particular company or industrial sector, they might be making themselves better in their current job, but not making themselves more marketable or competitive outside of a particular industry,” Conway says. “It’s almost a negative incentive. The better they are in their current jobs in a specific sector, the more responsibility they’re going to get, the more demands that can be put on them in that role, but it’s so specific and such a niche, that it doesn’t help them if they want to look at a job in the banking industry for example. It is important to ensure personnel development goals are aligned and incentives are in place to train personnel for a role and retain those individuals.”
The idea of IT moving into the plant may also be a defensive one that is driven more by necessity than any particular strategic objective. “It is actually easier to train the OT expert and controls engineers on IT and IP-based technology and management of those assets than vice versa,” says Chet Namboodri, managing director of global manufacturing industry sales for Cisco. “The issue is that the controls engineering breed is a bit of a dying one. There’s less of that experience resource available now throughout the world, both in developed economies and more broadly. That said, there are examples of organizations that have been successful in following the convergence approach, and leveraging what they want out of a converged network. The real transformational business value is by using that inherent integration to get at use cases that drive efficiencies or even outside revenues.
The positive side of convergence
While the discussion so far may seem somewhat negative, as Namboodri observes, there are many positive aspects of IT moving more into a manufacturing environment beyond simply filling a staffing gap. “There is value in converging IT and OT from a networking standpoint,” he suggests. “Both IT and OT have a very strong role to play in that integration and the subsequent management of those networks. To highlight one as more important than the other is a disservice to both. There are some technical concerns over things like determinism in motion control, but those are resolving themselves. Those are less of a problem with technology and network convergence than what is really at the heart of making a successful transition into IP-based industrial networks, which is the cultural convergence that needs to take place.
“A number of companies have even gone the next step and organizationally converged manufacturing IT and controls engineering functions under one roof. They recognize that this network, even at the device level, enables visualization with remote access. It enables collaboration between their production experts and operations personnel, maintenance personnel, technicians, and so on, that are on-site through a secure architecture. That’s what they’re starting to leverage systematically. Instead of sending their experts to another plant on a plane to troubleshoot a problem, they’re able to do that with the visualization since a lot of that is seeing what’s happening. They also get data and diagnostic feeds of what’s going on so they can troubleshoot with the people on the plant floor to get an operation back up and running much faster. The trust has been built around the value of IT.”
Both can have seats at the table
Time is not necessarily on the side of traditional OT functions, but given the slow pace of technology change in most manufacturing environments, the devices and networking techniques unique to process plants and manufacturing floors will be around for a long time. Nonetheless, more and more of those networking functions are going to be replaced by IP-based technologies. For the foreseeable future, both will be necessary, although it may be harder to figure out where the fences are.
“The IT/OT line is blurring,” notes Jason Montroy, client relationship manager for Maverick Technologies. “You need engineers that know about both sides. It’s going to remain compartmentalized, at least for a while, to where you’re going to have an IT resource that knows a little bit about automation, and you’ll have an automation resource that knows a little about IT. Those engineers that have skills outside their core competencies will be very highly valued. One positive thing that we’re seeing right now, as we go into opportunities that involve DCS migrations, you’re getting an IT person at the table early in the project. This is very important, because IT is providing the ecosystem that the control system will reside in. Bringing them into the game early on is very beneficial.”
The challenge going forward will be to get the right people into the right positions, even when highly qualified individuals will be harder to find. Companies will have to be more creative, and that in itself could be the biggest challenge to established practices and existing company cultures. IT is coming, and when it’s all done, your networks and information management may be vastly different than they are today.
Peter Welander is a content manager for Control Engineering, firstname.lastname@example.org
- IT is pushing into more areas that were traditionally separated from OT.
- Expansion of IP-based technologies will displace more industrial protocols.
- Convergence of IT and OT is growing, and moving into new environments.