Operational technology cybersecurity: Easier said than done

How do engineers deal with construction challenges associated with implementing cybersecurity in operations technology? This article will discuss the lack of experience system integrators have with cybersecurity and the lack of experience cybersecurity implementation firms have with design/bid/build project delivery.

 

Learning Objectives

  • Understand the challenges system integrators encounter when implementing cybersecurity controls for process control systems.
  • Learn about the challenges cybersecurity providers have when delivering products in design/bid/build projects.
  • Learn about some ways to ensure the project cybersecurity requirements are met.

Cybersecurity insights

  • The implementation of cybersecurity in operational technology (OT) systems is crucial yet challenging, often hindered by unqualified integrators and vendors unfamiliar with construction project protocols, leading to suboptimal security measures.
  • Effective cybersecurity for OT systems requires collaboration between system integrators and cybersecurity providers, clear design documentation and proactive project management to ensure robust and reliable security configurations.

Cybersecurity is becoming one of the most discussed topics as it relates to operational technology (OT) systems used in utilities and manufacturing. The National Institute of Standards and Technology defines OT as “programmable systems or devices that interact with the physical environment,” which is comprehensive of most electronic devices deployed in industry today.

We cannot avoid the headlines of the next water treatment facility, pipeline, food processing plant or industrial manufacturer that has suffered yet another ransomware attack. Cybersecurity is one of the biggest gaps in existing infrastructure and are often included in many recent design projects.


Figure 1: A control systems engineer works on a human machine interface/supervisory control and data acquisition system server. Courtesy of CDM Smith.
Figure 1: A control systems engineer works on a human machine interface/supervisory control and data acquisition system server. Courtesy of CDM Smith.

Leaving cybersecurity vulnerabilities unaddressed could lead to significant issues that affect an organization’s reputation, profitability and safety. Equally troubling is when an unqualified design engineer includes broad requirements for “un-hackable” systems without any specific requirements.

While many understand they need to improve cybersecurity, many don’t understand what it means to implement comprehensive cybersecurity controls. For years, the thought process for systems such as supervisory control and data acquisition (SCADA) systems or programmable logic controllers (PLCs) was get them running quickly and keep them running as efficiently as possible.

Many times, OT systems have been supported by electricians or dedicated instrumentation and controls staff who have limited exposure to advanced networking and software components. Including cybersecurity as a capital improvement project designed by an engineering firm and installed by a contractor can be an efficient way to implement a comprehensive and secure system. However, what happens when the experts responsible for implementing cybersecurity controls for the project are not familiar with how construction projects are executed?

Figure 2: A firewall installed in a server rack is used to protect the supervisory control and data acquisition network. Courtesy of CDM Smith.
Figure 2: A firewall installed in a server rack is used to protect the supervisory control and data acquisition network. Courtesy of CDM Smith.

Design engineers learn quickly that if their documentation (drawings and specifications) is not clear, specific and detailed, the product their client will receive will be less than adequate. The expertise of the contractor often plays a role in the quality of the product, which can complicate things.

As I started implementing cybersecurity requirements in projects, I thought, “Just be very clear what the expectations are, utilize industry best standards, and enforce the execution, similar to how we enforce requirements for system integrators implementing traditional SCADA systems.”

While it might be easy to create requirements during design, when construction begins, the organizations responsible for providing the system are generally either:

  • Unqualified to execute the project’s requirements. This leads to a subpar system where foundational requirements are implemented poorly or not at all or,

  • Unexperienced with the design/bid/build process. This leads to challenges in a typical contract execution and can cause project delays and undesired changes to system requirements when installers default to their standard offerings.

Issues with system integrators unqualified in cybersecurity

Traditional system integrators are experts in constructing industrial enclosures, programming PLCs, and human-machine interface (HMI) graphic creation. Ethernet allows these systems to communicate and quickly replaced earlier technologies because of its ease of use and widespread adoption by equipment manufacturers. The ease of including more and more devices on an Ethernet network was appealing from a cost standpoint. As a result, integrators needed to become familiar with the basics of Ethernet networks, however, this is commonly where their experience in Ethernet ends.

Figure 3: Server racks with hypervisors are used to run virtual machines associated with operational technology systems. Courtesy of CDM Smith.
Figure 3: Server racks with hypervisors are used to run virtual machines associated with operational technology systems. Courtesy of CDM Smith.

While some integrators continue to develop their expertise, in many cases, systems integrators still deploy systems on “flat” networks with no or limited network segmentation. Any device on the network can see any other device, and if the user has physical access to the network, they have full control. Furthermore, HMI software packages are often installed on a Windows-based OS. In many cases, once the OS is installed, it is off to the races to get the HMI up and operational. Security is often an afterthought when something is not functioning correctly. Call any tech support, and they will first ask users to turn the firewall off. However, do they tell anyone to turn it back on when the issue is resolved? It’s a common oversight because in the past the focus has been on process control rather than protecting the equipment performing process control.

It is not realistic to assume integrators can deploy new security requirements without having a background in cybersecurity. While some integrators are savvy enough to figure out the implementation of cybersecurity controls, it is difficult for inexperienced staff to determine if they were implemented correctly. End users have to trust it has been done correctly or end up spending money to verify this themselves. It is hard to trust a company with little experience in the field. We want to be very confident security configurations are implemented correctly because the consequences can be devastating if they fail.

Issues with cybersecurity professionals inexperienced with project delivery

Many cybersecurity vendors have a business model that starts when an end user contacts the vendor to improve their system. The vendor will arrive at the project site, perform an initial assessment, educate the client on their recommendations, and implement their typical offerings. During a traditional design/bid/build project, the problem is much of the initial site survey, end user education and design have already been performed.

Figure 4: Network cabling associated with server equipment used for a supervisory control and data acquisition system. Courtesy of CDM Smith.
Figure 4: Network cabling associated with server equipment used for a supervisory control and data acquisition system. Courtesy of CDM Smith.

In a risk-based approach, the designer implements controls in a fashion that reflects the use of the system and the risk to the system with a focus on critical assets. Implementations differ from client to client, and standard or typical offerings from a security vendor may not apply in all cases.

Drawings and specifications are used to detail the scope of work that will be performed. However, many cybersecurity vendors are not well-versed in the design documentation that contractors and system integrators work with every day. The vendor may follow their typical project flow without realizing what their contract specifies they need to provide. To further compound the problem, many equipment manufacturers used to improve cybersecurity are not familiar with this type of project execution. For example, they offer little support on how to submit documentation for approval, which is typical during construction.

What to do moving forward

While system integrators become more proficient with cybersecurity, there are a few things cybersecurity providers can do to help. First, they need to have discussions with the companies that supply these types of services. Speak with the integrators you work with frequently, and let them know the concerns with cybersecurity requirements. Try to make new contacts with companies who specialize in cybersecurity implementations, and describe the types of projects they are responsible for designing so they better understand how to execute a project. It also is helpful to start introducing system integrators to cybersecurity providers.

In some cases, it can be helpful to discuss design information with cybersecurity companies. Ask them what they would have done differently in the design or if the requirements deviate from their standard offerings. Even if their comments do not align with the end user’s requirements, it helps to understand their perspective.

It often is in the best interest of a system integrator to hire a cybersecurity vendor as a subcontractor. This lets the system integrator to lead the effort, teaching the cybersecurity provider how to best execute the project requirements. While this might seem like an ideal solution, project construction rarely goes smoothly. Design engineers or construction managers also need to be diligent in ensuring the project execution is delivered. Making sure important coordination workshops, submittals and test procedures occur is imperative to project success. If these activities are not encouraged, they can be left to occur too late in the project to be effective.

While implementation is certainly a challenge, cybersecurity is particularly important to include in designs. Challenges that occur during construction should not cause designers to deviate from project requirements. Understanding the landscape can also help to shape projects in a more impactful way. With added diligence during design and construction to guide system providers, there is a path forward to improving cybersecurity in industrial control systems (ICSs).

Matthew J. Lick, PE, CISSP, is an automation engineer and operational technology cybersecurity discipline leader at CDM Smith. Edited by Chris Vavra, senior editor, Control Engineering, WTWH Media, [email protected].

MORE ANSWERS

Keywords: cybersecurity

CONSIDER THIS

What are you doing to meet cybersecurity requirements and overcome challenges?

Written by

Matthew J. Lick, PE, CISSP

Matthew J. Lick, PE, CISSP, is an automation engineer and operational technology cybersecurity discipline leader at CDM Smith