Five-phase approach to improving operational technology PCs

Digital attacks targeting industrial control systems (ICSs) and operational technology (OT) increased by more than 2,000 percent from 2018 to 2020. This huge increase in attacks made manufacturing the second most targeted industry for cyberattacks in 2020 – up from eighth in 2019 and ranking behind only the insurance and finance industry. This same report also found that 35 percent of cyberattacks are leveraging existing vulnerabilities in systems, making scan-and-exploit the number one source of attacks over phishing for the first time.

As a manufacturer, this report may seem shocking and frightening, which means it is probably time to ask a few questions. How  long have the OT PCs been in use on the plant floor? When those PCs were deployed, were these six strategies implemented to protect your assets? When was the last time your organization evaluated the OT PCs for potential vulnerabilities?

If the answer to any of these questions is, “I have no idea” or is measured in years, you are not alone. Many OT PCs are rarely hardened at installation, which means over time initial vulnerabilities will only grow. We also see that many organizations put a lot of effort and money into establishing a strategy for network cybersecurity, especially for the enterprise, yet OT PCs are frequently not part of this plan because they are usually thought of as being air gapped or removed from the network in some way. But this is often not the case.

OT PCs that are not monitored for weaknesses and tested against emerging threats will have vulnerabilities that could be exploited by attackers. To avoid this as part of a defense-in-depth cybersecurity strategy, consider implementing the following five-phase approach to retroactively harden OT PCs.

Phase 1: Aligning key stakeholders and identifying requirements

Like many improvement-focused projects, success starts by holding productive conversations with key stakeholders. When it comes to OT assets, these stakeholders should include members of plant operations, key OT engineering staff, and potentially a representative from IT depending on how the organization’s cybersecurity is maintained. The two main goals of these initial discussions should be to assess the organization’s cyber posture in relation to OT PCs and to understand the software and services operators require on each PC for successful operations.

From these discussions, users should be able to remove assumptions about the way OT PCs operate and eliminate the need for implementing vague or overly restrictive security measures. For example, locking down all network ports on a PC adds a big layer of security, but this approach may impact an operator’s ability to do their job if they can no longer connect to other systems that are crucial to operations.

Additionally, this group also needs to identify any regulatory requirements or security standards industry or application should adhere to. This could include industry-wide requirements such as Chemical Facility Anti-Terrorism Standards (CFATS) for the chemical industry or the requirements set forth in the North American Electric Reliability Corporation Critical Infrastructure Protection (NERC CIP) plan. Or this may involve determining standards YOUR organization needs to, or really should consider, complying with such as those set by the National Institute of Standards and Technology 800 series (specifically the 800-82 overlay, which is a guide to ICS security) or the US Department of Defense Security Technical Implementation Guidelines (STIGs). A skilled systems integrator can help review best practices for meeting these types of requirements and standards and create the most appropriate configuration to use on your PCs.

Once these discussions are complete, this is a good time to perform a risk analysis on the OT PCs to ensure the most important gaps are identified. This can be done internally or by partnering with a third-party expert and can use an internal standardized cybersecurity program or an external framework such as the NIST Cybersecurity Framework (CSF).

Phase 2: Performing a thorough investigation

The goal of the investigation phase is to identify everything that is running on or has access to the OT PCs and then determine the operational purpose of that software or access. Just like the first phase, this approach requires a lot of discussion and open communication with operators to be successful. This is because it is not always clear what is running on every OT PC and what the function of each service or piece of software truly is. And sometimes, even if the purpose is understood, the software or service may be functioning in a manner that is different than what was previously assumed.

A well-documented holistic evaluation also helps later phases run more smoothly since we are trying to convert as many “unknown, unknowns” into “knowns” or “known, unknowns.” It probably goes without saying, but there is not a one-size-fits-all approach to this evaluation process as each plant is unique and what is acceptable for one plant is not always acceptable to another.

During the investigation phase we often reveal hidden dependencies and vulnerabilities due to unnecessary program operations that no one knew about.

Phase 3: Planning, executing, and documenting procedures

After the investigation phase, the next step is to combine what is now known about existing operations with the identified requirements from phase 1 to determine where changes are needed. From here, a prioritized remediation plan that closes the identified discrepancies and gaps needs to be generated and agreed upon by all stakeholders. To assemble this plan, it is best to supplement the cybersecurity knowledge of the IT department by bringing in an organization such as ACE that has expert knowledge of OT components.

We draw on our many years’ experience working with manufacturers to harden new and existing OT assets to develop, plan, and coordinate the best option for the organization that also minimizes or eliminates downtime when implementing the necessary changes. We also are familiar with industry and vendor best practices to ensure the identified requirements and standards decided on in phase 1 are followed. We help simplify the execution of any required changes by standardizing our recommended upgrades and changes for as much as the site will allow across all assets. These recommendations can be used to develop hardening guidelines to use with future system implementations and test procedures.

We know the logistics of implementing changes, especially when something appears to be functioning correctly on the surface, is frequently a sticking point for manufacturers since it is unlikely downtime can be scheduled just for OT PC upgrades. We try to design our hardening project plans so that a full outage is not needed for most of the required changes. We do this by using the upgrade procedures outlined by many software vendors or by implementing and testing required changes on a testbed first if one is in place. If an outage is necessary, we recommend coordinating outage times with an existing shutdown for a process such as routine maintenance.

Phase 4: Performing post-implementation testing

While easy to overlook, post-implementation testing is a critical phase of this approach that helps ensure changes made to existing OT PCs do not have unknown consequences. While every effort is made in the first three phases to turn all unknowns into knowns or “known unknowns,” there still can be unintended consequences when changes are made in the field. This is because OT PCs in many plants have morphed over time as one-off changes and upgrades were made as necessary. A well-defined testing and acceptance document helps identify any complications caused by field conditions so they can be corrected during implementation.

It is also crucial to have the key group of stakeholders identified in phase 1 involved in this phase, as well. These stakeholders can help quickly identify unintended operational impacts since they are most familiar with the functionality of the systems. Then, once one successful round of changes and testing is executed, the execution plan can be adjusted to account for new information exposed during the initial implementation. A project plan that alternates between execution and testing phases limits operational risk and improves the final configuration.

Phase 5: Reevaluate OT PCs regularly

It is not ideal to only think about vulnerabilities in OT PCs every five to ten years. That is why the documentation that comes out of this approach to retroactively harden OT PCs should also include a plan for ongoing maintenance. Examples of information that should be documented in the reevaluation plan include software and hardware lifecycle management, procedures for updating and patching software, and processes for evaluating the addition of any new software or services to an OT PC. Having these procedures in place will help eliminate possible vulnerabilities cyber attackers can exploit.

– This originally appeared on Applied Control Engineering’s website. Applied Control Engineering is a CFE Media and Technology content partner. Edited by Chris Vavra, web content manager, Control Engineering, CFE Media and Technology, cvavra@cfemedia.com.

Do you have experience and expertise with the topics mentioned in this content? You should consider contributing to our CFE Media editorial team and getting the recognition you and your company deserve. Click here to start this process.