IT/OT convergence advice, best practices for manufacturers

Suzanne Gill of Control Engineering Europe spoke with a few thought leaders on the cybersecurity challenges manufacturers face and how they can best overcome them. Their suggestions and tips follow.

Biggest security risks and challenges of information technology/operational technology (IT/OT) convergence

Prof. Dr. Tobias Heer, senior architect at Belden, believes the biggest issue is the heterogeneity and complexity created by mixing IT and OT systems, resulting in systems that are hard to understand from a security perspective. The result, he said, is a lack of control over what’s happening in the network. Especially when less restrictive networks grow over the years it becomes difficult to know what’s going on. Getting rid of the heterogeneity is not always possible since we have vastly different device classes (control systems on the one hand and IT servers on the other hand). So, making sure you stay on top of what’s happening is key.

Heer said recent ransomware attacks have often used the fact that communication was too liberal in industrial networks and known vulnerabilities existed in the connected systems. That’s exactly the result of an open and possibly unknown network structure. The result of complex and unorganized networks can get worse if multiple groups manage the network and can attach devices to it. Of course, work gets done if everything has a network that works.

However, critical security controls are hard to implement in such a situation. On the other hand, starting from the point of knowing what’s happening in the network enables one to implement a whole chain of security controls. Asset management enables you to find vulnerabilities and outdated software versions. Knowing the software in use enables you to implement patch management to always have your devices up to date. Knowing all devices and their peers helps you to segment the network into different zones to restrict attacker communication. Knowing the permitted communication patterns allows you to detect deviations and to ultimately identify attacker behavior. It all starts with a network in which you have full visibility and control. Then suddenly other core tasks become manageable.

Andrew Bullock, industrial automation and network cyber security specialist at Rockwell Automation, believes one of the biggest security risks and challenges of IT/OT convergence is poor leadership. It is not having a defined hierarchy of responsibility for the security lifecycle process. Bullock pointed out where there is no security responsibility, people with the good intention of improving operational visibility, and insight and hence performance and operating efficiency allow the proliferation of unmanaged devices on to the network, particularly open IoT. However, by doing so, they increase the attack surface by increasing the number of unprotected entry points to the network.

Creating a structured hierarchy of security responsibilities enables organizations to establish a culture of good security practices and confirm that appropriate security procedures and technologies are used. Leaders must actively get involved and sponsor the development of programs and working groups as diverse as, the education of an IT professional, or IT department, to help them understand the differing priorities between IT and OT networks, or an engineering manager to understand the risks associated with unmonitored and unsecure external remote connectivity to allow “ease of modification,” to operator awareness of unexpected behavior of an operating screen or machine.

Aengus Gorey, a security systems engineer at Analog Devices, said OT asset management is a critical challenge to a converged network. He said an OT network typically does not have a live device map with rich profile data, which allows the devices access to the network to be controlled & managed. In a converged network, such a feature allows rogue devices to attach to the network and interact more easily inside the network, creating security holes for observation or malicious actions on the network.

How security concepts are applied and managed in the converged network become increasingly important. Equipment manufacture, commissioning and servicing processes need to be designed with a security mindset. Without a security lifecycle process, it will expose devices in an OT network to threats created by poor security hygiene.

Gorey believes, as manufacturing move towards an Industry 5.0 model with greater human/machine dynamic, the interfaces to an OT network will look much more like an IT network than previously and consequently this may expose OT networks to a greater degree of social engineering threats.

Phillip J. Corner, industrial cybersecurity engineer at Thales UK, said interconnection of process systems and business  systems can offer benefits to enterprises not only operationally, but also for cyber resilience. Endpoint protection, patch management, monitoring systems, backup and recovery, and other key components underpinning cyber resilience are likely to be more advanced in business systems and can be efficiently expanded to serve process security needs.

Additionally, the majority of current state-of-the-art solutions have a vendor-hosted cloud component for configuration, management, and monitoring. However, Corner said the benefits of connectivity compared with isolated systems must be carefully planned and managed throughout the lifecycle of the system and its component assets to balance the risk posed by that same connectivity lest it also present a significant cyber security risk. Particularly in existing systems with legacy assets which may not have robust built-in security.

Interconnection may also be to the public Internet providing external access to ICS assets. The recent pandemic saw a sharp rise in remote access technologies being rapidly implemented for both business and process systems. Without continual management, boundary systems like these become insecure and can essentially leave the front door open to attackers.

Studies of vulnerability disclosures in several popular remote access products revealed a worrying number of vulnerable systems publicly detectable on the Internet, sometimes years after the vendor had released updates indicating that some organizations are failing to understand and address even basic best practice in their high-risk assets.

Advice on securing converged IT/OT networks

Heer pointed out converged IT/OT networks are often a mix of different systems with different communication protocols and software. Attackers can choose the weakest link in the chain and pick the most vulnerable devices if their communication is not restricted. Hence, converged networks require a restrictive planning of communication capabilities and access. Having just one big connected component almost guarantees security problems. Introducing additional zones and firewall rules between these is key to creating a resilient industrial network.

By segmenting a network into different zones, Heer said the possible communication peers of a compromised device are limited. Having tightly controlled passages between the zones, for example by using restrictive firewall rules, also helps to block malware from spilling over from the more IT-friendly and open parts of the network.

Keeping the attacker away from the vulnerable parts of the plant is much easier if there is a good zones-and-conduit concept. This has been the most important countermeasure against the current rampage of ransomware that caused havoc at numerous industrial sites. Keeping devices apart that don’t need to communicate means keeping existing known or unknown vulnerabilities inaccessible from each other.

This concept of zones-and-conduits is so important it has become a core part of ISO/IEC 62443 which provides guidance to operators, integrators and vendors on how to shape a secure industrial site. Heer’s advice is to embrace this standard. If you don’t intend to use or implement it, then at least implement the concept of zones and conduit by segmenting your network into different independent zones to block or slow down attackers that get an initial foothold in the network.

Bullock said industrial security must be holistic. It should extend from the enterprise through the plant level and even out to end devices, and address risks across people, processes, and technologies. It should involve collaboration between IT and OT personnel. Both sides have vital roles to play in establishing a secure network architecture. He said a fully-connected enterprise requires a comprehensive approach to security.

A complex, interconnected system includes challenges. It is critical to understand the potential risks and start building security into industrial automation control systems. However, before any security can be defined, it is important to establish responsibility. Knowing who has responsibility is key to starting the process of securing converged IT/OT networks. Whether that be individuals, job roles or departments, having that “chain of command” in place means the overall security program will be correctly structured and defined.

Bullock said developing and implementing an effective industrial security program first requires an understanding of the risks and areas of vulnerability that exist within the organization. A security assessment’s deliverables should include at a minimum: An inventory of authorized and unauthorized devices and software; Detailed observation and documentation of system performance; Identification of tolerance thresholds and risk/vulnerability indications; Prioritization of each vulnerability, based on impact and exploitation potential.

Corner said when interconnecting a system, it is crucial operators ensure they have a suitably detailed and clearly documented understanding of that system, its functional requirements, components, and their behavior. Some vendors offer products to assist in this process which can be helpful at a large scale, but its equally practical for experienced persons to conduct a deep dive using common free tools to establish an accurate record of the physical and logical topology of the system, its configuration, and “as fitted” behavior from a network communications perspective.

It is important to survey physical and logical aspects, this forms a basis for identifying and addressing cyber risk and often also identifies potential operational resilience deficiencies, this can include damaged communications cables, failed power supplies, lack of configuration backups, and incorrect configurations.

It is possible to create a baseline of the required core functionality and then implement controls that restrict access to the minimum required, this can includes segmenting the system to control the logical flow of data (an IEC 62443 aligned zoning approach) and also implementing robust identification and authentication of individuals, allocating the least privilege possible. For external facing systems like remote access, multi-factor authentication is an absolute must.

OT security best practices for control engineers

There is a plethora of security best practices to follow, Heer said. They can be best categorized as IT-like or OT-like security best practices. Overall, this is a set of important measures to keep everything clean, simple and to avoid complexity. This must be done regardless of having a converged IT/OT network or not.

The technological push and innovation in security is coming from the IT side, though, Heer said. In recent years, there have been more successful applications of security information event management (SIEM) systems, log management solutions, network access control and intrusion detection systems to identify attackers and efficiently manage the flood of available data that could or could not hint at the existence of an attacker. These technologies have benefits and disadvantages. As a benefit, many systems make managing security more efficient.

However, they are often complex and require special skills. So, choosing the tools carefully is important to avoid headaches.

Making sure security can be checked and improved is probably even more important than any of the technical systems and this starts with someone who is responsible for taking care of security. Assign someone the task and make sure they have the time and the knowledge to really do it.

Next, Heer recommends taking one of the relevant security standards appropriate to the sector. A good standard provides enough abstract advice on security best practices so you can get the actual work going. Get accustomed to what’s required to fulfill it even if you don’t plan to implement it.

Finally, he said, decide what makes sense. Focus on responsibilities and process first, instead of prioritizing a technical solution. When starting out, working with processes and with humans provides the best leverage. Also, be sure to formalize and document these processes. The more precise you can describe what’s intended to happen in a plant in policies and procedures, the better you can defend the site or detect dangerous misbehavior.

According to Bullock, industrial security is best implemented as a complete system across all operations. Defense-in-depth (DiD) security supports this approach. Based on the notion that any one point of protection can and likely will be defeated, DiD security establishes multiple layers of protection through a combination of physical, electronic and procedural safeguards.

A DiD security approach should consist of six main components – policies and procedures, physical, network, computer, application and device. Policies and procedures play a critical role in shaping cultural behaviors to follow good security practices and confirming the appropriate security technologies are used.

Physical security should limit personnel access to not only areas of a facility but also to entry points on the physical network infrastructure, such as control panels, cabling, and devices. At the facility level, access control technology such as networked key cards can help restrict access to the plant floor, control rooms and other areas to authorized personnel only.

A network security framework should be established to help safeguard the network infrastructure against cyberattacks. This requires close cooperation between IT and OT, including a robust discussion about the technologies and policies needed to best protect the assets and ability to innovate. One of the technologies discussed should be an industrial demilitarized zone (IDMZ), which creates a critical barrier of protection between the enterprise and industrial zones. Segmenting areas of the plant floor into virtual local area networks (VLANs) is a good security practice at the network level. Firewalls with intrusion detection and prevention systems (IDS/IPS) should be deployed within and around the industrial network to manage and limit network traffic.

Security patch management should be established to track, evaluate, test, and install cybersecurity software patches. Authentication, authorization, and accounting (AAA) software can restrict and monitor application access and changes. Antivirus software, application whitelisting and host intrusion-detection systems can further harden computer assets.

Security devices should also be incorporated at the manufacturing or industrial application level as part of a DiD approach. A role-based access control system can restrict access to critical process functions or require operators to enter log-in information before they access applications.

Device authentication and unauthorized device identification can help make sure only trusted devices are used. Additionally, changing the out-of-the-box default configurations for embedded devices can help make them more secure in areas such as restrictive access and change management.

Corner’s advice is to look closely at maximizing the use of built in security features of components before augmenting them with additional controls to provide DiD. This requires an appreciation that commodity components are likely to exhibit insecure characteristics by default. Integrators and operators are increasingly looking to accredited vendors showing a ‘secure by design’ principle, but do not necessarily appreciate the limitations. Accreditations are often achieved by a product disabling all features by default. This doesn’t mean insecure features cannot be configured, so it’s vital to ensure these products are also used in a way that is “secure by integration” and “secure by operation.”

There needs to be a commitment to build and maintain a deep understanding of the operation of components in use and their technical principles, achieved through training and experience.

Organizations should maximize the use of specialists from both business and process teams where possible. Their ICA engineers will have an innate understanding of the process requirements but even at their most capable are unlikely to be as experienced in communications networks or servers as their IT counterparts. Fostering cooperation and collaboration at all levels is crucial. Bilateral peer mentoring is an effective approach to achieve though pairing up enthusiastic individuals from different specialisms to share their knowledge and experience.

This article originally appeared on Control Engineering Europe’s website. Edited by Chris Vavra, web content manager, Control Engineering, CFE Media, cvavra@cfemedia.com.

Original content can be found at www.controlengeurope.com.

Do you have experience and expertise with the topics mentioned in this content? You should consider contributing to our CFE Media editorial team and getting the recognition you and your company deserve. Click here to start this process.