Software patching is vital to secure operations, but introduces more risks
It’s common to think of security updates as self-contained packages, as if the latest anti-virus or Microsoft Windows update was simply a new feature that gets added to the security stack, keeping trouble that much farther away. Yet, when it comes to patching cyber assets on industrial control systems (ICS), one needs to take a little more care than for an office or home PC.
When the office or home PC gets updated (automatically of course), it’s understood that there’s a possibility of unexpected consequences. Unless there is a major glitch like a lock-up, blue screen, or a primary application’s malfunction, the assumption is that everything will work out for the better. In the worst case, the PC gets a reboot, and the expectation is that the next set of updates will correct the inconvenience.
In an industrial plant setting this kind of thinking and lack of awareness begs for disaster. The continuity of operations is critical. Even a minor communication hiccup or loss of view can have undesired results such as interruption of operations, or even catastrophic damage to major equipment [which can increase risk for personnel as well as production].
Regularly applying tested and validated software patches helps maintain access to plant infrastructure and provides critical cyber protection and reliability for daily operations. When operators/owners take a do-it-yourself approach to patching, they often experience unanticipated challenges and risks because of the bandwidth and resources required to properly identify and test software updates before uploading them onto the cyber assets. Manufacturer-provided patching is an excellent starting point for operators to safely execute updates and maintain operational conditions in the plant.
Is the patch needed?
Do we really need this patch on the PCs?
Maybe! Software manufacturers continuously update, test, and retest their products to improve security and operational efficiency. Hackers continually attempt to find vulnerabilities. This combination leads to the release of updates more frequently than many operators would like to see. Yet, are all of the updates really needed by the plant? Just because a company like Microsoft, which has numerous users operating across a broad range of environments, says that a particular update is critical, it may not be the case for an individual plant’s operations. In fact, while some updates may be critical for millions of users, they may be irrelevant for many others. On the other hand, a critical and timely update, for an application such as .NET, could be overlooked by a plant operator due to the lack of knowledge of the internal software functions. This is why it is beneficial for plant operators to ask their equipment manufacturers for help to identify, test, and upload patches following a systematic process.
Assess patch risk: 11 critical questions
Assessing the relevance of a given patch can be a complex exercise. Knowledge base articles from software manufacturers that provide details on updates are generally comprehensive, and quite detailed. Questions to ask include:
- Are the operating systems it affects in use in your operation?
- If so, are the vulnerabilities it addresses active on your machines?
- What antivirus signature update may detect and delete a .DDL from my SCADA application?
- Is the system using SQL server or Internet Explorer?
- What about Java or Adobe?
- What other third-party applications are in use? (The list of third-party applications on many PCs can be longer than expected.)
- Will the update affect my firewall settings or host intrusion detection application (HIDS)? You may find that a patch labeled "critical" protects Windows machines using a DVD authoring app from a possible Trojan horse infection. If DVD authoring is not installed on your systems, then this is one you can live without.
- What are patches? Gather all patches for the computer operating system, the application, and other third-party applications.
- Which patches are critical? Figure out which ones are critical.
- How should the patches be tested? Determine how to test these patches. The cycle starts over every 30 days.
- What are the risks and priorities? For that critical patch from Microsoft, should it go into the standard cycle or should you just install it? The patches that pass the relevance test are the ones that will not cause any noticeable changes to the work environment and continue to provide additional protection against security threats. [What are the related operational security and safety risks?]
Troubleshoot control system interactions
The preferred way to validate patches is to run a set of controlled tests on a representative hardware/software platform. A maintenance system or simulator typically provides an environment where a bad patch result will not interrupt plant operations. Once the patch set has passed this series of tests, the manufacturer begins an incremental installation on the actual plant control systems. This can be a tall order with many different testing environments required, depending on the heterogeneity of the installed base cyber assets.
A secure lab environment with a variety of representative equipment, various operating systems, and typical configurations provides the ideal conditions for testing patches to ensure an error-free update. For most companies, the problem of comprehensive testing before installation is the most challenging step. Securely updating a plant’s software is time consuming and requires a significant level of continuous expertise. [subhead]
Selection, validation testing
Many operators are required to keep systems with the most current patches and updates by regulation or company policy. For others, it is an industry best practice that is highly recommend. A good process of gathering, selection, and validation testing should be used to avoid the nightmare scenarios and even minor disruptions to plant operations. Thoroughness is the key, and patching is an essential part of ongoing maintenance to keep plant assets reliable and safe.
– Mark Hammer is a product line manager at GE Measurement & Control, responsible for developing and creating implementation procedures for control system cyber security programs in the power generation and oil and gas industries. Edited by Mark T. Hoske, content manager, CFE Media, Control Engineering, firstname.lastname@example.org.
www.controleng.com/archives May, under this headline, find additional advice, links, and resources about the end of Microsoft Windows XP support.
Control Engineering has an online cyber security training series of videos.
- Company policies, regulations, and best practices can guide best practices.
- Gathering, selection, and validation testing should be used to lower risks
- Thorough patching process is an essential part of ongoing maintenance to keep plant assets reliable and safe.
Price of poor patching could include unplanned outages, risk to safety, or loss of critical company assets and information.
More about the author: Mark Hammer is a product line manager at GE Measurement & Control. He is responsible for developing and creating implementation procedures for control system cyber security programs within the power generation and oil and gas industries. He has more than 25 years of experience in the controls and automation industry with a number of leading automation and safety system vendors. He holds both a bachelor’s degree in mechanical engineering and master’s in business.
– See related articles below.