Microsoft patches: Know when it is better to wait
After yet another “Patch Tuesday,” when Microsoft releases monthly patches, and a challenging Microsoft Windows 8.1 upgrade, I was reminded that sometimes it’s much better to wait before installing patches. The upgraded system still does not have a working wireless interface, and the browsers are acting strangely. Fortunately, the upgraded system was not a mission critical Industrial Automation and Control System (IACS) but was installed on a test system following a best practice for patch management.
Patch management as a risk management process
Applying patches is a risk management issue, because patching a system means changing the system, and changes can negatively affect an IACS’s safety, operability, or reliability if not performed correctly. Bad patch management practices have resulted in plant shutdowns and significant financial losses, when the patches crashed the IACS. On the other hand, not installing patches has also resulted in plant shutdowns and significant financial losses when systems crashed due to worm, virus, and hacker attacks. Applying patches requires a balance between ensuring the patch does not negatively impact the IACS and the risk of leaving the system unpatched.
Preparing an IACS to be patched can require a tremendous amount of work. For each IACS device in operation, an asset owner must discover the available patches, determine if the patches are applicable to their versions of the products, install and verify the patches on a test system, ensure backups are created before and after patching the operational system, ensure testing before turning the system back over to operations, and finally documenting all changes made.
If the cost of a shutdown to apply patches is greater than the estimated risk, then patches are often delayed by the asset owners. The resources and efforts required to install patches mean that most organizations schedule IACS patches during normal routine maintenance outages. Sometimes these outage windows are monthly, quarterly, or even longer. Some extremely critical systems may have no allowed outage windows available, and can therefore not be patched regularly. For these critical systems, other security controls are usually put in place to mitigate the risk, such as severely limiting any network access to the system.
Preparing IACS patches also is costly and time consuming for product suppliers. They must discover what patches are available, determine which patches apply to which products, determine which supported versions of the products are affected, test the patches against each product version, and then distribute the patch information and testing results to their customers. Meanwhile their customers are often impatiently waiting or calling to try and get the status of the latest critical patch.
The best practices for installing and testing upgrades and patches for IACSs are being documented by the ISA 99 committee. The guidelines for IACS patch management will be issued as an ISA Technical Report (ISA-TR62443-2-3) and as a technical specification in the IEC 62443 series. The technical report is focused on security-related patches, but the patch management process that it defines also can be effectively applied to non-security-related patches and product updates.
The ISA 99 technical report defines the policies and procedures that asset owners should put in place to evaluate, test, and install patches. In the case of Patch Tuesday patches, it is better to wait for the product supplier to run its tests and approve the patches before installing them on operational systems.
The purpose of the technical report is to provide guidance to asset owners and product suppliers. Asset owners are defined as the owners and users of systems which can be patched, and product suppliers are suppliers of systems or components. The technical report is in draft form, undergoing final cleanup, with an expected release in late 2014. It provides specific details, based on industry best practices, that asset owners can use to establish and operate an IACS patch management program. It also describes the industry best practices that should be used by product suppliers to find, evaluate, test, and distribute patches that they receive from their operating system, database, and library suppliers. The report does not differentiate between suppliers that provide infrastructure components and those that provide the IACS applications; it contains guidance for all patches that can be applicable to the IACS.
Discover patch availability, status
Many industrial sites have 50 to 100 industrial applications installed. Tracing which O/S, database, or library patch can be applied to which final product can be very complex. This usually requires information from the product supplier, because they test patches against their products to determine which patches can be applied. Unfortunately each IACS product supplier has a different approach, format, and method for communicating the testing and qualification of patches. IACS asset owners must contact each IACS product supplier and compare the information against their asset inventory. To simplify management of this information, TR62264-3-2 includes the definition of a specific format that product suppliers can use to distribute their patch information. The format is an XML-based document that identifies the IACS product supplier, the product, the supplier providing the patch (such as the company providing the operating system), the patch supplier’s product (such as the OS version), an indication if the patch applies to the IACS product, an indication of the status of any testing of the patch against the IACS product, and an indication of the results of the testing. The goal of the ISA 99 committee is to have all IACS product suppliers publish this information, hopefully on their support website. Asset owners can then quickly download the status of the patches, compare them against their asset inventory, and determine which patches can be safely applied. While a common format for patch information will not simplify installing patches, it will significantly reduce the time to determine which patches to apply and which can be safely ignored.
Managing patches in a complex IACS environment is a difficult process. Use the ISA 99 standards and technical reports as a starting point for your own patch management processes. The best practices defined in these documents will help ensure that you minimize the risk of installing patches and minimize the risk of having unpatched automation and control systems.
– Dennis Brandl is president of BR&L Consulting in Cary, N.C., www.brlconsulting.com. His firm focuses on manufacturing IT. Edited by Mark T. Hoske, content manager, CFE Media, Control Engineering and Plant Engineering, firstname.lastname@example.org.
This posted version contains more information than the print / digital edition issue of Control Engineering.
At www.controleng.com, search Brandl for more on related topics.
See other articles at www.controleng.com/archives.