An unsupported ICS is not an easy upgrade for end users
Industrial control system (ICS) software moving into the unsupported realm is a true fear and problem for users. What are their options? The quick answer is to upgrade, but in the complex world of ICSs that is easier said than done. Inevitably, vulnerability will crop up in the unsupported system. In that case, an upgrade will take time to test to make sure the user irons out all compatibility issues. Risks just don’t stop because you are aware of a vulnerability. Keeping a system running with a vulnerability is a risky endeavor. The user needs to take some type of action.
"It is important to realize that industrial control systems are installed and commissioned to perform very specific tasks," said Joel Langill, operational security professional, ICS cyber security expert and founder of SCADAhacker.com. "If these systems are performing these tasks satisfactorily, many users may incur additional risk by deciding to upgrade from something that is ‘working’ to a product that is ‘unknown.’ For this reason, it is difficult to balance the decision to upgrade versus a potential impact to operational integrity."
Along those lines, oftentimes "ICS end users are resistant to upgrading to the first release in a new major software revision and decide to wait until the first minor release," Langill said.
Upgrading to an existing brownfield site has its benefits and detriments. Additional functionality is definitely a bonus, that is, of course, if the manufacturer ends up using the added features. The downside is the potential danger of some type of incident the upgrade could cause.
Upgrading pros and cons
"When it comes to upgrades, there are two mutually exclusive aspects one should consider," Langill said. "On one side, each release will bring new functionality. For many of us, this new functionality may not be required, and, in fact, there is always a subtle fear that a feature upon which you depend is no longer available in a new release. In this case, when the new functionality is required, the decision to upgrade is rather straightforward.
"On the other side, new releases typically address security-related issues (both publicly and privately known). Many believe that the decision to upgrade is obvious. However, remember that in addition to the functionality issue just mentioned, there is also the potential the upgrade could result in an impact to operations therefore causing an outage, an environment incident, or any number of other undesirable events. For this reason, the decision to upgrade in these cases should be solely with the end user and not ‘forced’ or ‘mandated’ by the vendor."
"Sure, they may not have fully supported software, but the cost of per-incident support could be a ‘drop in the bucket’ compared to a manufacturing outage. This is the reason why vendors should over construct mitigation strategies to assist end users when they cannot immediately perform an upgrade. These mitigations will obviously not solve the problem but can provide credible compensating controls that could reduce risk to acceptable levels that may be significantly below those resulting from an operational incident."
In the IT world, a system upgrade can occur on a regular basis, and manufacturers can install the upgrade over the weekend when no one, or very few people, is at the office. In the manufacturing automation environment, availability 24×7 is vital. Systems just can’t come down.
"Users are often reluctant to upgrade their systems, or at least not every year or two as in the IT world," said Graham Speake, vice president and chief product architect at NexDefense, Inc. "Systems are expensive and often complex, and upgrading part or all of the system may have to include a long and expensive test to ensure that everything has been configured correctly. Typically, an upgrade cycle of five years plus is the norm, but often this is stretched a lot more. Especially in the U.S., regulation may also play a part in the decision. Pharma, for instance, is often happy to run on more outdated equipment because it has been tested and approved, and an upgrade often entails an expensive retest—coupled with a loss of production."
"Remote sites such as offshore platforms are also often overlooked in the upgrade regime. The isolation (or perceived isolation) often means that these are running on much older versions of software. Minor upgrades in particular are usually overlooked-as well as a lot of security upgrades.
Speake also talked about the issues facing embedded systems. "Another area to consider is the large pieces of equipment which have a computer and operating system in it, but sometimes not in an obvious manner," he said. "Some IT equipment, such as printers, falls into this trap, but in industrial systems, items such as analyzers, which have a high capital cost and are essential to the business, are often hard to impossible to upgrade."
Solutions are available, but, as Speake said, they may not be implemented due to cost, complexity, or just a willingness to accept the risk assuming they even do a risk analysis.
"Defense-in-depth and deployment of network intrusion detection systems (NIDS), host intrusion detection systems (HIDS), [and] whitelisting are methods of managing security," said Marco Ayala, senior project manager at aeSolutions. Speake added five possible solutions beyond the ones suggested by Ayala:
1. Industrial firewalls. "These work for some solutions, but not always ideal for larger or whole system issues. I have seen these work in front of analyzers for instance, but if the whole system needs upgrading, it can be an issue."
2. Larger industrial firewalls. "These work at the boundary and are having more industrial protocols, but really are just protecting the different layers. These are getting better and will improve over the next couple of years."
3. Segmentation. "This can work with a combination of the first two to ensure the older systems are separated out."
4. Unidirectional gateways. "These can work, but are usually high dollar price and also need the right kind of system to be implemented with. The nuclear industry does use them a lot, and it is a good segmentation method. I also know that one oil and gas company is planning to deploy them on their drilling vessels."
5. Passive detection tools. "There are passive detection tools, such as an IDS or network anomaly detection tool. An IDS will need to be tuned and have rules put into it, and this may be beyond the scope of a lot of OT people. Also, there are very few open-source IDS rules for industrial protocols. There are a number of industrial behavior and anomaly detection tools coming onto the market. These have more and more awareness of industrial protocols often with deep-packet inspection built into it."
Gregory Hale is the editor and founder of Industrial Safety and Security Source (ISSSource.com), a news and information website covering safety and security issues in the manufacturing automation sector. This content originally appeared on ISSSource. Edited by Chris Vavra, production editor, CFE Media, firstname.lastname@example.org.