Matt Luallen and Steve Hamburg of Encari

The Importance of 24x7 Event Monitoring in Process Control Systems

The Importance of 24x7 Event Monitoring in Process Control Systems

Well, I (Steve here) finally succumbed to my temptation and saw the blockbuster hit Eagle Eye. (http://www.eagleeyemovie.com/ and http://www.fandango.com/eagleeye_110111/movieoverview). I’ll do my best not to ruin this movie in the event that you have not yet seen it while trying to make the key point of this blog.

Eagle Eye is based on the premise that someone, well in actuality, something (okay, I’ll stop with the hints) has extensively hacked into the power grid. While watching the movie, you’ll witness remote control of public transportation systems, cranes, demolition equipment, electricity transmission lines, etc. While the extent of exposure conveyed in the movie is quite far fetched, it does bring up a very important consideration: What are you doing in regards to your event monitoring operations?

In Eagle Eye, several severe breaches transpired before any one involved in several of our critical infrastructures became aware that such breaches had occurred. Further, a considerable amount of time following the point at which the breaches became known elapsed before the root cause of the breaches was identified. Pursuant to my previous consideration, it is critical to understand the implications associated with SCADA and process control systems security.

Unlike several professions, many aspects of critical infrastructure operations and other business operations involving SCADA and process control systems can be life threatening if a security breach occurs. For example, what could happen to a patient receiving radiation therapy from a device that is accessible from within a hospital’s wireless network? What could happen to passengers in a subway if the operations of the rail system were somehow overtaken by malicious parties? Similar questions are numerous.

While it is not possible to achieve 100% security; that is, to be impenetrable to malicious attack or inadvertent human error, it is possible to achieve robust and sufficient monitoring capabilities. However, what continues to plague this aspect of companies’ business operations is the perception that it is too costly and impractical to implement and execute sufficient 24x7 event monitoring and notification capabilities. The question is, “Is it really impractical to implement and execute sufficient 24x7 event monitoring and notification capabilities?” Matt, take it from here…

Steve, you bring up a very significant and valid point. The answer is, “No, it is not impractical as long as the organization understands what their critical business operations and their underlying support systems are.” And, Eagle Eye does depict some things entirely far-fetched, such as the transmission line over voltage scenario. What was that? For some reason I hear power engineers scoffing. 

Nonetheless, commonly a lack of clarity exists between traditional business operations, information technology, and industrial control. The challenge is to understand the business’ role and then map this in to technology appropriately. 

For instance, what does a failed access attempt mean to business operations? 

1. Someone is attempting to log into an email account

2. Someone is attempting to log into the corporate VPN

3. Someone has attempted to open a door

4. Someone is attempting to log into the operator HMI

5. A new system is attempting to interact with the historian

6. A new remote host is polling via ICCP

7. …

All of these situations have varying corporate responses that may differ with association and event context. For instance, whose email account? Log into the corporate VPN with what laptop or from what IP address? Open the door at what time? Log into the operator HMI before or after opening a door? Another directly related example would be your home. Would you respond differently to someone knocking on your front door versus your backdoor? Absolutely!

Context is very important to detecting events of interest. It is also very important to understand a system’s normal behavior. Specifically, how it operates normally - what processes are running, how much memory and CPU are utilized and to what does it normally communicate. Identifying system normalcy provides a valuable baseline to analyze against and, luckily, our process control systems are very deterministic in nature in one of two operating states - normal and emergency conditions. 

We recommend starting with very specific monitoring criteria for control systems. Typically the focus is on access attempts and with what a system communicates; exactly how to architect this depends on the control system, network, and vendor. We have recommended on many occasions using a modified network IDS to only track conversations and system event log generation, followed by sending these events to a correlation and log engine for real-time monitoring and event archival. It is recommended to start simple with the following event monitoring:

1. System logging of authentication system events directly from the HMI, FEP, and Historians; and
2. Generating netflow communications data between nodes to monitor approved and therefore unapproved system to system communications. 

Some of our peers recommend using IDS signatures that are unique to control networks; however, our experience shows that this is not necessarily the best solution. Traditional signature based IDSs have proven repeatedly to fail in their implementations. Many vendors exist for monitoring traditional IT systems – and you can use this same environment for control systems. Make sure to ask your control system vendor about what they will support; you do not want to void your support agreement by modifying the system or network and you do not want the extent of such an investment to collect dust as ‘shelf ware’. 

The best option is to reference Department of Homeland Security’s “Cyber Security Procurement Language for Control Systems”. Of course, this is most helpful when you are in the initial procurement stages of a system or support agreement; however, it can also be used to identify appropriate security controls for your specific environment. Note that caution has been continuously urged by Rita Wells of Idaho National Laboratory to use the procurement language as a tool for the process, but do not simply staple the language to a vendor order. The language is provided as a reference for selecting security options and if simply added to a vendor contract will include many conflicting alternatives.

www.encari.com

Posted by Matt Luallen and Steve Hamburg of Encari on November 12, 2008 | Comments (4)