Cyber security advice from the field highlights
In February, the SANS Institute held its ICS and SCADA Security Summit near Orlando, Fla. Control Engineering was able to spend some time with Michael Assante and Tim Conway, who were both on the program. Assante is currently ICS and SCADA lead for SANS, and was vice president and chief security officer at NERC. He led a key control systems group at Idaho National Labs, and was vice president and chief security officer for American Electric Power. Conway is director of NERC compliance and operations technology at NIPSCO. Matt Luallen, frequent security contributor, asked the questions. The complete 28-minute video discussion is available online, but here are a few edited highlights.
Luallen: You’re an asset owner, and you have just received word that your system has been broken into. What should you do?
Assante: How you’re notified will begin a process that should be pre-planned. There should be steps to begin the analysis:
- What do we think the compromise affected?
- How do we collect the right information?
- How do we make the right decisions?
A lot of people will make decisions right away and act, but sometimes it’s better to get a feel for what’s happening before you begin to take action in terms of a response. Hopefully, you have a playbook, and you exercise that playbook. Understanding what’s been affected is critical, and being able to inform the operations staff that the integrity and availability of a system that they might be relying upon is affected is paramount.
You could be fooled very easily to think that where the artifact or evidence came from is where the compromise occurred, and is the only area which has been affected. You have to step back and realize that, if you look at statistics, the mean time to compromise is over 400 days. So just because the evidence comes from one box doesn’t mean that it is the only box involved in this event. You need to err on the side of warning too many folks that there is an incident underway that might affect operations, then you try to get an indication of what all could be affected. Then you attack it.
Conway: The notification and communication is key—everybody must understand the information as it’s coming in: what you know, what it’s going to impact, how you’re going to respond, and that you’re going to exercise the way you’ve been practicing for years. If you’re a company that hasn’t practiced your incident response plans, you haven’t practiced who you’re going to notify, how you’re going to communicate, or how you’re going to contain this within an industrial control system environment; dealing with it when it happens isn’t the time you want to do it. You want to pretend all the time that you’re a company that’s just been attacked. Practice this at various levels of your network, at various levels of your organization. Then determine what you need to do to contain it, eradicate it, and remove it. Then, when an event actually happens, you’re following that playbook. You follow operating procedures like an operator sitting at a desk. When something happens in the electrical system, operators have operating guides and they have procedures that they follow. Our cyber teams need to do the same thing.
Luallen: When dealing with a compromise, what are the important relationships, inside and outside the organization, that an asset owner needs to call upon?
Conway: Definitely your corporate security teams, physical security teams, and your corporate communications teams. Outside of the organization, the FBI and ES-ISAC (Electricity Sector Information Sharing and Analysis Center) are two critical resources for us. Local FBI and state investigators are very important. We involve them in our exercises and drills. Most people see the ES-ISAC relationship as a required reporting function, but we have routine conversations with them about things that we’re seeing, events of interest, voluntary notifications that are not required at all, but they are very valuable conversations.
Assante: I would add that you need to think about external parties, not just the government and law enforcement, but you also need to think about your critical suppliers. We’re very dependent on them for how they troubleshoot our systems, especially in the ICS domain, and quite honestly, I think it would be very hard to mount an effective response in a control system without their expertise. Contractually, or even if a contract is not in place, suppliers have said, “Reach out to us. We’re interested in seeing how our technology is being exploited, and we want to help.” There are other third parties. In my NERC experience and at AEP, I reached out to third parties because I was usually able to get help. When I was at NERC, I thought there was an obligation to reach out, especially to my interconnected utilities. If my system was impacted and I’m interconnected with other utilities where we’re sharing monitoring on a device in a substation in a system, then it’s critical that I’m also talking to them, seeing if they’re seeing the same thing. Maybe they were the actual path. I want to understand that. Where did it begin to affect my systems? Or, maybe I’m the path to them. In an interconnected sense, that’s very important.
Luallen: You’re a plant manager, and your boss tells you that your corporate IT department is going to manage cyber security for the plant. You know there’s nothing you can do to change the decision, so what should you do?
Assante: There are tough realities out there, and at the end of the day, a plant manager needs to know that he has to build a successful team in order to do the work. That’s what plant managers are good at. If that’s what’s going to happen, a plant manager has to be up front about what it takes to qualify somebody to work in these environments. There’s a whole regime of safety that they’re going to have to get before IT professionals can come and interface with the equipment. They’ll have to partner with process control engineers and the plant staff to be able to perform some of this work. They’ll need aggressive education. There are commonalities in terms of the work stations at the operating system level, but they’re not going to be familiar with the protocols used at the fieldbus level—that’s going to be new to an IT person. They will have to respect the operational constraints and drivers as to how and why they perform their work in these environments. The plant manager is going to have to spend a lot of time with the CIO, and with the CEO above the CIO, to explain why he can accomplish it, but we have to follow a path, and the path is going to include education, a strong understanding of how the asset works, so there’s going to be a time where they’re taking lots of field trips and spending some time in the plant. We know we’ve qualified them to be in a work environment, but can they change their philosophy on how they perform the work of IT and IT security. You’re used to doing something in a server environment that you can’t necessarily do the same way on a plant floor. You need to learn operational things like scheduling, outages, and how you plan your work. If you’re going to do it, you’re going to be a very busy guy for the next six to eight months.
Conway: So much about that situation is cultural in order for it to work. It would almost be an easier lift to take the current operations technology engineers and educate and teach them on what they need to do to work within the IT environment. The reason that such a decision would have been made likely would have been because of the benefits that exist on the IT side with some of the management tools, the monitoring tools, the alerting, patching, and the processes and procedures that are well defined in IT, and operations technology is still struggling to get there. But it would be culturally easier to train engineers on what they need to know to perform those IT functions and enable them to do it. That would be much more successful. If I’m a plant manager, I’m mission focused. I know what I need to do in my role, and in order to guarantee success, I would pursue that option of taking my operations technology engineers and enabling them on the IT side.
Assante: I would suggest, instead of waiting to hear this news one day, I think organizations have to look hard and try to think about the future. In the future, and we’re seeing this already, plant floor analytics are moving into the cloud. You’re constrained as to what you can do at the plant floor, so we’re integrating our plant systems with operational performance management systems—everything from the supply chain side of things to actual production scheduling tools. You have a lot of IT environments where IT specialists can really have a benefit. Our OT world is changing. The fieldbus might not change, and there are going to be places that change slowly, but it’s not a bad idea to start thinking about how you do start getting IT professionals to understand how we work at this level and where the crossovers are today. I would argue that you don’t want to wait. You should start that process now, knowing that in the future we’re going to depend more on IT. There are a lot of applications that exist in our OT environments, such as databases, where the OT folks don’t necessarily have a lot of expertise or experience. We could learn a lot from a good database security person that could help improve the security of your industrial control system.