Industrial cyber security: It’s best to learn from the mistakes of others

Engineering and IT Insight: When we don’t learn from past mistakes, we are forced to repeat them, and true to form, it has happened again. An outsourced IT department–unaware of the manufacturing elements of IT–recently shutdown production in a multi-billion dollar manufacturing company.

By Dennis Brandl January 26, 2016

Outsourcing IT security is a fairly common practice; few companies can afford the army of specialists required to maintain a secure environment and protect against attacks. An outsourced IT department—unaware of the manufacturing elements of IT—recently shutdown production in a multi-billion dollar manufacturing company.

One part of the security contract was to perform regular network security scans to look for rogue and illicit devices and unprotected ports. The IT security group did not communicate the test schedule to the company, and, mysteriously, every week, the manufacturing systems at multiple sites would shut down. Programmable logic controllers (PLCs) would mysteriously stop and require reboots or even program reloads, connected devices would reset themselves, and it took hours to get everything running again. The shutdowns occurred after normal business working hours, but manufacturing ran around the clock, so the control department was called in after hours to fix the problem. The sites thought they had a local problem, but couldn’t determine the cause.

Finally, one site noticed a network storm before the shutdown. A "network storm" is the term used to describe a flood of traffic that slows down the network and connected devices. The outsourced IT security group was simulating an intrusion, outside normal working hours, to ensure that the IT systems could handle the attack. They were doing their job. They didn’t communicate the network scan to the entire company because it would have given an intruder time to take illicit devices off-line. 

No DMZ in place

The shutdown sites did not have implemented separate business and manufacturing systems through a demilitarized zone (DMZ). It is an IT network that sits between business/corporate networks and real-time control networks. There is no direct connection through the DMZ, and all communication is routed through servers and databases. There is a firewall on each side of the DMZ and sometimes a separate user domain within it. The DMZ, firewalls, and indirect communications are the best available methods for protecting critical control networks and related devices.

The network storm hit all of the PLCs and embedded devices with more network traffic than they could handle on those sites that had not setup a DMZ to protect the control systems. Some of the PLCs and embedded devices were more than 10 years old and were not designed to handle network storms. Their communication buffers were flooded, wrote into program or data memory, and stopped. Fortunately, the processes were not inherently dangerous, so the shutdowns did not harm personnel or damage equipment but did destroy products being manufactured. The shutdowns and subsequent hours to reload and restart production cost the company tens of thousands of dollars per minute.

Devices, networks, no rules

The corporate IT response, when confronted with the problem, responded with: "Well, what are you going to do to protect against these types of attacks?" This pointed out the problem: there were no formal policies or rules for the division of responsibilities between the IT organization and the control department.

The IT organization "owned" the networks and switches; the control department "owned" the end devices. The control networks were not considered part of the control systems by IT but were by the control department. The control department had no way to fix the problem, and the IT department had no way to fix the embedded devices. There was corporate guidance for separation of networks but no monitoring of compliance.

The lesson to be learned is that corporate policies and rules for the separation of control and IT networks through DMZs are necessary, along with the need for procedures and checks to monitor sites for compliance. This company was lucky—no personnel were injured, no equipment was damaged, and they learned their lesson before a real attack happened. However, the lesson only cost millions of dollars.

Avoid a million-dollar lesson

Smart companies will learn from this problem and protect its manufacturing systems before an attack and before it costs millions in lost production. If there are no corporate policies and monitoring procedures in place to protect real-time control systems, then start developing them now.

Dennis Brandl is president of BR&L Consulting in Cary, N.C. His firm focuses on manufacturing IT. Edited by Eric R. Eissler, editor-in-chief, Oil & Gas Engineering,

ONLINE extra

This posted version contains more information than the print/digital edition issue of Control Engineering.

At Home, search Brandl for more on related topics.

See other Manufacturing IT articles.

ONLINE extra

– See related stories linked below.