Industrial cyber security: It’s best to learn from the mistakes of others

Engineering and IT Insight: When we don’t learn from past mistakes, we are forced to repeat them, and true to form, it has happened again. An outsourced IT department–unaware of the manufacturing elements of IT–recently shutdown production in a multi-billion dollar manufacturing company.

01/26/2016


Outsourcing IT security is a fairly common practice; few companies can afford the army of specialists required to maintain a secure environment and protect against attacks. An outsourced IT department—unaware of the manufacturing elements of IT—recently shutdown production in a multi-billion dollar manufacturing company.

One part of the security contract was to perform regular network security scans to look for rogue and illicit devices and unprotected ports. The IT security group did not communicate the test schedule to the company, and, mysteriously, every week, the manufacturing systems at multiple sites would shut down. Programmable logic controllers (PLCs) would mysteriously stop and require reboots or even program reloads, connected devices would reset themselves, and it took hours to get everything running again. The shutdowns occurred after normal business working hours, but manufacturing ran around the clock, so the control department was called in after hours to fix the problem. The sites thought they had a local problem, but couldn't determine the cause.

Finally, one site noticed a network storm before the shutdown. A "network storm" is the term used to describe a flood of traffic that slows down the network and connected devices. The outsourced IT security group was simulating an intrusion, outside normal working hours, to ensure that the IT systems could handle the attack. They were doing their job. They didn't communicate the network scan to the entire company because it would have given an intruder time to take illicit devices off-line. 

No DMZ in place

The shutdown sites did not have implemented separate business and manufacturing systems through a demilitarized zone (DMZ). It is an IT network that sits between business/corporate networks and real-time control networks. There is no direct connection through the DMZ, and all communication is routed through servers and databases. There is a firewall on each side of the DMZ and sometimes a separate user domain within it. The DMZ, firewalls, and indirect communications are the best available methods for protecting critical control networks and related devices.

The network storm hit all of the PLCs and embedded devices with more network traffic than they could handle on those sites that had not setup a DMZ to protect the control systems. Some of the PLCs and embedded devices were more than 10 years old and were not designed to handle network storms. Their communication buffers were flooded, wrote into program or data memory, and stopped. Fortunately, the processes were not inherently dangerous, so the shutdowns did not harm personnel or damage equipment but did destroy products being manufactured. The shutdowns and subsequent hours to reload and restart production cost the company tens of thousands of dollars per minute.

Devices, networks, no rules

The corporate IT response, when confronted with the problem, responded with: "Well, what are you going to do to protect against these types of attacks?" This pointed out the problem: there were no formal policies or rules for the division of responsibilities between the IT organization and the control department.

The IT organization "owned" the networks and switches; the control department "owned" the end devices. The control networks were not considered part of the control systems by IT but were by the control department. The control department had no way to fix the problem, and the IT department had no way to fix the embedded devices. There was corporate guidance for separation of networks but no monitoring of compliance.

The lesson to be learned is that corporate policies and rules for the separation of control and IT networks through DMZs are necessary, along with the need for procedures and checks to monitor sites for compliance. This company was lucky—no personnel were injured, no equipment was damaged, and they learned their lesson before a real attack happened. However, the lesson only cost millions of dollars.

Dennis Brandl is a founder and chief consultant at BR&L Consulting Inc., specializing in helping companies use Manufacturing IT to improve production, laboratory, and logistics processes. Courtesy: BR&L Consulting Inc.Avoid a million-dollar lesson

Smart companies will learn from this problem and protect its manufacturing systems before an attack and before it costs millions in lost production. If there are no corporate policies and monitoring procedures in place to protect real-time control systems, then start developing them now.

Dennis Brandl is president of BR&L Consulting in Cary, N.C. His firm focuses on manufacturing IT. Edited by Eric R. Eissler, editor-in-chief, Oil & Gas Engineering, eeissler@cfemedia.com.

ONLINE extra

This posted version contains more information than the print/digital edition issue of Control Engineering.

At www.controleng.com, search Brandl for more on related topics.

See other Manufacturing IT articles.

ONLINE extra

- See related stories linked below.



No comments
The Engineers' Choice Awards highlight some of the best new control, instrumentation and automation products as chosen by...
The System Integrator Giants program lists the top 100 system integrators among companies listed in CFE Media's Global System Integrator Database.
The Engineering Leaders Under 40 program identifies and gives recognition to young engineers who...
This eGuide illustrates solutions, applications and benefits of machine vision systems.
Learn how to increase device reliability in harsh environments and decrease unplanned system downtime.
This eGuide contains a series of articles and videos that considers theoretical and practical; immediate needs and a look into the future.
Sensor-to-cloud interoperability; PID and digital control efficiency; Alarm management system design; Automotive industry advances
Make Big Data and Industrial Internet of Things work for you, 2017 Engineers' Choice Finalists, Avoid control design pitfalls, Managing IIoT processes
Engineering Leaders Under 40; System integration improving packaging operation; Process sensing; PID velocity; Cybersecurity and functional safety
This article collection contains several articles on the Industrial Internet of Things (IIoT) and how it is transforming manufacturing.

Find and connect with the most suitable service provider for your unique application. Start searching the Global System Integrator Database Now!

SCADA at the junction, Managing risk through maintenance, Moving at the speed of data
Flexible offshore fire protection; Big Data's impact on operations; Bridging the skills gap; Identifying security risks
The digital oilfield: Utilizing Big Data can yield big savings; Virtualization a real solution; Tracking SIS performance
click me