Industrial cyber security: It’s best to learn from the mistakes of others

Engineering and IT Insight: When we don’t learn from past mistakes, we are forced to repeat them, and true to form, it has happened again. An outsourced IT department–unaware of the manufacturing elements of IT–recently shutdown production in a multi-billion dollar manufacturing company.


Outsourcing IT security is a fairly common practice; few companies can afford the army of specialists required to maintain a secure environment and protect against attacks. An outsourced IT department—unaware of the manufacturing elements of IT—recently shutdown production in a multi-billion dollar manufacturing company.

One part of the security contract was to perform regular network security scans to look for rogue and illicit devices and unprotected ports. The IT security group did not communicate the test schedule to the company, and, mysteriously, every week, the manufacturing systems at multiple sites would shut down. Programmable logic controllers (PLCs) would mysteriously stop and require reboots or even program reloads, connected devices would reset themselves, and it took hours to get everything running again. The shutdowns occurred after normal business working hours, but manufacturing ran around the clock, so the control department was called in after hours to fix the problem. The sites thought they had a local problem, but couldn't determine the cause.

Finally, one site noticed a network storm before the shutdown. A "network storm" is the term used to describe a flood of traffic that slows down the network and connected devices. The outsourced IT security group was simulating an intrusion, outside normal working hours, to ensure that the IT systems could handle the attack. They were doing their job. They didn't communicate the network scan to the entire company because it would have given an intruder time to take illicit devices off-line. 

No DMZ in place

The shutdown sites did not have implemented separate business and manufacturing systems through a demilitarized zone (DMZ). It is an IT network that sits between business/corporate networks and real-time control networks. There is no direct connection through the DMZ, and all communication is routed through servers and databases. There is a firewall on each side of the DMZ and sometimes a separate user domain within it. The DMZ, firewalls, and indirect communications are the best available methods for protecting critical control networks and related devices.

The network storm hit all of the PLCs and embedded devices with more network traffic than they could handle on those sites that had not setup a DMZ to protect the control systems. Some of the PLCs and embedded devices were more than 10 years old and were not designed to handle network storms. Their communication buffers were flooded, wrote into program or data memory, and stopped. Fortunately, the processes were not inherently dangerous, so the shutdowns did not harm personnel or damage equipment but did destroy products being manufactured. The shutdowns and subsequent hours to reload and restart production cost the company tens of thousands of dollars per minute.

Devices, networks, no rules

The corporate IT response, when confronted with the problem, responded with: "Well, what are you going to do to protect against these types of attacks?" This pointed out the problem: there were no formal policies or rules for the division of responsibilities between the IT organization and the control department.

The IT organization "owned" the networks and switches; the control department "owned" the end devices. The control networks were not considered part of the control systems by IT but were by the control department. The control department had no way to fix the problem, and the IT department had no way to fix the embedded devices. There was corporate guidance for separation of networks but no monitoring of compliance.

The lesson to be learned is that corporate policies and rules for the separation of control and IT networks through DMZs are necessary, along with the need for procedures and checks to monitor sites for compliance. This company was lucky—no personnel were injured, no equipment was damaged, and they learned their lesson before a real attack happened. However, the lesson only cost millions of dollars.

Dennis Brandl is a founder and chief consultant at BR&L Consulting Inc., specializing in helping companies use Manufacturing IT to improve production, laboratory, and logistics processes. Courtesy: BR&L Consulting Inc.Avoid a million-dollar lesson

Smart companies will learn from this problem and protect its manufacturing systems before an attack and before it costs millions in lost production. If there are no corporate policies and monitoring procedures in place to protect real-time control systems, then start developing them now.

Dennis Brandl is president of BR&L Consulting in Cary, N.C. His firm focuses on manufacturing IT. Edited by Eric R. Eissler, editor-in-chief, Oil & Gas Engineering,

ONLINE extra

This posted version contains more information than the print/digital edition issue of Control Engineering.

At, search Brandl for more on related topics.

See other Manufacturing IT articles.

ONLINE extra

- See related stories linked below.

No comments
The Engineers' Choice Awards highlight some of the best new control, instrumentation and automation products as chosen by...
The System Integrator Giants program lists the top 100 system integrators among companies listed in CFE Media's Global System Integrator Database.
Each year, a panel of Control Engineering and Plant Engineering editors and industry expert judges select the System Integrator of the Year Award winners in three categories.
This eGuide illustrates solutions, applications and benefits of machine vision systems.
Learn how to increase device reliability in harsh environments and decrease unplanned system downtime.
This eGuide contains a series of articles and videos that considers theoretical and practical; immediate needs and a look into the future.
Additive manufacturing benefits; HMI and sensor tips; System integrator advice; Innovations from the industry
Robotic safety, collaboration, standards; DCS migration tips; IT/OT convergence; 2017 Control Engineering Salary and Career Survey
Integrated mobility; Artificial intelligence; Predictive motion control; Sensors and control system inputs; Asset Management; Cybersecurity
Featured articles highlight technologies that enable the Industrial Internet of Things, IIoT-related products and strategies to get data more easily to the user.
This article collection contains several articles on how automation and controls are helping human-machine interface (HMI) hardware and software advance.
This digital report will explore several aspects of how IIoT will transform manufacturing in the coming years.

Find and connect with the most suitable service provider for your unique application. Start searching the Global System Integrator Database Now!

Infrastructure for natural gas expansion; Artificial lift methods; Disruptive technology and fugitive gas emissions
Mobility as the means to offshore innovation; Preventing another Deepwater Horizon; ROVs as subsea robots; SCADA and the radio spectrum
Future of oil and gas projects; Reservoir models; The importance of SCADA to oil and gas
Automation Engineer; Wood Group
System Integrator; Cross Integrated Systems Group
Jose S. Vasquez, Jr.
Fire & Life Safety Engineer; Technip USA Inc.
This course focuses on climate analysis, appropriateness of cooling system selection, and combining cooling systems.
This course will help identify and reveal electrical hazards and identify the solutions to implementing and maintaining a safe work environment.
This course explains how maintaining power and communication systems through emergency power-generation systems is critical.
click me