Industrial cyber security: It’s best to learn from the mistakes of others

Engineering and IT Insight: When we don’t learn from past mistakes, we are forced to repeat them, and true to form, it has happened again. An outsourced IT department–unaware of the manufacturing elements of IT–recently shutdown production in a multi-billion dollar manufacturing company.

01/26/2016


Outsourcing IT security is a fairly common practice; few companies can afford the army of specialists required to maintain a secure environment and protect against attacks. An outsourced IT department—unaware of the manufacturing elements of IT—recently shutdown production in a multi-billion dollar manufacturing company.

One part of the security contract was to perform regular network security scans to look for rogue and illicit devices and unprotected ports. The IT security group did not communicate the test schedule to the company, and, mysteriously, every week, the manufacturing systems at multiple sites would shut down. Programmable logic controllers (PLCs) would mysteriously stop and require reboots or even program reloads, connected devices would reset themselves, and it took hours to get everything running again. The shutdowns occurred after normal business working hours, but manufacturing ran around the clock, so the control department was called in after hours to fix the problem. The sites thought they had a local problem, but couldn't determine the cause.

Finally, one site noticed a network storm before the shutdown. A "network storm" is the term used to describe a flood of traffic that slows down the network and connected devices. The outsourced IT security group was simulating an intrusion, outside normal working hours, to ensure that the IT systems could handle the attack. They were doing their job. They didn't communicate the network scan to the entire company because it would have given an intruder time to take illicit devices off-line. 

No DMZ in place

The shutdown sites did not have implemented separate business and manufacturing systems through a demilitarized zone (DMZ). It is an IT network that sits between business/corporate networks and real-time control networks. There is no direct connection through the DMZ, and all communication is routed through servers and databases. There is a firewall on each side of the DMZ and sometimes a separate user domain within it. The DMZ, firewalls, and indirect communications are the best available methods for protecting critical control networks and related devices.

The network storm hit all of the PLCs and embedded devices with more network traffic than they could handle on those sites that had not setup a DMZ to protect the control systems. Some of the PLCs and embedded devices were more than 10 years old and were not designed to handle network storms. Their communication buffers were flooded, wrote into program or data memory, and stopped. Fortunately, the processes were not inherently dangerous, so the shutdowns did not harm personnel or damage equipment but did destroy products being manufactured. The shutdowns and subsequent hours to reload and restart production cost the company tens of thousands of dollars per minute.

Devices, networks, no rules

The corporate IT response, when confronted with the problem, responded with: "Well, what are you going to do to protect against these types of attacks?" This pointed out the problem: there were no formal policies or rules for the division of responsibilities between the IT organization and the control department.

The IT organization "owned" the networks and switches; the control department "owned" the end devices. The control networks were not considered part of the control systems by IT but were by the control department. The control department had no way to fix the problem, and the IT department had no way to fix the embedded devices. There was corporate guidance for separation of networks but no monitoring of compliance.

The lesson to be learned is that corporate policies and rules for the separation of control and IT networks through DMZs are necessary, along with the need for procedures and checks to monitor sites for compliance. This company was lucky—no personnel were injured, no equipment was damaged, and they learned their lesson before a real attack happened. However, the lesson only cost millions of dollars.

Dennis Brandl is a founder and chief consultant at BR&L Consulting Inc., specializing in helping companies use Manufacturing IT to improve production, laboratory, and logistics processes. Courtesy: BR&L Consulting Inc.Avoid a million-dollar lesson

Smart companies will learn from this problem and protect its manufacturing systems before an attack and before it costs millions in lost production. If there are no corporate policies and monitoring procedures in place to protect real-time control systems, then start developing them now.

Dennis Brandl is president of BR&L Consulting in Cary, N.C. His firm focuses on manufacturing IT. Edited by Eric R. Eissler, editor-in-chief, Oil & Gas Engineering, eeissler@cfemedia.com.

ONLINE extra

This posted version contains more information than the print/digital edition issue of Control Engineering.

At www.controleng.com, search Brandl for more on related topics.

See other Manufacturing IT articles.

ONLINE extra

- See related stories linked below.



The Engineers' Choice Awards highlight some of the best new control, instrumentation and automation products as chosen by Control Engineering subscribers. Vote now (if qualified)!
The System Integrator Giants program lists the top 100 system integrators among companies listed in CFE Media's Global System Integrator Database.
Each year, a panel of Control Engineering and Plant Engineering editors and industry expert judges select the System Integrator of the Year Award winners in three categories.
This eGuide illustrates solutions, applications and benefits of machine vision systems.
Learn how to increase device reliability in harsh environments and decrease unplanned system downtime.
This eGuide contains a series of articles and videos that considers theoretical and practical; immediate needs and a look into the future.
Maximize ROI with integrated control system approach; Microcontrollers vs. PLCs; Power quality; Accelerate and rewire IIoT; Traits for excellent engineers
HMI effectiveness; Distributed I/O; Engineers' Choice Award finalists; System Integrator advice; Inside Machines
Women in engineering; Engineering Leaders Under 40; PID benefits and drawbacks; Ladder logic; Cloud computing
Programmable logic controllers (PLCs) represent the logic (decision) part of the control loop of sense, decide, and actuate. As we know, PLCs aren’t the only option for making decisions in a control loop, but they are likely why you’re here.
This digital report explains how plant engineers and subject matter experts (SME) need support for time series data and its many challenges.
This article collection contains several articles on how advancements in vision system designs, computing power, algorithms, optics, and communications are making machine vision more cost effective than ever before.

Find and connect with the most suitable service provider for your unique application. Start searching the Global System Integrator Database Now!

Control room technology innovation; Practical approaches to corrosion protection; Pipeline regulator revises quality programs
Cloud, mobility, and remote operations; SCADA and contextual mobility; Custom UPS empowering a secure pipeline
Infrastructure for natural gas expansion; Artificial lift methods; Disruptive technology and fugitive gas emissions
Automation Engineer; Wood Group
System Integrator; Cross Integrated Systems Group
Jose S. Vasquez, Jr.
Fire & Life Safety Engineer; Technip USA Inc.
This course focuses on climate analysis, appropriateness of cooling system selection, and combining cooling systems.
This course will help identify and reveal electrical hazards and identify the solutions to implementing and maintaining a safe work environment.
This course explains how maintaining power and communication systems through emergency power-generation systems is critical.
The Engineers' Choice Awards highlight some of the best new control, instrumentation and automation products as chosen by Control Engineering subscribers. Vote now (if qualified)!
The System Integrator Giants program lists the top 100 system integrators among companies listed in CFE Media's Global System Integrator Database.
Each year, a panel of Control Engineering and Plant Engineering editors and industry expert judges select the System Integrator of the Year Award winners in three categories.
This eGuide illustrates solutions, applications and benefits of machine vision systems.
Learn how to increase device reliability in harsh environments and decrease unplanned system downtime.
This eGuide contains a series of articles and videos that considers theoretical and practical; immediate needs and a look into the future.
Maximize ROI with integrated control system approach; Microcontrollers vs. PLCs; Power quality; Accelerate and rewire IIoT; Traits for excellent engineers
HMI effectiveness; Distributed I/O; Engineers' Choice Award finalists; System Integrator advice; Inside Machines
Women in engineering; Engineering Leaders Under 40; PID benefits and drawbacks; Ladder logic; Cloud computing
Programmable logic controllers (PLCs) represent the logic (decision) part of the control loop of sense, decide, and actuate. As we know, PLCs aren’t the only option for making decisions in a control loop, but they are likely why you’re here.
This digital report explains how plant engineers and subject matter experts (SME) need support for time series data and its many challenges.
This article collection contains several articles on how advancements in vision system designs, computing power, algorithms, optics, and communications are making machine vision more cost effective than ever before.

Find and connect with the most suitable service provider for your unique application. Start searching the Global System Integrator Database Now!

Control room technology innovation; Practical approaches to corrosion protection; Pipeline regulator revises quality programs
Cloud, mobility, and remote operations; SCADA and contextual mobility; Custom UPS empowering a secure pipeline
Infrastructure for natural gas expansion; Artificial lift methods; Disruptive technology and fugitive gas emissions
Automation Engineer; Wood Group
System Integrator; Cross Integrated Systems Group
Jose S. Vasquez, Jr.
Fire & Life Safety Engineer; Technip USA Inc.
This course focuses on climate analysis, appropriateness of cooling system selection, and combining cooling systems.
This course will help identify and reveal electrical hazards and identify the solutions to implementing and maintaining a safe work environment.
This course explains how maintaining power and communication systems through emergency power-generation systems is critical.
The Engineers' Choice Awards highlight some of the best new control, instrumentation and automation products as chosen by Control Engineering subscribers. Vote now (if qualified)!
The System Integrator Giants program lists the top 100 system integrators among companies listed in CFE Media's Global System Integrator Database.
Each year, a panel of Control Engineering and Plant Engineering editors and industry expert judges select the System Integrator of the Year Award winners in three categories.
This eGuide illustrates solutions, applications and benefits of machine vision systems.
Learn how to increase device reliability in harsh environments and decrease unplanned system downtime.
This eGuide contains a series of articles and videos that considers theoretical and practical; immediate needs and a look into the future.
Maximize ROI with integrated control system approach; Microcontrollers vs. PLCs; Power quality; Accelerate and rewire IIoT; Traits for excellent engineers
HMI effectiveness; Distributed I/O; Engineers' Choice Award finalists; System Integrator advice; Inside Machines
Women in engineering; Engineering Leaders Under 40; PID benefits and drawbacks; Ladder logic; Cloud computing
Programmable logic controllers (PLCs) represent the logic (decision) part of the control loop of sense, decide, and actuate. As we know, PLCs aren’t the only option for making decisions in a control loop, but they are likely why you’re here.
This digital report explains how plant engineers and subject matter experts (SME) need support for time series data and its many challenges.
This article collection contains several articles on how advancements in vision system designs, computing power, algorithms, optics, and communications are making machine vision more cost effective than ever before.

Find and connect with the most suitable service provider for your unique application. Start searching the Global System Integrator Database Now!

Control room technology innovation; Practical approaches to corrosion protection; Pipeline regulator revises quality programs
Cloud, mobility, and remote operations; SCADA and contextual mobility; Custom UPS empowering a secure pipeline
Infrastructure for natural gas expansion; Artificial lift methods; Disruptive technology and fugitive gas emissions
Automation Engineer; Wood Group
System Integrator; Cross Integrated Systems Group
Jose S. Vasquez, Jr.
Fire & Life Safety Engineer; Technip USA Inc.
This course focuses on climate analysis, appropriateness of cooling system selection, and combining cooling systems.
This course will help identify and reveal electrical hazards and identify the solutions to implementing and maintaining a safe work environment.
This course explains how maintaining power and communication systems through emergency power-generation systems is critical.
click me