Industrial cyber security: It’s best to learn from the mistakes of others

Engineering and IT Insight: When we don’t learn from past mistakes, we are forced to repeat them, and true to form, it has happened again. An outsourced IT department–unaware of the manufacturing elements of IT–recently shutdown production in a multi-billion dollar manufacturing company.

01/26/2016


Outsourcing IT security is a fairly common practice; few companies can afford the army of specialists required to maintain a secure environment and protect against attacks. An outsourced IT department—unaware of the manufacturing elements of IT—recently shutdown production in a multi-billion dollar manufacturing company.

One part of the security contract was to perform regular network security scans to look for rogue and illicit devices and unprotected ports. The IT security group did not communicate the test schedule to the company, and, mysteriously, every week, the manufacturing systems at multiple sites would shut down. Programmable logic controllers (PLCs) would mysteriously stop and require reboots or even program reloads, connected devices would reset themselves, and it took hours to get everything running again. The shutdowns occurred after normal business working hours, but manufacturing ran around the clock, so the control department was called in after hours to fix the problem. The sites thought they had a local problem, but couldn't determine the cause.

Finally, one site noticed a network storm before the shutdown. A "network storm" is the term used to describe a flood of traffic that slows down the network and connected devices. The outsourced IT security group was simulating an intrusion, outside normal working hours, to ensure that the IT systems could handle the attack. They were doing their job. They didn't communicate the network scan to the entire company because it would have given an intruder time to take illicit devices off-line. 

No DMZ in place

The shutdown sites did not have implemented separate business and manufacturing systems through a demilitarized zone (DMZ). It is an IT network that sits between business/corporate networks and real-time control networks. There is no direct connection through the DMZ, and all communication is routed through servers and databases. There is a firewall on each side of the DMZ and sometimes a separate user domain within it. The DMZ, firewalls, and indirect communications are the best available methods for protecting critical control networks and related devices.

The network storm hit all of the PLCs and embedded devices with more network traffic than they could handle on those sites that had not setup a DMZ to protect the control systems. Some of the PLCs and embedded devices were more than 10 years old and were not designed to handle network storms. Their communication buffers were flooded, wrote into program or data memory, and stopped. Fortunately, the processes were not inherently dangerous, so the shutdowns did not harm personnel or damage equipment but did destroy products being manufactured. The shutdowns and subsequent hours to reload and restart production cost the company tens of thousands of dollars per minute.

Devices, networks, no rules

The corporate IT response, when confronted with the problem, responded with: "Well, what are you going to do to protect against these types of attacks?" This pointed out the problem: there were no formal policies or rules for the division of responsibilities between the IT organization and the control department.

The IT organization "owned" the networks and switches; the control department "owned" the end devices. The control networks were not considered part of the control systems by IT but were by the control department. The control department had no way to fix the problem, and the IT department had no way to fix the embedded devices. There was corporate guidance for separation of networks but no monitoring of compliance.

The lesson to be learned is that corporate policies and rules for the separation of control and IT networks through DMZs are necessary, along with the need for procedures and checks to monitor sites for compliance. This company was lucky—no personnel were injured, no equipment was damaged, and they learned their lesson before a real attack happened. However, the lesson only cost millions of dollars.

Dennis Brandl is a founder and chief consultant at BR&L Consulting Inc., specializing in helping companies use Manufacturing IT to improve production, laboratory, and logistics processes. Courtesy: BR&L Consulting Inc.Avoid a million-dollar lesson

Smart companies will learn from this problem and protect its manufacturing systems before an attack and before it costs millions in lost production. If there are no corporate policies and monitoring procedures in place to protect real-time control systems, then start developing them now.

Dennis Brandl is president of BR&L Consulting in Cary, N.C. His firm focuses on manufacturing IT. Edited by Eric R. Eissler, editor-in-chief, Oil & Gas Engineering, eeissler@cfemedia.com.

ONLINE extra

This posted version contains more information than the print/digital edition issue of Control Engineering.

At www.controleng.com, search Brandl for more on related topics.

See other Manufacturing IT articles.

ONLINE extra

- See related stories linked below.



Engineers' Choice Awards
The Engineers' Choice Awards highlight some of the best new control, instrumentation and automation products as chosen by Control Engineering subscribers.
System Integrator Giants
The System Integrator Giants program lists the top 100 system integrators among companies listed in CFE Media's Global System Integrator Database.
System Integrator of the Year
Each year, a panel of Control Engineering and Plant Engineering editors and industry expert judges select the System Integrator of the Year Award winners in three categories.
Design of Safe and Reliable Hydraulic Systems for Subsea Applications
This eGuide explains how the operation of hydraulic systems for subsea applications requires the user to consider additional aspects because of the unique conditions that apply to the setting
How to Maximize Factory Automation Efficiency with Low Cost Machine Vision
This eGuide illustrates solutions, applications and benefits of machine vision systems.
Wireless Reliability in Harsh Environments
Learn how to increase device reliability in harsh environments and decrease unplanned system downtime.
October 2018
HMI hardware evolution, Data acquisition strategies, Matching motors and drives, Machine vision advice
September 2018
Optimize controls via cloud software, ladder logic simulation, industrial wireless best practices
August 2018
Augmented reality and virtual reality education, autotuning PID control, cybersecurity advice, educating engineers
Edge Computing
This article collection contains several articles on how today's technologies heap benefits onto an edge-computing architecture such as faster computing, better networking, more memory, smarter analytics, cloud-based intelligence, and lower costs.
Data Center Design
Data centers, data closets, edge and cloud computing, co-location facilities, and similar topics are among the fastest-changing in the industry.
IIoT: Machines, Equipment, & Asset Management
Articles in this digital report highlight technologies that enable Industrial Internet of Things, IIoT-related products and strategies.
SIDB

Find and connect with the most suitable service provider for your unique application. Start searching the Global System Integrator Database Now!

October 2018
2018 Product of the Year; Subsurface data methodologies; Digital twins; Well lifecycle data
August 2018
SCADA standardization, capital expenditures, data-driven drilling and execution
June 2018
Machine learning, produced water benefits, progressive cavity pumps
John O. Ayuk, PE, CFSE, PMP, CAP
Automation Engineer; Wood Group
Doug Baker
System Integrator; Cross Integrated Systems Group
Jose S. Vasquez, Jr.
Jose S. Vasquez, Jr.
Fire & Life Safety Engineer; Technip USA Inc.
Data Centers: Impacts of Climate and Cooling Technology
This course focuses on climate analysis, appropriateness of cooling system selection, and combining cooling systems.
Safety First: Arc Flash 101
This course will help identify and reveal electrical hazards and identify the solutions to implementing and maintaining a safe work environment.
Critical Power: Hospital Electrical Systems
This course explains how maintaining power and communication systems through emergency power-generation systems is critical.
Engineers' Choice Awards
The Engineers' Choice Awards highlight some of the best new control, instrumentation and automation products as chosen by Control Engineering subscribers.
System Integrator Giants
The System Integrator Giants program lists the top 100 system integrators among companies listed in CFE Media's Global System Integrator Database.
System Integrator of the Year
Each year, a panel of Control Engineering and Plant Engineering editors and industry expert judges select the System Integrator of the Year Award winners in three categories.
Design of Safe and Reliable Hydraulic Systems for Subsea Applications
This eGuide explains how the operation of hydraulic systems for subsea applications requires the user to consider additional aspects because of the unique conditions that apply to the setting
How to Maximize Factory Automation Efficiency with Low Cost Machine Vision
This eGuide illustrates solutions, applications and benefits of machine vision systems.
Wireless Reliability in Harsh Environments
Learn how to increase device reliability in harsh environments and decrease unplanned system downtime.
October 2018
HMI hardware evolution, Data acquisition strategies, Matching motors and drives, Machine vision advice
September 2018
Optimize controls via cloud software, ladder logic simulation, industrial wireless best practices
August 2018
Augmented reality and virtual reality education, autotuning PID control, cybersecurity advice, educating engineers
Edge Computing
This article collection contains several articles on how today's technologies heap benefits onto an edge-computing architecture such as faster computing, better networking, more memory, smarter analytics, cloud-based intelligence, and lower costs.
Data Center Design
Data centers, data closets, edge and cloud computing, co-location facilities, and similar topics are among the fastest-changing in the industry.
IIoT: Machines, Equipment, & Asset Management
Articles in this digital report highlight technologies that enable Industrial Internet of Things, IIoT-related products and strategies.
SIDB

Find and connect with the most suitable service provider for your unique application. Start searching the Global System Integrator Database Now!

October 2018
2018 Product of the Year; Subsurface data methodologies; Digital twins; Well lifecycle data
August 2018
SCADA standardization, capital expenditures, data-driven drilling and execution
June 2018
Machine learning, produced water benefits, progressive cavity pumps
John O. Ayuk, PE, CFSE, PMP, CAP
Automation Engineer; Wood Group
Doug Baker
System Integrator; Cross Integrated Systems Group
Jose S. Vasquez, Jr.
Jose S. Vasquez, Jr.
Fire & Life Safety Engineer; Technip USA Inc.
Data Centers: Impacts of Climate and Cooling Technology
This course focuses on climate analysis, appropriateness of cooling system selection, and combining cooling systems.
Safety First: Arc Flash 101
This course will help identify and reveal electrical hazards and identify the solutions to implementing and maintaining a safe work environment.
Critical Power: Hospital Electrical Systems
This course explains how maintaining power and communication systems through emergency power-generation systems is critical.
Engineers' Choice Awards
The Engineers' Choice Awards highlight some of the best new control, instrumentation and automation products as chosen by Control Engineering subscribers.
System Integrator Giants
The System Integrator Giants program lists the top 100 system integrators among companies listed in CFE Media's Global System Integrator Database.
System Integrator of the Year
Each year, a panel of Control Engineering and Plant Engineering editors and industry expert judges select the System Integrator of the Year Award winners in three categories.
Design of Safe and Reliable Hydraulic Systems for Subsea Applications
This eGuide explains how the operation of hydraulic systems for subsea applications requires the user to consider additional aspects because of the unique conditions that apply to the setting
How to Maximize Factory Automation Efficiency with Low Cost Machine Vision
This eGuide illustrates solutions, applications and benefits of machine vision systems.
Wireless Reliability in Harsh Environments
Learn how to increase device reliability in harsh environments and decrease unplanned system downtime.
October 2018
HMI hardware evolution, Data acquisition strategies, Matching motors and drives, Machine vision advice
September 2018
Optimize controls via cloud software, ladder logic simulation, industrial wireless best practices
August 2018
Augmented reality and virtual reality education, autotuning PID control, cybersecurity advice, educating engineers
Edge Computing
This article collection contains several articles on how today's technologies heap benefits onto an edge-computing architecture such as faster computing, better networking, more memory, smarter analytics, cloud-based intelligence, and lower costs.
Data Center Design
Data centers, data closets, edge and cloud computing, co-location facilities, and similar topics are among the fastest-changing in the industry.
IIoT: Machines, Equipment, & Asset Management
Articles in this digital report highlight technologies that enable Industrial Internet of Things, IIoT-related products and strategies.
SIDB

Find and connect with the most suitable service provider for your unique application. Start searching the Global System Integrator Database Now!

October 2018
2018 Product of the Year; Subsurface data methodologies; Digital twins; Well lifecycle data
August 2018
SCADA standardization, capital expenditures, data-driven drilling and execution
June 2018
Machine learning, produced water benefits, progressive cavity pumps
John O. Ayuk, PE, CFSE, PMP, CAP
Automation Engineer; Wood Group
Doug Baker
System Integrator; Cross Integrated Systems Group
Jose S. Vasquez, Jr.
Jose S. Vasquez, Jr.
Fire & Life Safety Engineer; Technip USA Inc.
Data Centers: Impacts of Climate and Cooling Technology
This course focuses on climate analysis, appropriateness of cooling system selection, and combining cooling systems.
Safety First: Arc Flash 101
This course will help identify and reveal electrical hazards and identify the solutions to implementing and maintaining a safe work environment.
Critical Power: Hospital Electrical Systems
This course explains how maintaining power and communication systems through emergency power-generation systems is critical.
click me