Focus on avoidance, not on the risk itself

When riding my bicycle, wearing my bright yellow jersey, most drivers will make sure that they pass safely, but some will focus on my yellow jersey and slowly drift into my lane. These people are focusing on the risk, not focusing on the actions needed to avoid the risk.

The advice to focus on the actions of avoidance and not the risk also applies to manufacturing and automation systems. Multiple risks to manufacturing computer systems include: loss of electrical power, poor quality electricity, hardware failures, software failures, viruses, hackers, and environmental hazards (flood, fire, wind). While these risks can be real, remember that multiple risks can be addressed if you focus on actions needed to avoid the risk.

What to avoid

It may be impossible to enumerate all of the external threats to your manufacturing systems, but simple actions can be used to minimize the risks. For starters, only allow outside access through firewalls and tightly constrained proxy servers with a DMZ (demilitarized zone). A DMZ isolates critical systems from other corporate systems and Internet access. These actions do not mean you can avoid external risks, but they will reduce your expose to these risks.

Physical risk can be avoided through a well thought out and implemented backup protection plan. Electrical power risks can be avoided through use of UPS (uninterruptable power supply) systems and industrial grade surge protectors. UPS systems should be connected to server systems and workstations through USB ports to allow orderly shutdown and preservation of critical data. UPS system tests must be added to normal maintenance tests. UPS batteries last only a few years so they must be regularly tested and replaced. The worst UPS system to have is one that doesn’t work when needed.

Hardware failure risk can be reduced by focusing on hot, warm, or cold standby systems. The decision of which standby method to use should be based on the criticality of control and acceptable business downtime. Many production facilities lose hundreds of thousands, if not millions, of dollars for each day down. A small fraction of that amount spent on avoidance can reduce the risk. Hot standby systems are the most expensive, but they have switch-over times of milliseconds to seconds. Warm standby systems are typically implemented using ready-to-run copies of the primary systems and have switch-over times of minutes. Cold standby systems have the same hardware configuration as the primary but no software. When a primary system fails, the standby is loaded with a ghost image of the primary system. Usually one standby can cover multiple primary systems. Cold standby switch-over times are usually measured in hours.

Environmental hazards can be avoided by having an alternate backup site. The alternate site should be physically separated and isolated from the primary site. It should be in a separate building, on higher floors to avoid floods or lower floors to avoid wind, with separate power lines and network communications. Alternate sites should have the capacity to run for extended periods of time, even if at a reduced performance.

Software risks can be reduced by a philosophy of “test-first, test-always” for patches and updates. A common cause of software failure is from patches and updates that change undocumented features in application software. Cold standby systems are often used to test patches and updates, reducing the cost of software risk avoidance.

With these tips in mind, remember: It is important to be aware of the risks to manufacturing and automation systems, but it is more important to focus on the actions required to avoid the risks.

Author Information

Dennis Brandl is president of BR&L Consulting in Cary, NC, www.brlconsulting.com . His firm focuses on manufacturing IT. Contact Dennis at dbrandl@brlconsulting.com .

Do you have experience and expertise with the topics mentioned in this content? You should consider contributing to our CFE Media editorial team and getting the recognition you and your company deserve. Click here to start this process.