Fog computing for industrial automation

How to develop a secure, distributed automation architecture in a data-driven world: Two examples and five advantages of fog computing are highlighted.

By Charles Byers, Cisco Systems and OpenFog Consortium Technical Committee March 8, 2018

The manufacturing industry is experiencing substantial benefits as industrial operators use the Industrial Internet of Things (IIoT) to automate systems, deploy sensors to measure, monitor, and analyze data, improve efficiencies, and increase revenue opportunities for manufacturing operations. Using eight pillars of a fog computing architecture can help.

The amount of data from these newly-connected plants can be measured in the petabytes (1 million gigabytes): Millions of streaming, connected sensors on industrial control systems (ICSs), dozens of autonomous drones, industrial robots, video surveillance cameras covering plants, and so on.

Traditional information technology (IT) approaches to operational technology (OT) environments cannot keep up with the necessary volume, latency, mobility, reliability, security, privacy, and network bandwidth challenges in controlled, supplier-connected, or rugged operational environments. It’s time for a new architectural approach to allow IIoT to reach its potential with fog computing. 

Defining fog computing

Fog computing is designed for data-dense, high-performance computing, high-stakes environments. Fog is an emerging, distributed architecture that bridges the continuum between cloud and connected devices that doesn’t require persistent cloud connectivity in the field and factory. Fog works by selectively moving compute, storage, communication, control, and decision making closer to IoT sensors and actuators, where the data is generated and used. It augments, not replaces, investments in the cloud to enable an efficient, cost-effective, secure, and constructive use of the IIoT in manufacturing environments.

Fog is sometimes referred to as edge computing, but there are key differences. Fog is a superset of edge functionality. The fog architecture pools the resources and data sources between devices residing at the edge in north-south (cloud-to-sensor), east-west (function-to-function or peer-to-peer) hierarchies working with the cloud for maximum efficiency. Edge computing tends to be limited to a small number of north-south layers often associated with simple protocol gateway functions.

Fog nodes are foundational elements of the fog architecture. A fog node is any device that provides computational, networking, storage, and acceleration elements of the fog architecture. Examples include industrial controllers, switches, routers, embedded servers, sophisticated gateways, programmable logic controllers (PLCs), and intelligent IoT endpoints such as video surveillance cameras. 

Fog architecture benefits for a factory

Factories can become more connected and can take advantage of streaming data through a layer of fog nodes. A fog node at the lower level of the hierarchy—for example, located on an individual machine—can be connected to a set of local sensors and actuators so it can analyze the data, interpret an anomaly, and then if authorized, could autonomously react and compensate for the problem or fix the issue. Alternatively, the fog node can send the appropriate requests for service higher up the fog hierarchy to more skilled technical resources, machine learning capabilities, or a maintenance service provider.

If the situation requires real-time decision-making—for example, shutting down equipment before damage occurs, or adjusting critical process parameters—fog nodes can provide millisecond-level latency analysis and action. The manufacturer doesn’t have to route this real-time decision making through the cloud data center. This helps avoid potential latency issues, queue delays, or network/server downtime that could result in industrial accidents, reduced production efficiency, or poor product quality.

In the factory, fog nodes higher up in the hierarchy can have a broader perspective on industrial processes. They can add more functionality such as visualization of production line operation, monitoring the status of malfunctioning machines, tuning of production parameters, modification of production planning, ordering supplies, and sending alerts to the appropriate people. 

Fog architecture for an oil pipeline

To illustrate how fog computing works in more rugged environments, consider an oil pipeline with pressure and flow sensors and control valves and pumps. Traditionally, remote sensor readings are transmitted to the cloud using expensive satellite links for data analysis to detect abnormal conditions. The cloud would send commands back to the operator to adjust valve positions and so on.

This approach is suboptimal because network bandwidth is very expensive and connectivity can go down—especially in severe weather. Transmitted data may also be susceptible to hackers and the round-trip latency is too long (hundreds of milliseconds), which causes slower reaction time to react to critical incidents.

With the addition of fog computing, a hierarchy of local fog nodes is placed near the pipeline to connect to sensors and actuators with inexpensive and fast local networking facilities. The nodes add additional security, lessening the opportunity for breaches, and can be given the authority to react to abnormal conditions in milliseconds, quickly closing valves to greatly reduce the severity of spills. The fog nodes run over wireline, optical, and wireless networks and also inside these networks, making it ideally suited for connection to industrial elements based on wired supervisory control and data acquisition (SCADA) systems, OPC UA interfaces, Modbus, etc. Analytics on the fog node at the local site reduces demand of the bandwidth to the cloud, keeping overall costs low.

Balancing control between the cloud and the fog produces a better outcome across the business (costs, control, safety and security). Moving most of the decision-making functions to the fog—and using the cloud to occasionally report status or receive commands or updates—creates a superior control system.

Five advantages of fog computing

Security, cognition, agility, latency, and efficiency (SCALE) can be described as fog computing’s major advantages. While selected IIoT equipment and technologies address specific aspects, fog computing addresses the full SCALE of requirements.

1. Security: Traditional manufacturing security measures focuses on providing perimeter-based protection for threat detection to industrial control systems. New control requests are redirected to the cloud for authentication and authorization processing. Hardware and software updates are typically installed during a system’s scheduled down time. If threats breach the firewall, the common response is to manually take the system offline to be quarantined and cleaned, which can bring down an entire factory. These processes are no longer adequate today.

With fog computing, security functions locally—not in the cloud—use the same corporate IT policy, controls, and procedures. Most fog nodes include a hardware root of trust, which serves as the basis for a chain of trust from the lowest level sensor and actuator up the fog hierarchy to the cloud. Traffic is monitored from the internet into the distributed fog network, which uses machine learning to detect unusual activity in the local context to detect a probable attack.

If a threat is detected, fog nodes act as gateways by blocking traffic from the attackers and protecting the critical factory network. Highly-sensitive data can be processed locally without leaving the manufacturing plant. The fog architecture also provides a distributed layer of cognitive capabilities over low-cost IoT endpoints to enable cost-effective network-wide blockchain deployments even if some network elements lack the computational resources for full blockchain implementation.

2. Cognition: The fog architecture determines the optimal places to carry out the computing, storage and control functions along the cloud-to-thing continuum. Decisions can be made on an IoT endpoint device or via a nearby fog node, avoiding the need to transport the data solely to the cloud. Data is processed and analyzed locally, autonomously, and efficiently. A filtered version of the information is sent through the fog hierarchy to the headquarters’ data center for analysis. This supports future planning and longer-term improvements.

Smart sensors can make autonomous decisions and trade-offs in regards to manufacturing execution. Multiple connected machines can communicate within production environments and learn from their decisions, improving performance over time.

In particular, the fog architecture can reproduce the skills of experienced operators through the creation of a "digital twin," which uses data collected from sensors on production tools. The digital twin is a digital replica of physical assets, processes and systems that provides both the elements and the dynamics of how an IoT device operates throughout its lifecycle. One operator can monitor and manage multiple devices in multiple places—even in the past, present, and future—through digital twin simulation.

This enables the fog-based infrastructure to learn from past history, provide real-time control of current production processes, and extrapolate the production parameters that may be needed for future production runs, all in one integrated system. Digital twins implemented in the fog layers permit operators to simulate process parameter changes digitally on a system very close in response to the real industrial plant without the danger of mistakes impacting product quality, damaging production equipment, or causing unsafe conditions.

3. Agility: Production environments can fluctuate in regards to the volume of data and the volume of output. Through a hierarchy of fog nodes, fog can manage inconsistent demands on the system to distribute loads to underutilized machines. Fog nodes’ hierarchical structure form dynamic groups to exchange information for efficient collaboration. For example, a temporary shortage of production capacity by one factory can be shifted to another company’s idle asset equipment.

The ability to dynamically change also helps to orchestrate and control information between machines. For example, machine 1 drills a hole in a sheet of metal, into which machine 2 will subsequently insert a screw. If the hole made by machine 1 deviates from the specified position by more than a certain amount, the product identification and information about the deviation is sent to machine 2, which then auto adjusts, preventing a defective product.

Fog also facilitates richer, software-based production environments. As factories move to customized, lower volume-higher mix production schedules, the ability to reprogram fog nodes up the hierarchy enables the digital enterprise to keep up.

4. Latency: Many industrial control systems require end-to-end latencies within sub-milliseconds, falling outside the range for mainstream cloud services. In a factory, 10 μsec order latency may be required to prevent manufacturing line shutdowns, avert accidents, restore electrical service or correct manufacturing errors. Fog reduces latency since it removes the round-trip time lag between production line sensors, to the cloud, and back to production line actuators. This latency includes transport latency to reach the cloud (over wireless or fiber facilities), queueing delays, and cloud server delays—and could be over 100 milliseconds round trip even in well-designed cloud networks. The local fog node can react and make decisions within sub-milliseconds.

In particular, fog can help robotics and drones perform as intended. Robotics and drones require a complex sequence of rapid movements. Control of these movements must be performed within milliseconds—or microseconds in the case of a motor drive control—for safety and precision. If too much delay is introduced into many classes of control systems, control of the physical plant can become unstable. Fog represents a system with all the versatility benefits of the cloud, but with latencies to help ensure control system stability.

5. Efficiency: Interoperability is the key to operational efficiency. As industrial systems evolve from purpose-built and discrete systems to software-defined and modularized systems, manufacturers are realizing there is insufficient system interoperability due to the complexity of connecting systems and sensors, which use different protocols and communication methods.

Fog nodes can work as interpreters or protocol gateways for these legacy systems. Fog nodes collect and normalize data from different formats and protocols across equipment sensors as well as the connections between the sensors and systems, allowing the sensors and systems to easily connect without being aware of the different access methods of each system.

Fog computing also promotes system efficiencies by offloading peak capacity on available machines. In each factory, fog nodes collect, aggregate, and analyze the necessary data related to general production capacity. The virtual platform dynamically interconnects fog nodes between different factories. The platform creates a virtual group of fog nodes trying to produce consistent product, regardless of the specific factory they are working in. The data collected by the fog nodes are sent back to the virtual platform for decision-making on resource utilization across factories. 

Predictive and proactive maintenance

Predictive and proactive maintenance were touted as one of the biggest beneficiaries of IIoT. In reality, this area has been slow to materialize due to the complexities of sharing different data formats and protocols. There are also concerns about releasing sensitive information outside the factory to third-party maintenance providers.

Fog architecture provides the layer of interoperability for systems to ensure efficient communications and actions between factory systems and maintenance. Fog computing helps identify equipment malfunctions before they happen through an aggregated process. The fog node located on or near an individual machine is connected to an actuator for faster resolution, as was described in the smart factory scenario above.

Fog nodes automatically select the most appropriate communication routes to equipment manufacturers and maintenance providers and securely send data to the maintenance system in real time.

If the maintenance system detects a sign of failure once it analyzes the received data, the inventory is immediately checked for replacement parts and schedules are adjusted with little impact on production. Small deviations from normal sensor readings could indicate an impending failure and enables the system to repair it before it a catastrophic failure. To keep factory data confidential, fog nodes autonomously determine the content of data and send the appropriate data to only the required systems only. Using data classification, encryption, and virtual private networks (VPNs) can provide even more secure communication. This greatly reduces the danger of unintended cross-disclosure of proprietary information in either direction. 

Eight fog computing architecture pillars

The OpenFog Consortium has created the framework for an open, interoperable fog architecture based on eight high-level attributes called the pillars. These eight pillars are security, scalability, openness, autonomy, reliability, agility, hierarchical organization, and programmability. Here’s how they map into automation environments.

1. Security: As previously described, security is critical to fog environments. Fog allows production systems to securely transmit and act upon data in an end-to-end computing environment. Things-to-fog (T2F), fog-to-fog (F2F) and fog-to-cloud (F2C) connections can be established dynamically across various applications.

2. Scalability: By processing most information locally, fog computing can reduce the amount of data needed to transmit from a factory to the cloud. This improves cost and bandwidth performance to production resources and third-party providers. Computational capacity, network bandwidth, and the fog networks’ storage size can be dynamically scaled to match demand.

3. Open: The OpenFog interoperable architecture enables transparent resource discovery and sharing via open application programming interfaces (APIs). APIs also enable the factory’s production equipment to connect to remote maintenance service providers and other partners.

4. Autonomy: The autonomy provided by fog computing allows the manufacturer to perform designated actions even when there is limited or no communication with the data center, including resource sharing with other factories. This can reduce the number of stoppages on the assembly line through early detection of possible failures and predictive maintenance. Critical systems can continue to function even if the cloud is unreachable or overloaded.

5. Reliability/availability/serviceability (RAS): High reliability, availability, and serviceability of the fog nodes are designed to operate in rugged, mission-critical production environments. These attributes help in remote maintenance and predictive maintenance functions, and speed along any required repairs.

6. Agility: Fog computing allows quick localized and intelligent decision-making within a fog system. A minor failure in the factory’s production equipment can be detected and addressed immediately and manufacturing lines can rapidly adapt to new requirements. Agility also enables predictive maintenance, which reduces factory downtime.

7. Hierarchy: The OpenFog architecture allows for device or machine-to-fog, F2F, and F2C operations on and off the manufacturing premises. It also allows hybrid and multiple services to be run on fog node and cloud. Monitoring and control, operational support and business support for manufacturing can be implemented in a dynamic and flexible hierarchy of multiple layers of fog nodes, where each component of a factory’s control system is run at exactly the optimal level of the fog hierarchy.

8. Programmability: Factories can become more efficient by reallocating and repurposing resources based on business needs. Fog-based programmability can enable dynamic change of production lines and factory equipment while maintaining overall production efficiency. It can also create dynamic value chains and analyze the data on-site instead of sending it to the cloud.

Today’s connected production environments have a lot to gain by deploying IIoT technologies. Fundamental challenges remain as incompatible systems and strained networks seek to handle data volume, velocity, and security. Fog computing provides open, interoperable, and secure architectures to enable businesses to optimize the opportunities from this sensor-driven world. While fog computing is in its earliest stages, there are tremendous opportunities for manufacturers and process engineers looking to help shape the manufacturing world through this distributed, open architecture approach.

Charles Byers is a principal engineer at Cisco Systems and co-chair of the OpenFog Consortium technical committee. Edited by Emily Guenther, associate content manager, CFE Media, eguenther@cfemedia.com.

MORE ANSWERS

KEYWORD: Fog computing

Defining fog computing

Understanding the architecture of fog computing

Predictive maintenance with fog nodes.

Consider this:

How can fog computing benefit your plant’s approach to automation?

ONLINE

Available for download here. 

About the author

Charles Byers is a principal engineer and platform architect with Cisco’s Corporate Strategic Innovation Group. He works on the architecture and implementation of Fog Computing platforms, media processing systems, and the Internet of Everything. Before joining Cisco Systems, he was a Bell Labs Fellow at Alcatel-Lucent. During his 31 years in the telecommunications networking industry, he has made significant contributions in areas including voice switching, broadband access, converged networks, VoIP, multimedia, video, modular platforms and IoT. He has also been a leader in several standards bodies, including serving as a founding member of PICMG’s AdvancedTCA, AdvancedMC, and MicroTCA subcommittees, and currently serves as co-chair of the Technical Committee and the Architecture Framework Working Group of the OpenFog Consortium. Byers received his B.S. in Electrical and Computer Engineering and an M.S. in Electrical Engineering from the University of Wisconsin, Madison. He holds 66 U.S. patents.