Augmented reality enables smarter maintenance using smartphones and glasses

Visualizing in-situ services will make diagnosis and repair actions more intelligent and effective during industrial processing. A plant maintenance technician in the future could be led to the exact location of a mechanical component in need of maintenance, for example, a valve or a field transmitter, by a smartphone handset or “smart glasses.”

By Mika Karaila January 13, 2015

Just imagine that a plant maintenance technician in the future could be led to the exact location of a mechanical component in need of maintenance, for example, a valve or a field transmitter, by a smartphone handset or "smart glasses." Then he could look up the maintenance history of that specific component and perform an in-situ diagnostics test or vibration analysis. If trouble was noted by a cloud-based analysis he could look up device-specific information, consult a repair manual, or call a supplier help center—all hands-free without having to go back a control room or maintenance shop. With this on-the-spot visualization, a repair could be made quickly and surely before a failure happens. This so-called "augmented reality" sounds very futuristic, but really it is just around the corner, as these intelligent maintenance applications are being developed and tested as we speak.

There is a sound and justifiable reasoning behind these developments since the maintenance of a complex and widespread industrial manufacturing or resource processing plant is a daunting, time-consuming, and costly task. Yet it is critical for profitability to ensure that the operation is running at top efficiency with minimal disruptions and no costly failures. Most importantly, worker safety must be ensured. There are thousands of active maintenance points in an industrial complex, including valves and field devices that are required for operating the process efficiently and safely, and mechanical maintenance points that need periodic checking to ensure they are not showing imminent failure patterns.

Any new maintenance tools that can improve the effectiveness of maintenance to achieve higher availability and, at the same time, lower the costs of maintenance are well justified. To achieve high equipment availability, it is vital to detect faults early, schedule maintenance, and carry out repairs before production is impacted. This information needs to be easily accessed and intuitively operated so maintenance tasks and decisions are fast, decisive, and cost-effective. 

Fusion of technologies and maintenance needs

To make process and control device condition monitoring more comprehensive, less-time consuming, and less costly, a new mobile condition analysis and maintenance application is being developed. The application is based on commonly used smartphones as portable user interfaces and wireless communications to portable monitoring devices and data analysis based on cloud computing. In addition to a smartphone interface, the use of smart headgear with integral information display glasses, so-called smart glasses, is being developed as well.

Most significantly, this new way of approaching maintenance is a fusion of various enabling technologies and may be leading to a fusion of previously diverse maintenance needs, mechanical equipment. and process control devices that will have a positive impact on the effectiveness and cost of previously separate disciplines. 

Lower cost per point

In a recent Control Engineering article "Connectivity of things: Wireless for the last 100 m of IoT," Rolf Nilsson says there is huge potential for the last 100 meters connectivity of devices such as industrial condition monitoring equipment. He goes on to say that most of the future growth in wireless Internet connectivity will stem from this area. The qualities of these monitoring devices must be low cost, low power consumption, small size, and high reliability for an industrial setting. In addition, the ability to connect to a variety of easy-to-use interfaces such as smartphones, tablet computers, and even smart glasses is essential. This is to streamline the maintenance actions so many points can be surveyed in a day and services actions taken effectively.

The newly developed application makes the best use of a number of enabling technologies and standards, including emerging magnetic navigation technology, low-power micro-electromechanical (MEMS) machinery condition sensors, low-power wireless Bluetooth communications, and cloud computing access via a Wi-Fi connection to the Internet. The new portable user interface extends the reach of hardwired systems (that are usually comprised of a few hundred monitored points) to thousands of accessible points. The sensors can be placed during a scheduled maintenance round and can be relocated at a later time.

The concept of the smartphone interface is easy to understand. A maintenance worker will be given a daily maintenance task list determined by a computerized maintenance management system (CMMS). The user interface then guides the worker through the complex plant to the device to be inspected. The phone can be used to select necessary sensor measurements directly by a Bluetooth wireless connection. The sensor sends data using Bluetooth to a Wi-Fi gateway into the cloud service. Some intelligent messages can be enacted before data is uploaded into the cloud storage-for instance, low device battery levels or instrument calibration or cleaning requests. This kind of action can be implemented as a simple object access protocol (SOAP) message that is sent to an asset management system (like SAP) or others.

Based on universal standards

To be practical as a data collection and concentration application in an existing mill network with a distributed control system (DCS), fieldbus interfaces, and diverse field devices, a common, universal interface is required. The current mobile maintenance application is being developed with a standard protocol using OPC UA. It can be built up from subsystems with their own OPC UA servers and information can be aggregated to a higher level. The OPC UA itself can be installed on top of different operating systems like Microsoft Windows and Linux, and even on Android. This design provides a secure way to interact with information provided by sensors and devices from different network topologies.

Multiple protocols can be used for transferring condition monitoring data from wireless sensors. The selection is based on available energy, range, and sensor density in the process area. Bluetooth low energy offers the possibility to use smartphones as one access point to data. Other protocols for connecting a large amount of sensors are 6LoWPAN (<1000) and Zigbee (<500).

Maintenance terminal in your pocket or on your eyes

The phone or smart glass interface shows condition monitoring measurement trends from a portable sensor temporarily installed on the casing of the pump, motor, bearing, or other component. This portability is a low-cost way of extending the monitoring capability within a plant. Monitoring and fault detection can now be done on demand, and there is no need to go to a computer terminal since the handset with all the necessary service tasks and communications channels is in your pocket or on your forehead.

The sensor signal analyses in the cloud computing network include sophisticated mathematical techniques like spectral analysis. The device or mechanical component is given a clean bill of health or a specific fault is identified and a service task is initiated. A maintenance worker can initiate the task if a sensor analysis is abnormal or if the cloud determines that a service interval is due. The analysis data is passed along to the cloud for archiving and time trending to see if a problem is getting worse and may need attention in the future. After the action is complete, the worker passes the smartphone past a near field communication (NFC) tag on the portable sensor module. Alternately, a QR code can be read. When the action is complete it is captured in a cloud-based maintenance task database along with a photo, if needed. Subsequent inspections can be scheduled and displayed on the handset when it is due.

Magnetic signature navigation

How does the smartphone or other user interface guide the worker through the complicated plant maze? Global positioning systems (GPS) don’t work well indoors because of the interfering effects of the building structure. Instead, the new navigation technology used in Metso’s application is based on reading a magnetic signature that is unique to a building’s structure. The smartphone comes equipped with the necessary magnetometers. The principle is similar to a natural phenomenon whereby sea turtles and lobsters use magnetic pattern recognition to navigate their way in the ocean. The magnetic signature of the plant interiors can locate a serviceable monitoring point with good precision. There is no more hunting around and wasting time. It also benefits newly hired technicians and third-party maintenance contractors who may not know the plant’s complex layout well. 

Sensor intelligence in the cloud

Portable and low-energy mechanical condition sensing is also a cornerstone of the new mobile maintenance application. These miniscule sensors are based on emerging MEMS technology. The MEMS sensor analyzes transient elastic waves created by sudden redistribution of stress in a material. This is a result of vibration stress patterns. Typically, a condition monitoring point includes vibration and temperature measurements. This capability can be extended to include miniaturized measurements of acoustics (for leak detection), light emissions, humidity, pressure, and magnetic and gyroscopic fluctuations. These capabilities can be turned on and off as needed. The power consumption of these sensors is low, so the battery life of MEMS sensors can be up to one year. The handset senses battery strength and can alert the worker if it should be changed. These communications are based on standard Bluetooth profiles.

The intelligence derived from the raw sensor signals comes from the cloud computing network where the signal spectra and other condition monitoring techniques like envelope analysis are performed. That analysis turns a raw signal into actionable intelligent information. As an example of cloud computing power, you may have seen advertisements that the Lotus Formula 1 racing team is using Microsoft cloud-based software to monitor and analyze the signals from over 200 wireless sensors in its race cars. The objective is to make fast and sure adjustments to win races. On the other hand, a pulp mill maintenance department wants to increase process uptime. That’s a worthy objective as well.

This low-cost and scalable mobile condition monitoring application promises to extend a plant maintenance department’s analysis and service task capability to include many process equipment points that were not feasible before because of the high cost. It also streamlines the management of the regular maintenance route by quickly directing a worker to a maintenance task point and making an intelligent assessment of the component’s condition. This multi-dimensional application will give end users and maintenance managers the capability to integrate live data and equipment status with historical data to predict maintenance needs and plan maintenance activities in the most effective way to optimize plant uptime and minimize costs.

The application is now being tested at a mine site and a power plant in Finland.

Mika Karaila is manager of research programs, Metso Automation. Edited by Joy Chang, Control Engineering,