'At some point, tool wear, material quality, or a component failure will drive the results of the robotic system outside your initial specifications,' says Roy Kok, director of HMI/SCADA product marketing for GE Fanuc's. 'Then, you must be able to monitor, adjust, and capture information so that an operator can view it and coordinate update of the machines. The HMI or SCADA system becomes the application on top of the robotic application to monitor, manipulate, and coordinate tooling effectiveness or machine uptime and downtime. It is involved in scheduling maintenance, and ensuring quality control. The HMI is only one aspect of providing a better view of what's going on with a machine and a facility.'

The need for a rich HMI on the control floor is understood, adds Rich Carpenter, Solutions business leader for GE Fanuc. 'You have to have it.' But it's not all you have to have, Carpenter goes on: 'The front-end HMI performs the initial control, but also hosts a Web server inside to do remote diagnostics. The HMI can be integrating data from lots of different sources. That information can then be transferred to a historian, which becomes a repository for archiving information literally forever, and gives you a base from which to perform analyses.'

Integration of the HMI beyond the robot and throughout the enterprise is not the only area of progress for these systems. Technological advancements that touch on safer operations, portability, and more elaborate displays are poising robotic HMIs to leap to the next level of sophistication on the plant control floor.

Safety concerns are critical. With a robotic HMI, those concerns are tied to monitoring, tracking, troubleshooting, commissioning, and change-management, among other tasks.

'We're seeing a significant increase in the purchase, interest, and standardization of change-management applications,' observes GE Fanuc's Kok. 'There are programs that compare what's running on the equipment versus what is expected to run on the equipment. Who made what changes and why? There are compelling safety reasons for knowing these things. Safety is a key factor, and it is being built into the HMI systems, especially with regard to change control.'

George Schuster, Solutions Development Manager, Rockwell Automation, agrees, noting: 'Panel-mounted robotic HMIs had only a minimal impact on the safety space, but use of portable or handheld HMIs is changing that for several reasons: they integrate the data/diagnostic/monitoring/visualization function with the safety function all in the palm of your hand.

'Once dedicated to specific robots, portable HMIs or 'teach pendants' not only serve the 'robot proper,' but also the process equipment with which the robot is most closely associated.' Continues Schuster: 'It's even going one step further. A single HMI might control multiple robot arms and multiple pieces of process equipment. We're starting to see the portable HMI as the primary low-level maintenance interface.'

He emphasizes, however, that portable HMI is small in size and has a lower level visualization function. 'A tiered approach is required: portable HMI is the one you take into the work cell to make things move; the panel outside the work cell is the high-level advanced diagnostic and visualization tool.'

Progress in portability ties inextricably to safety. Portability has made initialization/commissioning as well as troubleshooting/repair simpler, and also safer. A traditional approach requires a two-person procedure. At least one person enters the work cell as an observer and is exposed to its inherent potential hazards. A second person outside the area controls the machinery, making adjustments from the panel mount HMI. The workers communicate using two-way radios—or even call back and forth—one observing and one adjusting.

Handhelds allow one employee working from a portable HMI—that displays the same screen as the panel-mount HMI—to enter the work cell and make adjustments first-hand. Adjacent to the function he is controlling—instead of merely watching what someone from the outside is doing—he can see if what he intended has been accomplished and directly make modifications.

Rockwell's Schuster elaborates: 'Holding a portable HMI or robot pendant, a single worker controls what is happening as he observes it. The handheld unit can be equipped with a control mechanism that has a three-position enabling device. Only the half-way or middle-point enables power. If the operator either squeezes tightly or lets go entirely, power is cut off. These devices eliminate communication problems and add a safety trigger right on the device. Such measures dramatically improve safety, as well cut labor requirements. What used to take two people now takes one.'

At the moment, says Schuster, the biggest barrier to 'going wireless' is cost. 'More and more portable devices are based on front-office, Windows-based technology. They are more integrated into plant systems. If operators are not tethered to the system, instead of hanging a teach pendant on each machine, you can hang one on each operator's belt. They go around the control floor with a wireless interface, log into each machine, do what needs to be done, disconnect, and move on. Some technology issues exist that aren't fully worked out, but we are well on the way to making that scenario reality.'

ABB's Hanby agrees: 'Portability has had a huge impact on robotic HMIs. The personal computer/PDA market has revolutionized these applications. If you look at our robot pendant, it is really just a PDA. We are leveraging technologies already available in the market. Our next generation of 'teach pendants' will be totally wireless.'

Another area of robotic HMI development is in the quality and quantity of data displayed by these systems. Early on, operators worked with one- and two-line text displays and were typically frustrated trying to find and format useful information. As systems began to leverage PC technology, displays entered a whole new realm. Now, the proliferation of server-based diagnostics is promoting the use of sophisticated graphics in all kinds of systems.

From a maintenance and repair standpoint, a worker faced with a robot run amuck might need to review a procedure before proceeding. Without leaving the area, he can call up a refresher on the screen. Information does not have to reside locally. With access to a high-level diagnostic server-based system, what is needed is instantly available in a few keystrokes, whether the information be in a brief MPG-format movie, or a complete repair manual.

Says Rockwell's Schuster, 'This expedites the repair process and improves safety because it's done right the first time. We're seeing some systems that allow an operator to view a machine as it is, then overlay an enhanced reality image on top of it. These kinds of technologies are a ways down the road, but integrated systems are moving us in that direction.'

One example of the importance of graphical displays in robotic HMIs is evident in the incorporation of hardware and software into a single system. Iconics and Kuka Robotics have joined forces to integrate Iconics Genesis32 suite into the latter's robotic systems. Oliver Grüner, OEM account manager for Kuka says the move was made for a number of reasons, among them the software's ability to display information in several languages. However, it is the package's animation features that drive the partnership.

'The program includes unique features for doing animation,' notes Grüner, including an ability to develop animated components that can be reused repeatedly, piece by piece, whenever needed in various systems. 'No scripting is involved. You simply trace how you want a robot arm to move on the screen, and that's all that is needed. We try to build features into the system so that the end-user does not have to script or program,' says Grüner. (See 'Animated graphics...' sidebar for more on HMI graphics.)

What can the controls and automation field expect from robotic HMIs of the future? According to ABB's Hanby, the next frontier is in sensory systems. 'We're making robots simpler to program and interact with by allowing robots to know more about their environment. When a human assembles something, he uses tactile feedback. We're working with educational institutions to study how people think and assemble things. We're employing this information in the form of algorithms in the robot using sensory systems and we're working to incorporate those sensory systems so that robots can start thinking, in some respects, like human beings. Not intellectual processing, of course, but lower-level tactile tasks.

If we can parallel how a human thinks, the user can apply the robot more easily ... because the machine will think like he does. Eventually, the HMI will disappear. It will part of the robot.'

Dr. Paolo Pirjanian, chief scientist at Evolution Robotics, agrees. He describes relationships with robots as HRI, or human-robot interaction. 'For robotics to be successful, it must be intuitive,' he points out. 'Robots are complex machines. The average person needs a simpler interface.'

He cites advancements in computer vision as important to robotic system interaction. 'A robot so equipped can match pictures in its memory with what it sees and respond, by itself, using the database it has compiled,' Pirjanian says 'It can continually expand its database and, in a sense, 'learn' its environment.'

Perhaps ABB's Hanby isn't far off when he quips, 'If the robot can figure out its environment, what do you need an HMI for?'