HMIs in Robotics

In complex, sophisticated manufacturing environments where robots are increasingly deployed, it is imperative to know at every moment what your robots are doing now as well as what they've done in the past. Whether the process involves initialization and commissioning, operations, or repair and maintenance, abundant information must be gathered, analyzed, and available at a moment's notice.


At A Glance


  • HMIs play key role

  • Robots increasingly intelligent

  • Integrated systems

  • Safety, portability tied

  • Sophisticated graphics

  • Future: more, better, simpler

Animated graphics add precision to automated washing system
Robots, software team to smooth machining process

In complex, sophisticated manufacturing environments where robots are increasingly deployed, it is imperative to know at every moment what your robots are doing now as well as what they've done in the past. Whether the process involves initialization and commissioning, operations, or repair and maintenance, abundant information must be gathered, analyzed, and available at a moment's notice.


Most HMI systems now available make it possible to do just that. That's why robots and the HMIs that track and control them are found wherever precise, repetitive, or hostile-environment tasks must be performed.

No robot is an island

While robots typically work at individual tasks, none works alone. 'The robot is not an island,' says David Hanby, VP of software development, North America, ABB Inc. 'The [robotic] system must interface with other plant systems.'


Today's robotic systems are integral parts of the plant enterprise. To perform effectively, they must be viewed and treated as such. Integrating long-term monitoring and analysis with robotic operation is crucial.

'At some point, tool wear, material quality, or a component failure will drive the results of the robotic system outside your initial specifications,' says Roy Kok, director of HMI/SCADA product marketing for GE Fanuc's. 'Then, you must be able to monitor, adjust, and capture information so that an operator can view it and coordinate update of the machines. The HMI or SCADA system becomes the application on top of the robotic application to monitor, manipulate, and coordinate tooling effectiveness or machine uptime and downtime. It is involved in scheduling maintenance, and ensuring quality control. The HMI is only one aspect of providing a better view of what's going on with a machine and a facility.'

The need for a rich HMI on the control floor is understood, adds Rich Carpenter, Solutions business leader for GE Fanuc. 'You have to have it.' But it's not all you have to have, Carpenter goes on: 'The front-end HMI performs the initial control, but also hosts a Web server inside to do remote diagnostics. The HMI can be integrating data from lots of different sources. That information can then be transferred to a historian, which becomes a repository for archiving information literally forever, and gives you a base from which to perform analyses.'

Integration of the HMI beyond the robot and throughout the enterprise is not the only area of progress for these systems. Technological advancements that touch on safer operations, portability, and more elaborate displays are poising robotic HMIs to leap to the next level of sophistication on the plant control floor.

Handheld safety

Safety concerns are critical. With a robotic HMI, those concerns are tied to monitoring, tracking, troubleshooting, commissioning, and change-management, among other tasks.

'We're seeing a significant increase in the purchase, interest, and standardization of change-management applications,' observes GE Fanuc's Kok. 'There are programs that compare what's running on the equipment versus what is expected to run on the equipment. Who made what changes and why? There are compelling safety reasons for knowing these things. Safety is a key factor, and it is being built into the HMI systems, especially with regard to change control.'

George Schuster, Solutions Development Manager, Rockwell Automation, agrees, noting: 'Panel-mounted robotic HMIs had only a minimal impact on the safety space, but use of portable or handheld HMIs is changing that for several reasons: they integrate the data/diagnostic/monitoring/visualization function with the safety function all in the palm of your hand.

'Once dedicated to specific robots, portable HMIs or 'teach pendants' not only serve the 'robot proper,' but also the process equipment with which the robot is most closely associated.' Continues Schuster: 'It's even going one step further. A single HMI might control multiple robot arms and multiple pieces of process equipment. We're starting to see the portable HMI as the primary low-level maintenance interface.'

He emphasizes, however, that portable HMI is small in size and has a lower level visualization function. 'A tiered approach is required: portable HMI is the one you take into the work cell to make things move; the panel outside the work cell is the high-level advanced diagnostic and visualization tool.'

Robots unplugged

Progress in portability ties inextricably to safety. Portability has made initialization/commissioning as well as troubleshooting/repair simpler, and also safer. A traditional approach requires a two-person procedure. At least one person enters the work cell as an observer and is exposed to its inherent potential hazards. A second person outside the area controls the machinery, making adjustments from the panel mount HMI. The workers communicate using two-way radios—or even call back and forth—one observing and one adjusting.

Handhelds allow one employee working from a portable HMI—that displays the same screen as the panel-mount HMI—to enter the work cell and make adjustments first-hand. Adjacent to the function he is controlling—instead of merely watching what someone from the outside is doing—he can see if what he intended has been accomplished and directly make modifications.

Rockwell's Schuster elaborates: 'Holding a portable HMI or robot pendant, a single worker controls what is happening as he observes it. The handheld unit can be equipped with a control mechanism that has a three-position enabling device. Only the half-way or middle-point enables power. If the operator either squeezes tightly or lets go entirely, power is cut off. These devices eliminate communication problems and add a safety trigger right on the device. Such measures dramatically improve safety, as well cut labor requirements. What used to take two people now takes one.'

At the moment, says Schuster, the biggest barrier to 'going wireless' is cost. 'More and more portable devices are based on front-office, Windows-based technology. They are more integrated into plant systems. If operators are not tethered to the system, instead of hanging a teach pendant on each machine, you can hang one on each operator's belt. They go around the control floor with a wireless interface, log into each machine, do what needs to be done, disconnect, and move on. Some technology issues exist that aren't fully worked out, but we are well on the way to making that scenario reality.'

ABB's Hanby agrees: 'Portability has had a huge impact on robotic HMIs. The personal computer/PDA market has revolutionized these applications. If you look at our robot pendant, it is really just a PDA. We are leveraging technologies already available in the market. Our next generation of 'teach pendants' will be totally wireless.'

Put on your 3-D glasses

Another area of robotic HMI development is in the quality and quantity of data displayed by these systems. Early on, operators worked with one- and two-line text displays and were typically frustrated trying to find and format useful information. As systems began to leverage PC technology, displays entered a whole new realm. Now, the proliferation of server-based diagnostics is promoting the use of sophisticated graphics in all kinds of systems.

From a maintenance and repair standpoint, a worker faced with a robot run amuck might need to review a procedure before proceeding. Without leaving the area, he can call up a refresher on the screen. Information does not have to reside locally. With access to a high-level diagnostic server-based system, what is needed is instantly available in a few keystrokes, whether the information be in a brief MPG-format movie, or a complete repair manual.

Says Rockwell's Schuster, 'This expedites the repair process and improves safety because it's done right the first time. We're seeing some systems that allow an operator to view a machine as it is, then overlay an enhanced reality image on top of it. These kinds of technologies are a ways down the road, but integrated systems are moving us in that direction.'

One example of the importance of graphical displays in robotic HMIs is evident in the incorporation of hardware and software into a single system. Iconics and Kuka Robotics have joined forces to integrate Iconics Genesis32 suite into the latter's robotic systems. Oliver Grüner, OEM account manager for Kuka says the move was made for a number of reasons, among them the software's ability to display information in several languages. However, it is the package's animation features that drive the partnership.

'The program includes unique features for doing animation,' notes Grüner, including an ability to develop animated components that can be reused repeatedly, piece by piece, whenever needed in various systems. 'No scripting is involved. You simply trace how you want a robot arm to move on the screen, and that's all that is needed. We try to build features into the system so that the end-user does not have to script or program,' says Grüner. (See 'Animated graphics...' sidebar for more on HMI graphics.)

The next frontier

What can the controls and automation field expect from robotic HMIs of the future? According to ABB's Hanby, the next frontier is in sensory systems. 'We're making robots simpler to program and interact with by allowing robots to know more about their environment. When a human assembles something, he uses tactile feedback. We're working with educational institutions to study how people think and assemble things. We're employing this information in the form of algorithms in the robot using sensory systems and we're working to incorporate those sensory systems so that robots can start thinking, in some respects, like human beings. Not intellectual processing, of course, but lower-level tactile tasks.

If we can parallel how a human thinks, the user can apply the robot more easily ... because the machine will think like he does. Eventually, the HMI will disappear. It will part of the robot.'

Dr. Paolo Pirjanian, chief scientist at Evolution Robotics, agrees. He describes relationships with robots as HRI, or human-robot interaction. 'For robotics to be successful, it must be intuitive,' he points out. 'Robots are complex machines. The average person needs a simpler interface.'

He cites advancements in computer vision as important to robotic system interaction. 'A robot so equipped can match pictures in its memory with what it sees and respond, by itself, using the database it has compiled,' Pirjanian says 'It can continually expand its database and, in a sense, 'learn' its environment.'

Perhaps ABB's Hanby isn't far off when he quips, 'If the robot can figure out its environment, what do you need an HMI for?'

For more products, visit For integrators, go to For more information on this topic, search the Control Engineering Web site at and visit the company Web sites listed below.

Animated graphics add precision to automated washing system

An automated train washing operation in Germany is monitored and controlled with Iconics' Genesis32 Enterprise Edition. The system cleans any type and shape of train. Operators enter train specifications; information on specific trains can be stored for future use.

A 'learning mode' lets the system make cleaning process adjustments. Parameters captured during the learning mode are downloaded to PLCs to adjust sprayers and brushes to ensure thorough washing. GraphWorX32 graphical design and runtime environment for Genesis32 lets all operations be viewed with 3-D graphics; its animation feature traces the path each train will take through the washer. Alarms, handled by the AlarmWorX32 portion of the package, can be acknowledged from anywhere in the system and are handled by a Microsoft SQL server. Trending data, both real-time and historical, are maintained by the TrendWorX32 module. A Phoenix Contact Interbus OPC Server provides OPC connectivity.

The application was designed and developed by S.A.H. Heyd group for Deutsche Bahn AG railways, one of Europe's largest transportation providers.

(Photo courtesy of Iconics)

Robots, software team to smooth machining process

Machining Enterprises Inc., Saginaw, MI, performs a cubing operation on blocks and heads for new sport utility vehicles engines. The operation involves machining various surfaces. Cubing validates the castings before they are shipped.

With the help of robots from Fanuc Robotics, linked to GE Fanuc's Cimplicity Machine Edition software running on Control Station NTs, the company is able to identify problems early in the production process. Three machining work cells on the plant floor each contain four Niigata horizontal machining centers. Two machines that run blocks and two that run heads are situated side-by-side. A pair of Fanuc M-710iT six-axis overhead rail-mounted robots services each cell.

'Typically, one robot services two machines, but two of our three cells are flexible, so one robot can perform two different tasks, such as manufacturing blocks and heads, side by side,' explains Brian Verzinski, manufacturing process engineer. A robot traverses between two centers, first working on one part, then stopping to change its end-of-arm tool (EOAT) and work on the other part, then stops and changes the tool again.

A single Control Station NT controls the two flexible cells. Each cell is equipped with two additional stations that serve as operator interfaces. A third cell uses one station for control and one as an operator interface. Software transmits information from the machining centers to the robot.

'A number of conditions must be met before a robot can execute a command,' says Verzinski. 'For example, determining whether a part is present, whether the light curtain is clear, and whether the robot has completed the previous task.'

Each controller has two separate I/O channels. One channel of Series 90-30 I/O interfaces via Ethernet with the robots, and one channel of DeviceNet interfaces with conveyors using VersaMax I/O. Conveyor transfer lines shuttle parts between the machining centers.

Speed was a major concern. The goal was to perform a unique 'A' and 'B' load on each machining center within the machining cycle time. Prior to the equipment build, a computer simulation model was developed, then proven in a trial run at Fanuc.

Installing and commissioning the system went 'extremely well,' says Verzinski. 'We wanted a four-hour fault-free run the first time out, and we got it.'

Given the complexity of the manufacturing environment, the graphical interface has a relatively simple screen layout to simplify the operation. 'It takes about a month for new employees to learn how to run it, but that's reasonable based on the application,' Verzinski says.

'This is a complex robotic operation,' adds Verzinski, but the overhead rail-mounted robots save floor space, machine tool access, and investment because one robot can service two machining centers.

Information for this section was provided by GE Fanuc.

No comments
The Engineers' Choice Awards highlight some of the best new control, instrumentation and automation products as chosen by...
The System Integrator Giants program lists the top 100 system integrators among companies listed in CFE Media's Global System Integrator Database.
Each year, a panel of Control Engineering and Plant Engineering editors and industry expert judges select the System Integrator of the Year Award winners in three categories.
This eGuide illustrates solutions, applications and benefits of machine vision systems.
Learn how to increase device reliability in harsh environments and decrease unplanned system downtime.
This eGuide contains a series of articles and videos that considers theoretical and practical; immediate needs and a look into the future.
Controller programming; Safety networks; Enclosure design; Power quality; Safety integrity levels; Increasing process efficiency
Additive manufacturing benefits; HMI and sensor tips; System integrator advice; Innovations from the industry
Robotic safety, collaboration, standards; DCS migration tips; IT/OT convergence; 2017 Control Engineering Salary and Career Survey
Featured articles highlight technologies that enable the Industrial Internet of Things, IIoT-related products and strategies to get data more easily to the user.
This article collection contains several articles on how automation and controls are helping human-machine interface (HMI) hardware and software advance.
This digital report will explore several aspects of how IIoT will transform manufacturing in the coming years.

Find and connect with the most suitable service provider for your unique application. Start searching the Global System Integrator Database Now!

Infrastructure for natural gas expansion; Artificial lift methods; Disruptive technology and fugitive gas emissions
Mobility as the means to offshore innovation; Preventing another Deepwater Horizon; ROVs as subsea robots; SCADA and the radio spectrum
Future of oil and gas projects; Reservoir models; The importance of SCADA to oil and gas
Automation Engineer; Wood Group
System Integrator; Cross Integrated Systems Group
Jose S. Vasquez, Jr.
Fire & Life Safety Engineer; Technip USA Inc.
This course focuses on climate analysis, appropriateness of cooling system selection, and combining cooling systems.
This course will help identify and reveal electrical hazards and identify the solutions to implementing and maintaining a safe work environment.
This course explains how maintaining power and communication systems through emergency power-generation systems is critical.
click me