Machine vision’s next frontier: Facial recognition

Starting in the 1990s, engineers started to teach computers to identify faces in photographs. Can machine control via eye roll be far behind? Such vision software advances will help site security, manufacturing quality and, perhaps, future human-machine interface applications. Images show how it operates differently than most industrial machine vision applications. Here's how it works.

By Peter Welander Control Engineering May 5, 2009

Starting in the 1990s, engineers started to teach computers to identify faces in photographs. Since then, the capability has moved from simple face detection to face recognition and even attribute estimation. Can machine control via eye roll be far behind?

Omron has developed a group of facial recognition products under the name Okao Vision , borrowing the Japanese word for “face.”
Machine Vision, May ’09, Control Engineering

Other articles in the May 2009 Control Engineering North American print edition supplement:
– 415 Parts Seen
– Motion and Vision Combine to Detect Flaws
– Intelligent Vision Stops Bypass of Quality Control

Imagine someone wants to enter a manufacturing plant through a remote entrance that requires both a smart ID card and PIN to open the gate. Due to some careful social engineering, the intruder has stolen a card and obtained the right PIN, but the gate remains locked because a computer vision system has looked at the intruder and realized that he is not the person to whom the card belongs.

A talkative teenage driver is an accident waiting to happen because he or she tends to look away from the road while chatting with friends in the car. If the car is moving and the driver turns to look at a passenger, imagine if the car beeps and says, “Watch where you’re going.”

Such applications are becoming more practica l and commonplace thanks to growing developments of facial recognition platforms, a spin-off from more traditional machine vision applications.

A basic capability allows a computer to look at a person’s face and compare that image to one in a database, thereby verifying the identity of the individual standing at the gate. That has become a fairly common application. But current technology can go much farther. Even medium-priced digital cameras now have the ability to recognize when subjects are smiling. When in “smile shutter” mode, a new Sony camera automatically takes a picture only when the subject is smiling. It evaluates mouth shape, visibility of teeth, position of cheek bones, and eye shape and activates the shutter only when the proper criteria are met. It has a harder time if the individual is wearing sunglasses.

More sophisticated developments allow the system to look at an individual and make some conclusions about that person, and even his or her state of mind. Omron has developed a group of facial recognition products under the name Okao Vision , borrowing the Japanese word for “face.”

Omron began by simply finding faces in an image but that was only a start. “From there it got more complicated,” says Sarah Hall, corporate communications for Omron. “We found ways to determine gender and age in the last couple of years. It can tell what direction someone is looking, and if that person has turned to the side. This is useful in an automotive application to determine if a driver has fallen asleep at the wheel or isn’t paying attention to the road.”

One of the particularly interesting recent developments is the new Smile Scan capability. Let’s say you own a restaurant or other business where your employees serve customers over a counter. It could be fast-food or bank tellers, but you’re concerned that your people project a friendly image. This new system can watch the individuals from the customer’s vantage point, and see if they’re smiling. At the end of the shift, you’ll know how long each server was smiling. While that may seem almost trivial, it is a valuable training tool and represents some remarkable engineering.

Hall explains, “Facial recognition technology is based on a 3-D mask that is fitted over an image of a face. It’s looking at wrinkles and movement around the lips and eyes so it operates differently than most industrial machine vision applications. It’s more subtle than a typical in-line application where you might be looking to see if a black spot is in exactly the same location on a group of products. The fundamentals are the same in that it’s looking at shapes and colors and the location of those things in a given object, but in terms of the actual process we’re using a different approach than normal machine vision inspection.”


Facial recognition technology is based on a 3-D mask that is fitted over an image of a face. It’s looking at wrinkles and movement around the lips and eyes so it operates differently than most industrial machine vision applications. Source: Omron

When you’re smiling

Detecting smiles is only one emotion that can be measured. The computer is able to determine if a subject is happy, distracted, sleepy, or in pain. “We’re trying to expand the range of emotions that it’s able to analyze, but it’s quite difficult,” Hall adds. “You could be squinting in pain or squinting in joy, and it looks the same to the computer.”

As the technology is developed, it can be incorporated into appropriate products. For example, a medical device such as a vital signs monitor can also watch the patient and look for signs of pain that might not appear as changes in the pulse rate or other respiration. A car or truck can evaluate the condition of the driver and respond to signs of fatigue or distraction.

Does it work? For the most part, yes. Devices tested in Japan to judge the age of an individual wanting to buy tobacco or alcohol were generally as good as or even better than human observers who had been trained to judge faces in jobs such as passport agents and workers in bars looking for underage drinkers. Still, determining if the subject is 17 or 22 is easier than spotting the difference between 53 and 58.

Facial recognition and evaluation systems have the ability to record and quantify reactions, without some of the interpretation of human evaluators. For example, if a company is testing consumer reaction to a new product such as a soft drink or ice cream flavor, the system can watch test participants and judge reactions. The system always applies the same standards to the subjects, which makes for more directly comparable results over time.

These technologies can also be put to work in more traditional machine vision applications. The ability to make subtle evaluations of faces could be applied to other three-dimensional objects, particularly those where there is the possibility of slight shape variations and inconsistent positions, just like human beings.

—By Peter Welander, process industries editor, Control Engineering, www.controleng.com PWelander@cfemedia.com

Online, see more on Omron vision sensing , including more on OKAO vision, silhouette vision and pattern vision.


Related Resources