Machine vision’s next frontier: Facial recognition

Starting in the 1990s, engineers started to teach computers to identify faces in photographs. Can machine control via eye roll be far behind? Such vision software advances will help site security, manufacturing quality and, perhaps, future human-machine interface applications. Images show how it operates differently than most industrial machine vision applications. Here's how it works.


Starting in the 1990s, engineers started to teach computers to identify faces in photographs. Since then, the capability has moved from simple face detection to face recognition and even attribute estimation. Can machine control via eye roll be far behind?

Omron has developed a group of facial recognition products under the name Okao Vision , borrowing the Japanese word for “face.”

Machine Vision, May '09, Control Engineering

Other articles in the May 2009 Control Engineering North American print edition supplement:
- 415 Parts Seen
- Motion and Vision Combine to Detect Flaws
- Intelligent Vision Stops Bypass of Quality Control

Imagine someone wants to enter a manufacturing plant through a remote entrance that requires both a smart ID card and PIN to open the gate. Due to some careful social engineering, the intruder has stolen a card and obtained the right PIN, but the gate remains locked because a computer vision system has looked at the intruder and realized that he is not the person to whom the card belongs.

A talkative teenage driver is an accident waiting to happen because he or she tends to look away from the road while chatting with friends in the car. If the car is moving and the driver turns to look at a passenger, imagine if the car beeps and says, “Watch where you’re going.”

Such applications are becoming more practica l and commonplace thanks to growing developments of facial recognition platforms, a spin-off from more traditional machine vision applications.

A basic capability allows a computer to look at a person’s face and compare that image to one in a database, thereby verifying the identity of the individual standing at the gate. That has become a fairly common application. But current technology can go much farther. Even medium-priced digital cameras now have the ability to recognize when subjects are smiling. When in “smile shutter” mode, a new Sony camera automatically takes a picture only when the subject is smiling. It evaluates mouth shape, visibility of teeth, position of cheek bones, and eye shape and activates the shutter only when the proper criteria are met. It has a harder time if the individual is wearing sunglasses.

More sophisticated developments allow the system to look at an individual and make some conclusions about that person, and even his or her state of mind. Omron has developed a group of facial recognition products under the name Okao Vision , borrowing the Japanese word for “face.”

Omron began by simply finding faces in an image but that was only a start. “From there it got more complicated,” says Sarah Hall, corporate communications for Omron. “We found ways to determine gender and age in the last couple of years. It can tell what direction someone is looking, and if that person has turned to the side. This is useful in an automotive application to determine if a driver has fallen asleep at the wheel or isn’t paying attention to the road.”

One of the particularly interesting recent developments is the new Smile Scan capability. Let’s say you own a restaurant or other business where your employees serve customers over a counter. It could be fast-food or bank tellers, but you’re concerned that your people project a friendly image. This new system can watch the individuals from the customer’s vantage point, and see if they’re smiling. At the end of the shift, you’ll know how long each server was smiling. While that may seem almost trivial, it is a valuable training tool and represents some remarkable engineering.

Hall explains, “Facial recognition technology is based on a 3-D mask that is fitted over an image of a face. It’s looking at wrinkles and movement around the lips and eyes so it operates differently than most industrial machine vision applications. It’s more subtle than a typical in-line application where you might be looking to see if a black spot is in exactly the same location on a group of products. The fundamentals are the same in that it’s looking at shapes and colors and the location of those things in a given object, but in terms of the actual process we’re using a different approach than normal machine vision inspection.”

Photo: Omron Okao Vision looks at rough structure, features, and details in faces.

Facial recognition technology is based on a 3-D mask that is fitted over an image of a face. It’s looking at wrinkles and movement around the lips and eyes so it operates differently than most industrial machine vision applications. Source: Omron

When you're smiling

Detecting smiles is only one emotion that can be measured. The computer is able to determine if a subject is happy, distracted, sleepy, or in pain. “We’re trying to expand the range of emotions that it’s able to analyze, but it’s quite difficult,” Hall adds. “You could be squinting in pain or squinting in joy, and it looks the same to the computer.”

As the technology is developed, it can be incorporated into appropriate products. For example, a medical device such as a vital signs monitor can also watch the patient and look for signs of pain that might not appear as changes in the pulse rate or other respiration. A car or truck can evaluate the condition of the driver and respond to signs of fatigue or distraction.

Does it work? For the most part, yes. Devices tested in Japan to judge the age of an individual wanting to buy tobacco or alcohol were generally as good as or even better than human observers who had been trained to judge faces in jobs such as passport agents and workers in bars looking for underage drinkers. Still, determining if the subject is 17 or 22 is easier than spotting the difference between 53 and 58.

Facial recognition and evaluation systems have the ability to record and quantify reactions, without some of the interpretation of human evaluators. For example, if a company is testing consumer reaction to a new product such as a soft drink or ice cream flavor, the system can watch test participants and judge reactions. The system always applies the same standards to the subjects, which makes for more directly comparable results over time.

These technologies can also be put to work in more traditional machine vision applications. The ability to make subtle evaluations of faces could be applied to other three-dimensional objects, particularly those where there is the possibility of slight shape variations and inconsistent positions, just like human beings.

—By Peter Welander, process industries editor, Control Engineering,

Online, see more on Omron vision sensing , including more on OKAO vision, silhouette vision and pattern vision.

No comments
The Engineers' Choice Awards highlight some of the best new control, instrumentation and automation products as chosen by...
Each year, a panel of Control Engineering editors and industry expert judges select the System Integrator of the Year Award winners.
The Engineering Leaders Under 40 program identifies and gives recognition to young engineers who...
Learn how to increase device reliability in harsh environments and decrease unplanned system downtime.
This eGuide contains a series of articles and videos that considers theoretical and practical; immediate needs and a look into the future.
Learn how to create value with re-use; gain productivity with lean automation and connectivity, and optimize panel design and construction.
Go deep: Automation tackles offshore oil challenges; Ethernet advice; Wireless robotics; Product exclusives; Digital edition exclusives
Lost in the gray scale? How to get effective HMIs; Best practices: Integrate old and new wireless systems; Smart software, networks; Service provider certifications
Fixing PID: Part 2: Tweaking controller strategy; Machine safety networks; Salary survey and career advice; Smart I/O architecture; Product exclusives
The Ask Control Engineering blog covers all aspects of automation, including motors, drives, sensors, motion control, machine control, and embedded systems.
Look at the basics of industrial wireless technologies, wireless concepts, wireless standards, and wireless best practices with Daniel E. Capano of Diversified Technical Services Inc.
Join this ongoing discussion of machine guarding topics, including solutions assessments, regulatory compliance, gap analysis...
This is a blog from the trenches – written by engineers who are implementing and upgrading control systems every day across every industry.
IMS Research, recently acquired by IHS Inc., is a leading independent supplier of market research and consultancy to the global electronics industry.

Find and connect with the most suitable service provider for your unique application. Start searching the Global System Integrator Database Now!

Case Study Database

Case Study Database

Get more exposure for your case study by uploading it to the Control Engineering case study database, where end-users can identify relevant solutions and explore what the experts are doing to effectively implement a variety of technology and productivity related projects.

These case studies provide examples of how knowledgeable solution providers have used technology, processes and people to create effective and successful implementations in real-world situations. Case studies can be completed by filling out a simple online form where you can outline the project title, abstract, and full story in 1500 words or less; upload photos, videos and a logo.

Click here to visit the Case Study Database and upload your case study.