Fingertip sensor gives robot unprecedented dexterity

A robot can grasp a USB plug and inserts it into a USB port after being equipped with an optical sensor.


Figure 1: Armed with the GelSight sensor, a robot can grasp a freely hanging USB cable and plug it into a USB port. Courtesy: Melanie Gonick, MITResearchers at MIT and Northeastern University have equipped a robot with a novel tactile sensor that lets it grasp a USB cable draped freely over a hook and insert it into a USB port.

The sensor is an adaptation of a technology called GelSight, which was developed by the lab of Edward Adelson, the John and Dorothy Wilson Professor of Vision Science at MIT, and first described in 2009. The new sensor isn't as sensitive as the original GelSight sensor, which could resolve details on the micrometer scale. But it's smaller — small enough to fit on a robot's gripper — and its processing algorithm is faster, so it can give the robot feedback in real time.

Industrial robots are capable of remarkable precision when the objects they're manipulating are perfectly positioned in advance. But according to Robert Platt, an assistant professor of computer science at Northeastern and the research team's robotics expert, for a robot taking its bearings as it goes, this type of fine-grained manipulation is unprecedented.

"People have been trying to do this for a long time," Platt said, "and they haven't succeeded because the sensors they're using aren't accurate enough and don't have enough information to localize the pose of the object that they're holding."

The researchers presented their results at the International Conference on Intelligent Robots and Systems this week. The MIT team — which consists of Adelson; first author Rui Li, a PhD student; Wenzhen Yuan, a master's student; and Mandayam Srinivasan, a senior research scientist in the Department of Mechanical Engineering — designed and built the sensor. Platt's team at Northeastern developed the robotic controller and conducted the experiments.

Figure 2: The GelSight sensor, attached to a robot's gripper. Courtesy: Melanie Gonick, MITSynesthesia

Whereas most tactile sensors use mechanical measurements to gauge mechanical forces, GelSight uses optics and computer-vision algorithms.

"I got interested in touch because I had children," Adelson said. "I expected to be fascinated by watching how they used their visual systems, but I was actually more fascinated by how they used their fingers. But since I'm a vision guy, the most sensible thing, if you wanted to look at the signals coming into the finger, was to figure out a way to transform the mechanical, tactile signal into a visual signal — because if it's an image, I know what to do with it."

A GelSight sensor — both the original and the new, robot-mounted version — consists of a slab of transparent, synthetic rubber coated on one side with a metallic paint. The rubber conforms to any object it's pressed against, and the metallic paint evens out the light-reflective properties of diverse materials, making it much easier to make precise optical measurements.

In the new device, the gel is mounted in a cubic plastic housing, with just the paint-covered face exposed. The four walls of the cube adjacent to the sensor face are translucent, and each conducts a different color of light — red, green, blue, or white — emitted by light-emitting diodes at the opposite end of the cube. When the gel is deformed, light bounces off of the metallic paint and is captured by a camera mounted on the same cube face as the diodes.

From the different intensities of the different-colored light, the algorithms developed by Adelson's team can infer the three-dimensional structure of ridges or depressions of the surface against which the sensor is pressed.

Figure 3: The GelSight sensor uses different-colored light, channeled down different faces of the cubic housing, to make three-dimensional measurements. Courtesy: Melanie Gonick, MITAlthough there are several ways of measuring human tactile acuity, one is to determine how far apart two small bumps need to be before a subject can distinguish them just by touching; the answer is usually about a millimeter. By that measure, even the lower-resolution, robot-mounted version of the GelSight sensor is about 100 times more sensitive than a human finger.

Plug 'n play

In Platt's experiments, a Baxter robot from MIT spinout Rethink Robotics was equipped with a two-pincer gripper, one of whose pincers had a GelSight sensor on its tip. Using conventional computer-vision algorithms, the robot identified the dangling USB plug and attempted to grasp it. It then determined the position of the USB plug relative to its gripper from an embossed USB symbol. Although there was a 3-mm variation, in each of two dimensions, in where the robot grasped the plug, it was still able to insert it into a USB port that tolerated only about a millimeter's error.

"As roboticists, we are always looking for new sensors," Lee added. "This is a promising prototype. It could be developed into a practical device."

Massachusetts Institute of Technology (MIT) 

- Edited by CFE Media. See more Control Engineering robotics stories.

No comments
The Engineers' Choice Awards highlight some of the best new control, instrumentation and automation products as chosen by...
The System Integrator Giants program lists the top 100 system integrators among companies listed in CFE Media's Global System Integrator Database.
Each year, a panel of Control Engineering and Plant Engineering editors and industry expert judges select the System Integrator of the Year Award winners in three categories.
This eGuide illustrates solutions, applications and benefits of machine vision systems.
Learn how to increase device reliability in harsh environments and decrease unplanned system downtime.
This eGuide contains a series of articles and videos that considers theoretical and practical; immediate needs and a look into the future.
Additive manufacturing benefits; HMI and sensor tips; System integrator advice; Innovations from the industry
Robotic safety, collaboration, standards; DCS migration tips; IT/OT convergence; 2017 Control Engineering Salary and Career Survey
Integrated mobility; Artificial intelligence; Predictive motion control; Sensors and control system inputs; Asset Management; Cybersecurity
Featured articles highlight technologies that enable the Industrial Internet of Things, IIoT-related products and strategies to get data more easily to the user.
This article collection contains several articles on how automation and controls are helping human-machine interface (HMI) hardware and software advance.
This digital report will explore several aspects of how IIoT will transform manufacturing in the coming years.

Find and connect with the most suitable service provider for your unique application. Start searching the Global System Integrator Database Now!

Infrastructure for natural gas expansion; Artificial lift methods; Disruptive technology and fugitive gas emissions
Mobility as the means to offshore innovation; Preventing another Deepwater Horizon; ROVs as subsea robots; SCADA and the radio spectrum
Future of oil and gas projects; Reservoir models; The importance of SCADA to oil and gas
Automation Engineer; Wood Group
System Integrator; Cross Integrated Systems Group
Jose S. Vasquez, Jr.
Fire & Life Safety Engineer; Technip USA Inc.
This course focuses on climate analysis, appropriateness of cooling system selection, and combining cooling systems.
This course will help identify and reveal electrical hazards and identify the solutions to implementing and maintaining a safe work environment.
This course explains how maintaining power and communication systems through emergency power-generation systems is critical.
click me