Mapping 3D spaces with 2D cameras using AI for autonomous vehicles

Researchers have developed a technique that allows AI programs to better map three-dimensional spaces using two-dimensional images captured by multiple cameras, which could improve autonomous vehicles' navigation capabilities.

By Matt Shipman June 27, 2024
Image courtesy: Brett Sayles

AI modeling and vision system insights

  • Researchers at NC State developed Multi-View Attentive Contextualization (MvACon), enhancing AI’s 3D mapping from 2D images with minimal computational overhead.
  • MvACon boosts object location, speed, and orientation detection in vision transformers, improving autonomous vehicle navigation significantly.

North Carolina State University researchers have developed a technique that allows artificial intelligence (AI) programs to better map three-dimensional spaces using two-dimensional images captured by multiple cameras. Because the technique works effectively with limited computational resources, it holds promise for improving the navigation of autonomous vehicles.

“Most autonomous vehicles use powerful AI programs called vision transformers to take 2D images from multiple cameras and create a representation of the 3D space around the vehicle,” said Tianfu Wu, corresponding author of a paper on the work and an associate professor of electrical and computer engineering at North Carolina State University. “However, while each of these AI programs takes a different approach, there is still substantial room for improvement.

“Our technique, called Multi-View Attentive Contextualization (MvACon), is a plug-and-play supplement that can be used in conjunction with these existing vision transformer AIs to improve their ability to map 3D spaces,” Wu said. “The vision transformers aren’t getting any additional data from their cameras, they’re just able to make better use of the data.”

MvACon effectively works by modifying an approach called Patch-to-Cluster attention (PaCa), which Wu and his collaborators released last year. PaCa allows transformer AIs to more efficiently and effectively identify objects in an image.

“The key advance here is applying what we demonstrated with PaCa to the challenge of mapping 3D space using multiple cameras,” Wu said.

To test the performance of MvACon, the researchers used it in conjunction with three leading vision transformers – BEVFormer, the BEVFormer DFA3D variant, and PETR. In each case, the vision transformers were collecting 2D images from six different cameras. In all three instances, MvACon significantly improved the performance of each vision transformer.

“Performance was particularly improved when it came to locating objects, as well as the speed and orientation of those objects,” Wu said. “And the increase in computational demand of adding MvACon to the vision transformers was almost negligible. Our next steps include testing MvACon against additional benchmark datasets, as well as testing it against actual video input from autonomous vehicles. If MvACon continues to outperform the existing vision transformers, we’re optimistic that it will be adopted for widespread use.”

Author Bio: Matt Shipman, research communications lead, North Carolina State University