AI for the last mile

Car manufacturers and legislators from Germany to Saudi Arabia are gearing up for the arrival of autonomous vehicles (AVs) on public roads this decade, but sceptics argue that the technology needed for completely automated driving is far from ready.

Despite the giant leaps in artificial intelligence (AI) algorithms trained on data gathered from sensors both inside and outside the car, the seemingly rapid advances to date may only prove to be the low hanging fruit. Cracking the last mile will be harder and take a lot longer.

“It is the last 10% of cases and situations that is proving a bottleneck in development,” Matthew Avery, Director of Research at motor insurance industry funded researcher Thatcham Research was quoted as saying in the Guardian.

The bulk of rules for AVs such as following the line of the road, sticking to a certain side and avoiding crashing into other cars can be addressed by AI algorithms.

Self-driving vehicles use sensors, cameras, radars and in some cases laser imaging, detection and ranging (LIDAR) technology to gather the data required to run autonomously. Several standards have been developed that can help with autonomous transport. IEC TC 47 publishes IEC 62969, which specifies the general requirements of power interfaces for automotive vehicle sensors.

IEC TC 100 issues several standards relating to multimedia systems in cars. One of its publications is IEC technical specification (TS) 63033,which specifies a model for generating the surrounding visual image of the drive monitoring system, which creates a composite 360° image from external cameras. This enables the correct positioning of a vehicle in relation to its surroundings, using input from a rear-view monitor for parking assistance as well as blind corner and bird’s eye monitors.

But it is far more difficult for algorithms to address what Avery refers to as “edge cases” – rare and unusual events that a self-driving vehicle has not encountered before. Examples might be a dog running into the road or an unexpected weather-related accident, for instance.

Cracking the last mile

There are five grades of automated vehicle systems as classified by the US-based Society of Automotive Engineers (SAE). These range from functions which automate distance control to totally autonomous vehicles, which means there is no requirement for a driver even to be present behind the wheel. Level 5 AVs may even lack a steering wheel as well as accelerator and brake pedals. Passengers might use voice commands to select a location or control what TV show they want to watch in transit. Crucially, level 5 vehicles are meant to be able to operate on roads anywhere, not just in certain designated areas.

Professor Michael Felsberg, Head of Sweden’s Linköping University’s computer vision lab says several problems stand in the way of achieving such a goal. One of them is image classification. “We know that for each image, this is a bicycle, this is a dog and this is a car,” he explains. “The images are hand-labelled by humans and the annotated images are used to train image recognition systems.”

Felsberg explains that AI algorithms require a period of supervised learning before a system can be deployed. In preparation for this phase, an army of annotators is needed to label the images for a given application. Images are annotated with not only the name of the class of objects the algorithm should look for, but also the location of the object within the image. 

For large-scale industrial use of AI, this amount of annotation is impractical, Felsberg says. “For autonomous vehicles to work on a large scale, algorithms should be able to recognize new classes of objects without having to undergo another round of supervised training. It takes too much time and effort to re-label the huge volumes of data. It would be much better if the algorithm could learn to recognize the new class after it has been deployed.”

Researchers have yet to come up with a robust and effective method for this process, which is referred to as ‘class incremental learning’.

Making sense of huge data sets

A computer vision specialist company , owned by a large American chip maker, acknowledges the problem but believes this is surmountable when given enough data to train the AI. The issue is that many AV developers “do not have the tools to effectively make sense of large data sets.”

The company is sitting on 200 petabytes (PB) of driving data which is stored between a popular external cloud solution and on-premise systems. “Data and the infrastructure in place to harness it is the hidden complexity of autonomous driving,” says Amnon Shashua, the company’s President and CEO. “Our company has spent 25 years collecting and analyzing what we believe to be the industry’s leading database of real-world and simulated driving experience.”

The company’s team uses an in-house search engine database with millions of images, video clips and scenarios ranging from “tractor covered in snow” to “traffic light in low sun,” which are fed to its algorithms. When combined with its computer vision technology and natural language understanding (NLU) models, the dataset can deliver thousands of results within seconds, even for incidents that fall into the category of rare and unexpected conditions and scenarios. The company plans to debut “the world’s first consumer AV with Level 4 autonomous capability” in collaboration with a Chinese EV manufacturer in the Asian country in 2024.

Giving AI systems “ common sense”

The other thing about edge cases is that they are not all that rare, which means that finding a key to recognizing them with AI is indispensable if self-driving vehicles are to really hit the roads at one point. “They might be infrequent for an individual driver, but if you average out over all the drivers in the world, these kinds of edge cases are happening very frequently to somebody,” Melanie Mitchell, computer scientist and Professor of Complexity at the Santa Fe Institute, tells The Guardian.

Humans can generalize from one scenario to the next but, while a self-driving system appears to “master” a certain situation, it doesn’t necessarily mean it will be able to replicate this under slightly different circumstances. “It’s a challenge to try to give AI systems common sense, because we don’t even know how it works in ourselves,” says Mitchell.

The stated aim of AV makers is to create cars that are safer than human-driven vehicles. That’s because humans are fallible, and drunk driving, for instance, is responsible for a very high number of deadly accidents. Yet in the limited history of AVs, there have already been fatalities. Elaine Herzberg, aged 49, was hit by a self-driving car as she wheeled a bicycle across the road in Tempe, Arizona, in 2018.

“I think that if every car was a self-driving car, and the roads were all mapped perfectly, and there were no pedestrians around, then autonomous vehicles would be very reliable and trustworthy,” says Mitchell. “It’s just that there’s this whole ecosystem of humans and other cars driven by humans that AI just doesn’t have the intelligence yet to deal with.”