When the Self-Driving Car Needs Human Backup

The death of an Arizona women killed by a driverless Uber test vehicle may cause the development of autonomous vehicles to adapt a human backup system in order for consumers and regulators to feel entirely safe, at least as a transitional measure. JL 

John Quain reports in the New York Times:

Autonomous vehicles may never be truly autonomous. Command centers with humans would be necessary "to deal with vehicles if they have an issue, but also with customers if they need help.”A car in need would automatically contact a center, where a remote operator could use the car’s cameras and sensors to see what was happening, then maneuver the vehicle out of trouble. The technology prefigures a time when most passengers wouldn’t be able to take control for the simple reason that they won’t know how to drive a car — or because the steering wheel and pedals have been removed.

A car wends its way through a line of taxis in the Las Vegas rain, carefully steering around a tangle of sedans vying for passengers. As the black Lincoln MKZ gets closer, the steering wheel saws back and forth, but there’s no one in the front seat. In fact, there’s no one in the car at all.

It’s disquieting to be picked up by an empty car, and it’s something of a milestone: Inside most autonomous research vehicles cruising public streets, there’s a minder to keep a watchful eye and take control should things go awry. But with the MKZ, there was no human custodian. At least not one within view.

Hundreds of miles away, Ben Shukman, a software engineer for Phantom Auto, was sitting in front of a phalanx of video screens in Mountain View, Calif. Using a live, two-way video connection along with the kind of steering wheel and pedals usually reserved for video games, he was driving the MKZ.

While major technology and car companies are teaching cars to drive themselves, Phantom Auto is working on remote control systems, often referred to as teleoperation, that many see as a necessary safety feature for the autonomous cars of the future. And that future is closer than you might think: California will allow companies to test autonomous vehicles without a safety driver — as long as the car can be operated remotely — starting next month.

While technologies ranging from advanced sensors to machine learning are being applied to self-driving cars, engineers acknowledge that even the most sophisticated vehicles will not be infallible. Developers of self-driving cars from Nissan to Zoox say such technology may be needed to address “edge cases” — the unique situations that software programs can’t anticipate. A fallen tree, a sinkhole, a string of strange pylons, a flash flood, a fire or some other obstruction on a lonely road could make an autonomous car stop safely, but then what?

Photo

Phantom Auto’s technology provides a two-way video connection between the car and the remote driver. Credit Christie Hemm Klok for The New York Times

What happens when a self-driving car can’t help itself?

“We want to be the OnStar for the autonomous industry,” said Shai Magzimof, a co-founder and the chief executive of Phantom Auto. He pictures the technology his company has developed being used in fleet vehicles, robotaxis, trucks and even self-driving cars owned by individuals.

A car in need of help would automatically contact a Phantom Auto center, where a remote operator could use the car’s cameras and sensors to see what was happening, then maneuver the vehicle out of trouble. The technology prefigures a time when most passengers wouldn’t be able to take control for the simple reason that they won’t know how to drive a car — or because the steering wheel and pedals have been removed.

It is a thorny problem other companies are also trying to solve.

Nissan, one of the first automakers to publicly address situations in which a self-driving car may be flummoxed by its surroundings, has proposed using a system called Seamless Autonomous Mobility, or S.A.M. It’s partly based on the remote control technology that NASA uses to operate rovers on Mars.

“The current idea is to draw the new route command onto a screen to direct the car,” said Maarten Sierhuis, the director of Nissan’s research center in Silicon Valley.

Such an approach avoids any possible communications glitches, and Mr. Sierhuis said the company was working on a more advanced solution, S.A.M. 2.0, but wasn’t ready to discuss it yet.

Waymo, the self-driving vehicle unit of Google’s parent company Alphabet, is testing autonomous taxis — but with an observer in the back seat. It is focused on having the car make all the driving decisions, but there is a system for handling edge cases. If a Waymo vehicle becomes confused — by, say, a new set of cones or a police barricade in the road — it can request confirmation from a remote human specialist. Once it receives confirmation of what it is sensing, the car — not the remote operator — then decides how to proceed.

The Waymo approach ensures that latency — a delay in the communications traffic — doesn’t compromise the car’s driving behavior by leaving a remote operator unable to react in real time.

Phantom Auto’s technology, on the other hand, uses a standard 4G cellular data connection and GPS information to link the car and its backup driver. The company has even been able to make it work in areas that may have less than perfect service.

To accomplish this, Phantom Auto mounted a computer the size of a hardcover book, spiked with antennas and four wireless modems, in the trunk. The car was outfitted with three video cameras, but no lidar or far infrared sensors. It’s an arrangement that has already attracted several automotive customers, according to the company, although because of contractual obligations it could not yet disclose specific clients.

Where a typical autonomous vehicle may drive like a timid student driver, the Phantom car behaved naturally — after all, there was a human there. He just happened to be sitting in another state.

Photo

The control room at Phantom Auto. The company has already attracted several automotive customers, although because of contractual obligations it could not yet disclose specific clients. Credit Christie Hemm Klok for The New York Times

Phantom hopes its approach will also reassure passengers by letting them talk to and see the remote operator — within limits. As our distant driver steered the car through heavy traffic in the rain, Mr. Magzimof advised against distracting the operator with too many questions: “He’s focused on driving.”

Gill Pratt, chief executive of the Toyota Research Institute, said that while he believed autonomous vehicles would need some sort of additional support and assistance systems in rare instances, he saw them as “not something people would look forward to using.”

Mr. Pratt also pointed out that teleoperation faced several hurdles, ranging from the limited availability of wireless networks to the time it takes for remote operators to become “situationally aware” — informed not just of the immediate problem, but important context including the location or weather or any other variable that someone in the driver’s seat would already know.

Kathy Winter, the general manager of Intel’s autonomous driving group, said that because of these limitations, her team was making every effort to “make sure the vehicle never gets into a situation where this would be necessary.” She noted that redundant systems — including multiple sensors, high-resolution maps, crowdsourced data and vehicle-to-vehicle communications — would ensure that the self-driving cars of the future would have multiple cues about their surroundings.

When pressed, developers like Zoox admit that autonomous vehicles may never be truly autonomous. Zoox’s chief executive, Tim Kentley-Klay, acknowledged at a Senate hearing on self-driving cars in January that command centers with humans would be necessary “both to deal with vehicles if they have an issue, but also to deal with customers if they need help.”

In other words, if your autonomous sedan gets a flat tire, someone is still going to have to come out and fix it.