Why Cars Are Increasingly Watching the Road - And the Driver

Tradeoffs between privacy and safety are becoming an important consideration, especially as demand for more information to be made available to drivers by carmakers, advertisers - and consumers - becomes a reality. JL

John Quain reports in the New York Times:

Cameras that recognize facial expressions, sensors that detect heart rates and software that assesses a driver’s state of awareness are increasingly viewed as part of an inevitable driving future.3-D vision and “micro vibration” sensing systems (can) recognize faces, check seatbelt use, adjust elements like airbag deployment velocity based on a passenger’s weight. To deliver other advanced services, like augmented reality information about nearby businesses and locations, it will be necessary to monitor what drivers are paying attention to. Eventually technology could judge the emotional state of people in the car.

The automobile, in American life, has long been a hallmark of freedom. A teenager’s first driver’s license offers freedom from Mom and Dad. A new car and the open road bring the freedom to chase the American dream. But as more technology creeps in to help drivers, so, too, will systems that eavesdrop on and monitor them, necessitated not by convenience but by new safety concerns.

Cameras that recognize facial expressions, sensors that detect heart rates and software that assesses a driver’s state of awareness may seem like superfluous flights of fancy, but they are increasingly viewed as part of an inevitable driving future.

At upstarts like the electric car company Byton and mainstream mainstays like Volvo, car designers are working on facial recognition, drowsy-driver alert systems and other features for keeping track of the people behind the wheel.

The most immediate impetus: concerns about the safe use of driver-assistance options like automatic lane-keeping that still require drivers to pay attention. And when truly autonomous vehicles finally arrive, the consensus among automakers and their suppliers is that new ways will be needed to check on drivers and passengers to make sure they are safe inside.

“It’s really taken off from no monitoring to tactile monitoring to taking a look at your eyes,” said Grant Courville, a vice president at BlackBerry QNX, which creates in-dash software systems. “I definitely see more of that coming as you get to Level 3 cars,” he added, referring to vehicles that can perform some self-driving functions in limited situations.

With its 2018 Cadillac CT6, for example, General Motors introduced an infrared camera that looks up from the steering column at drivers.

The feature is part of the car’s Super Cruise system, the first hands-free driving tool to operate on select United States highways. The camera tracks a driver’s head position and eye movements to ensure that the person is attentive and able to retake control of the car when needed.

Similar concerns about BMW’s semi-autonomous systems prompted the German carmaker to add a driver monitoring camera in its 2019 X5 sport utility vehicle. The video camera is mounted in the instrument cluster as part of BMW’s Extended Traffic Jam Assistant system, part of a $1,700 package, that allows the car to go autonomous — with driver monitoring — in stop-and-go traffic under 37 miles per hour.

“It looks at the head pose and the eyes of the driver,” said Dirk Wisselmann of BMW’s automated driving program. “We have to, because by doing so it empowers us to add more functionality.”

Automakers understand that tracking technology raises privacy issues, so BMW does not record or store the driver monitoring information, Mr. Wisselmann said.

Perhaps still smarting from lessons learned in the past, G.M. also does not record what transpires inside the car’s cabin, the company said. In 2011, G.M. tried to change the user agreement in its OnStar service to allow it to share driver information with third-party companies. The backlash from owners was so swift and severe that the Supreme Court cited the episode as proof that people had an expectation of privacy in their cars.

G.M. later introduced the first built-in video cameras and microphones meant to record the drive in 2014 Chevrolet Corvettes. A valet mode (“Think of it as a baby monitor for your car,” an official said in a Chevrolet news release at the time) that surreptitiously recorded Ferris Bueller-like joy rides made the company so nervous that it sent letters to owners warning about possible legal issues and asking them to refrain from using the setting.

Now that driver-assistance programs that can steer a vehicle down the highway are becoming more widespread, automakers fear that without such monitoring efforts, some drivers will abuse or misuse the semi-autonomous systems.

There are multiple cases of Tesla owners circumventing the requirement that drivers keep their hands on the wheel when the car’s lane-keeping Autopilot feature is engaged. The consequences have sometimes been fatal.

“But it’s not just about distraction management,” said Jada Smith, a vice president in the advanced engineering department at the auto supplier Aptiv. During an autonomous driving demonstration, she pointed out that such driver monitoring systems can assess a driver’s cognitive load levels — how many tasks the person is trying to juggle — and then adjust other car functions.

“If the driver is not fully aware,” Ms. Smith said, “we might brake faster.” Other ideas include putting radar inside the car for interior sensing like detecting that a child has been left behind. (Every nine days a child left in a car dies from vehicular heatstroke in the United States, according to KidsAndCars.org, an advocacy group.)

It was infants’ being left in cars that first prompted Guardian Optical Technologies, based in Tel Aviv, to develop in-cabin monitoring technology, said Tal Recanati, the company’s chief business officer. The company has now expanded its 3-D vision and “micro vibration” sensing system to recognize faces, check seatbelt use, even adjust elements like airbag deployment velocity based on a passenger’s approximate weight. Eventually Guardian’s technology could be able to judge the emotional state of people in the car.

Affectiva, a Boston company developing technology for measuring emotions, has been conducting such research for several years to assess driver behavior. On a closed test track peppered with distractions — people dressed as construction workers, a security vehicle with flashing lights, pedestrians, fake storefronts — Affectiva demonstrated how the company’s program works in tandem with a “collaborative driving” system made by the Swedish auto supplier Veoneer. Veoneer’s technology can control steering and braking on its own, with the occasional intervention of a human driver.

Affectiva collected a variety of driver information during the test, measuring elements like the amount of grip on the wheel, throttle action, vehicle speed and the driver’s eyes, facial and head movements. It then compared that information with what was happening around the car to determine how much trust the driver had in the semi-autonomous system and the perceived level of cognitive load.

“We want them to trust the car — but not too much,” said Ola Bostrom, a vice president of research at Veoneer. “The driver still has to be engaged” in order to take over the controls when a car encounters a situation it can’t handle.

To deliver other advanced services, like augmented reality information about nearby businesses and locations, it will also be necessary to monitor what drivers are paying attention to, said Andrew Poliak, a vice president at Panasonic Automotive Systems. And companies as diverse as Mercedes-Benz and the voice-recognition company Nuance want to add Alexa-like services, meaning that your sedan or S.U.V. may always be listening.

“So these systems are going to become standard in all cars,” said Nakul Duggal, who leads the automotive products group at Qualcomm.

Will privacy concerns then recede in the rearview mirror of advancing technologies?

When fully autonomous vehicles begin circulating on public roads, designers note, they will have to be able to detect when people enter or exit a vehicle, who the person is, whether they have left anything behind in the car, and especially if a person has become disabled (because of intoxication or a medical emergency). And that information will inevitably be shared online, although there may be ways that some people can still preserve their sense of independence in the car.

“In the future, it may be different for people who own their own cars, where there’s more privacy,” said Mr. Wisselmann at BMW, "and for people who use robo-taxis, where there will be less."