What Does a Vice President of Reality Capture Do?

The more important question, in an increasingly digitally dependent world, may be determining what reality is - and who gets to decide when it needs to change. JL

The Economist reports:

Trying to model reality down to every single bridge pillar and then triangulating it is too sensitive to change. Problems can arise if, for instance, a tree is cut down or a street scene alters from summer to winter. There are too many visual changes. Machine-learning algorithms find such things as lane markings and the edges of pavements. AI systems can identify road signs and traffic lights from still photos. Humans then modify and tweak the results, and check for errors.
A CAR sprouting a dome containing a spinning laser sensor and festooned with cameras barely draws a second glance as it edges through the crowded streets of Berkeley. Self-driving cars are no longer a rare sight on Californian roads. Over 100 autonomous vehicles from a dozen manufacturers are now being tested in public, covering hundreds of thousands of kilometres each year.
But this car is different: its human driver keeps his hands firmly on the wheel. The vehicle, nicknamed “George” by HERE, a Berlin-based mapping company owned by BMW, Audi and Daimler, is not driving itself but collecting data that enable other cars to do so.

For every second of its journey, a high-precision GPS receiver on George’s roof collects the car’s latitude, longitude and elevation ten times over; a motion-tracking inertial system records its yaw, pitch and roll 100 times; and the laser scanner calculates its distance from some 600,000 different points, such as trees, kerbs and buildings. At the same time, four cameras also shoot a 96-megapixel, 360-degree panoramic image for every 6 metres the vehicle moves along the road.

A day’s driving can accumulate 100 gigabytes or more of
data. Together, these allow HERE to build up an extremely detailed three-dimensional image of George’s route—what digital cartographers call a high definition (HD) map.
A few years ago, some carmakers hoped that autonomous vehicles might be able to position themselves using the low-definition maps found in today’s turn-by-turn navigation devices and apps. Sensors would do the rest. With clear road markings, for instance, visual sensors can already keep cars safely within their lanes, and even spot the solid or dotted lines that indicate stop signs and exits.
The trouble is, a fully driverless car needs to operate safely in all environments. “You don’t really need a map to do simple lane-keeping,” says John Ristevski, HERE’s grandiosely named vice-president of reality capture. “But if you’re on a five-lane freeway, you need to know which of those five lanes you’re in, which are safe to traverse, and at what exact point that exit ramp is coming up.”
The trouble is road markings can wear away or disappear under snow. And modern laser-surveying sensor systems (called LIDARs, after light detection and ranging) may not be accurate in those conditions. LIDARS calculate distances by illuminating a target with laser light and measuring the time it takes for the light to bounce back to the source. Radar does much the same thing with radio waves. In cars, LIDARS and radars have an effective range of around 50 metres, but that can shrink significantly in rain or when objects are obscured by vehicles ahead. Even the smartest car travelling at motorway speeds can “see” only around a second and a half ahead. What HD maps give self-driving cars is the ability to anticipate turns and junctions far beyond sensors’ horizons.
Even more important for an autonomous vehicle is the ability to locate itself precisely; an error of a couple of metres could place a car on the wrong side of the road. Commercial GPS systems are accurate only to around 5 metres, but can be wrong by 50 metres in urban canyons and fail completely in tunnels. HD maps, however, can include a so-called localisation layer that works with a variety of sensors to position a car within centimetres.
HERE is experimenting with several such layers. One involves extracting features like bridges, road signs and guard rails from images shot by the mapping vehicle, and then comparing them to what the car sees through its own cameras.
TomTom, a mapping firm based in the Netherlands, rejected this process as too unreliable. “We found that trying to model reality down to every single bridge pillar and then triangulating it is too sensitive to change,” says Pieter Gillegot-Vergauwen, one of the firm’s vice-presidents. Problems can arise if, for instance, a tree is cut down or a street scene alters from summer to winter. “There are too many visual changes,” he adds.
Instead, TomTom captures a “depth map” using its mapping vehicles’ LIDARS. This system continuously records the distinctive shapes and distances of roadside scenery, without trying to identify what the individual things are. By considering the whole stretch of road it is possible to correlate the output from the autonomous car’s own LIDAR unit with the pattern of the depth map and calculate its location even if, say, a tree grows or a lorry is in the way, says Mr Gillegot-Vergauwen.
Google, which has long been testing autonomous cars, builds its localisation layer in a similar fashion. HERE is also trying out a system that uses artificial intelligence to identify features from cameras and LIDARs. Whatever the approach, all three companies claim that they can now position a self-driving car on the road to an accuracy of within 10-20cm.
Some car firms, including Nissan, Ford, Kia and Tesla, think self-driving technology will be ready by 2020. Volvo plans to offer fully autonomous cars to 100 drivers as early as next year. All this increases the pressure to map the world in high definition before cars begin to drive themselves out of showrooms. HERE has several hundred vehicles like George mapping millions of kilometres of roads annually in 32 countries. TomTom has 70 on motorways and major roads in Europe and North America. Zenrin, a Japanese mapping firm partly owned by Toyota, is particularly active in Asia.
Analysing and processing data from so many vehicles is one of the biggest challenges. HERE originally had people inspecting the raw LIDAR data and turning it into a digital model using editing software—rather like “Minecraft for maps”, says Mr Ristevski. But manually extracting the data was painfully slow, and the company has now developed machine-learning algorithms to find automatically such things as lane markings and the edges of pavements. HERE’s AI systems can identify road signs and traffic lights from George’s still photos. Humans then modify and tweak the results, and check for errors.
Yet George’s data begin to age as soon as they are collected. Subsequent construction, roadworks or altered speed limits could lead to a self-driving car working from a dangerously outdated map. Maps will never be completely up-to-date, admits Mr Ristevski. “Our goal will be to keep the map as fresh and accurate as possible but vehicle sensors must be robust enough to handle discrepancies.”
Mapping vehicles are sent back to big cities like San Francisco regularly, but the vast majority of the roads they capture might be revisited annually, at best. A partial solution is to use what Mr Ristevski euphemistically calls “probe data”: the digital traces of millions of people using smartphones and connected in-car systems for navigation. HERE receives around 2 billion individual pieces of such data daily, comprising a car’s location, speed and heading, some of it from Windows devices (a hangover from when HERE was owned by Nokia, now part of Microsoft).
These data are aggregated and anonymised to preserve privacy, and allow HERE quickly to detect major changes such as road closures. As cars become more sophisticated, these data should become richer. Ultimately, reckons Mr Ristevski, self-driving cars will help to maintain their own maps. This is already the case with Google’s self-driving cars, which can detect and report traffic cones and construction workers in high-visibility vests. Not only does Google have more autonomous vehicles on the roads than any other carmaker, it has access to navigation and traffic data from the estimated 1.5 billion Android phones and devices active globally. (Google says it is currently concentrating its HD-mapping efforts on just its self-driving test locations in Mountain View, California; Austin, Texas; Kirkland, Washington, and Phoenix, Arizona.)
As more new cars are fitted with smart-driving features, such as automatic braking, lane control and overtaking, technology will continue to lead vehicles towards full autonomy. And HD maps will extend beyond the road. Both HERE and TomTom include low-level aerial information, such as utility wires, bridges, trees and, in some cases, details of buildings up to 15 storeys. Such data could be used for navigation by another type of robotic vehicle—drones—which is why one company with drone-delivery ambitions, Amazon, is in talks to buy a stake in HERE.