The iPhone's Best AI Is Not In Siri, But In Its Camera

It is difficult to imagine that the camera is more of a priority than the voice enabled assistant, but Apple appears to be investing heavily in augmented reality as a strategic base for generation of monetizable services, which may explain the focus on photographic enhancement for that technology. JL


Dave Gershgorn reports in Quartz:

Apple has been transitioning its cameras from devices that take a picture, into devices that capture data for AI to make into an image. It’s called computational photography. (It's) also key to augmented reality, where the machine learning technique has to differentiate between all the objects in a video. Being able to tell the shape of flat surfaces and walls allows for a sense of depth that’s needed to make the AR effect seem realistic.

When Apple unveiled its new slate of iPhones on Sept. 12, executives gushed over the device’s new second-generation “neural processor,” a custom chip meant just for AI.

But Apple didn’t boast about how much the new chip was going to improve Siri—Apple’s virtual assistant that got eight seconds of the 2 hour presentation—or boost battery life, as the company often claims its machine learning is used for. The real killer app for AI on the iPhone is photography.

The Cupertino-based company has been slowly transitioning its cameras from devices that take a picture, into devices that capture data for AI to make into an image. It’s called computational photography, and it isn’t unique to Apple—researchers have been thinking about the camera as a way to capture data for AI for years. But you’ll probably hear about it more often (at least if you make it a habit to watch Apple events).

This is how Apple’s phones are able to manipulate the focus of an image after its been taken, or change the lighting in portrait mode. As reported two years ago by Quartz, image processing like this has been a research focus for Apple’s AI team. The company also acquired computer vision startup Regaind in 2017, bolstering its research ranks.

Apple’s AI work includes image tasks like segmentation, a computer science term used for identifying where objects in an image are. When Apple’s AI segments an image like a selfie, it’s able to cut the person out from the background. That’s how portrait mode works to separate the subject from the background (though it had some issues.

at first), as well as portrait lighting and Apple’s new background blurring effect.

Segmentation is also key to augmented reality, where the machine learning technique has to differentiate between all the objects in a video. Being able to tell the shape of flat surfaces and walls allows for a sense of depth that’s needed to make the AR effect seem realistic.

Apple also says that machine learning is behind its Smart HDR feature, where parts of multiple images with the most detail are made into a single image. If you were trying to take a picture of a friend on a bright day, the phone would quickly take a bunch of pictures at different exposures—one where there’s detail on your friend’s face, one where there’s detail in the clouds, and one where there’s a detail in the leaves of the trees, and add the details from each to form one composite image. Again this isn’t unique to Apple: Google’s Pixel phones also have an AI-based HDR feature.

Of course, it’s difficult to gauge the novelty of each feature that’s powered by AI in the phone. Operating systems have tons of moving pieces, and something as simple as switching how an application uses the chips available in the phone could mean a massive speed boost—but logic tells us that a company trying to get the word out that it’s as AI-savvy as its competitors would have given a big Siri breakthrough more than 8 seconds on stage.