A Blog by Jonathan Low

 

Nov 9, 2018

Artificial Intelligence Makes Video Calls Better

As video overtakes voice and narrative, the quality of that communication becomes more important. It is starting to happen. JL

Khari Johnson reports in Venture Beat:

Video calls have gone (become) standard, but often with poor clarity that leads to frustration. For a lot of people periodic video calls to check in with family elsewhere in the world (are) deserving of video with little latency and clear communication. Smart Camera recognizes people in a video call, so every shot is framed based on the people in that shot. The camera is able to cover 140 degrees of view, so if you can see the Portal camera, it can see you. Smart Volume enhances calls by ensuring your voice is consistently heard, whether you’re standing in front of the device or across the room.
More than two years ago, Facebook tasked a team with creating a piece of hardware to integrate Facebook’s family of apps more closely into people’s lives. The result of that effort is Portal and Portal+, 10-inch and 15-inch devices, respectively, that were designed for making AI-enhanced video calls with Facebook Messenger.
They are available for the first time this week.
Portal responds to basic voice commands for making video calls and playing music and comes with Amazon’s Alexa.
Here are some thoughts on Facebook’s first piece of consumer hardware, which competes with similar smart display devices, like the Amazon Echo Show or Google Home Hub.

Video calls

The first thing to note after about a week with a Portal+ is that the AI-powered enhancements Portal brings to video calls are not just a novelty — as advertised, on-device machine learning makes video calls better.
Smart Camera is AI that anonymously recognizes people in a video call, so every shot is automatically framed based on the people in that shot. The camera on both Portal devices is able to cover 140 degrees of view so no matter where you are in the room, if you can see the Portal camera, it can see you.
Smart Volume also enhances calls by ensuring your voice is consistently heard, whether you’re standing in front of the device or across the room.
Facebook's 10-inch Portal (left) and 15-inch Portal+ (right)
Above: Facebook Portal (left) and Portal+ (right)
These two features are a big plus if you make video calls with a laptop or smartphone and are used to having to constantly reposition the camera or repeat what you’re saying because you were too far away from the microphone.
These are also features currently unavailable from other smart display competitors.
Smart Camera can be disorienting for people, however. On my first call with a friend, he said: “It looks like you’re in the same room but creepy.”
Another feature that sets Portal apart from almost any of its competitors is its in-call experiences. These include augmented reality effects and filters like the kinds available today on Messenger — so you can put a cat on your head or toss on a cool hat and shades.
There’s also Storytime, a collection of half a dozen five-minute stories.
Today they’re all for kids, but the Portal team at Facebook will invite third-party developers to build experiences with AR for video calls, which will introduce experiences adults can do together, as well a larger Storytime collection.

These features can be a lot of fun when you’re calling people close to you and they highlight the fact that Portal was made for calls to your favorite people, not as a firehose for Facebook activity.In-call experiences also include the ability to listen to streaming music like Spotify with a caller, but you both have to be using Portal devices for this sort of thing to work.
Another feature that sets Portal apart from its competitors is the ability to quickly sling a call from Messenger on Portal to Messenger on your smartphone.
This is important when you need to move away from Portal and is really useful for privacy. For example, during one call with a close friend, the conversation began to drift to a sick family member and his relationship with his girlfriend. Since there was another person in the room with me, I quickly moved the conversation to my phone by tapping a button in the Messenger app, and I left the room.
After Cambridge Analytica and repeated privacy breaches, the question for a lot of people is whether Facebook can be trusted to enter their home, though the same could be asked of Google and Amazon.
Using a device so closely associated with Facebook reminded me that — like a lot of people — I don’t use Facebook as often as I used to. It also made it clear how many of my friends have deleted their accounts or stepped away from Facebook.
Beyond core family members, and even though Facebook Messenger has 1.3 billion monthly active users, it was a challenge at times to get even close friends on the phone.
Even though calls are encrypted and there are no recordings, I don’t think the privacy issue will be made easier by the fact that Portal devices, as previously reported by Recode, use information about Portal usage habits to sell ads on Facebook’s apps.
To address privacy concerns, Facebook says Portal is unable to take or share any photos or video recordings during calls, or even to take screenshots. On the downside, this makes for a far less capable device and takes away the option of sharing interactions in social media.

Social media (or the lack thereof)

When it’s not in use, Portal device deploys something called Superframe, displaying photos from your Facebook profile and recent photos of friends that appear in stunning clarity on the device’s screen. Superframe also recommends friends you can call, though these suggestions were never able to nudge me to make extra calls.
Portal requires you to manually pick your favorite friends one by one, and their images are then included in Superframe.
This is rather different from the carousel of news articles and reminders Amazon has on its home screen or the Live Albums that smart displays with Google Assistant inside are able to use. After five or 10 minutes of watching the Superframe, I got a sense of how much I’ve missed from my core group of friends who still use Facebook, but while I enjoyed catching up, the content began to repeat itself rather quickly.
It was also frustrating that I couldn’t double tap the screen or use generic Portal voice commands to like a photo or comment or to scroll through a friend’s latest Instagram posts. The omission of such social elements on a device made by the biggest social media company on Earth is perplexing, to say the least.
Showing me awesome recent photos by friends and family without giving me the ability to like or comment is a bug, not a feature.
The lack of Stories for Portal at launch is also confusing. Facebook’s facial recognition software could have been extremely helpful for switching between accounts for multiple people in a household, though I’m pretty sure this would have made the heads of privacy advocates explode and could have potentially made Portal dead on arrival.
One feature Facebook — as well as the makers of ambient displays for Google and Amazon — could consider making is a physical gesture to remove a photo from an album. You don’t really have control over which photos your friends are tagged in, so when photos appear on the screen that may not be suitable for kids, you need a quick way to address that beyond skipping to the next photo with a swipe.

Sound

Of course, the gadgetry, AI, and quality display are all for nothing if the Portal can’t cut it when playing music, the top use case for smart speakers everywhere.
The Portal delivers sound quality and output comparable to a Sonos One or Amazon Echo Show, providing a mixture of rich quality, crisp tones, and bass that makes using the Spotify app on Portal fun. But in contrast to playing music on an Echo or Home speaker, you will not be able to treat your phone like a remote control.
Portal’s ability to recognize voice commands was a bit underwhelming. In comparison to Google Assistant or Alexa, music requests weren’t easily understood.

Final thoughts

If you’re a dedicated Facebook user who spends a lot of time keeping up with friends on the app, you may find Portal is worth the price of admission.
Whether you find Portal valuable or not may depend heavily on whether you’re the kind of person who makes a lot of family video calls. Portal allows video chats with up to seven people, so families could consider getting one to share a common digital photo album and make calls. And the ability to move freely around a room while still being seen and heard has inherent value.
In my lifetime, video calls have gone from non-existent to standard, but often with poor clarity and choppy footage that lead to frustration. For a lot of people, including me, periodic video calls to check in with family members elsewhere else in the world have become a routine deserving of a device that delivers video with little latency and clear communication.
People have a lot of choices when it comes to ways to make video calls, including Skype, the Alexa app on Amazon’s Echo Show, and Duo on Google smart displays. The Lenovo Smart Display even provides a portrait mode option like Portal+, though of course lacking things like Smart Volume and Smart Display.
Overall, you will have to ask yourself whether the calling feature is a big enough plus for you. The answer may also depend on whether you already have something like a Google Home or Amazon Echo.
Beyond making calls, I’m excited to see if working with third-party developers can improve Portal over time and stitch together an ecosystem of in-call experiences. At launch, the Portal+ has some noteworthy elements, but it will likely be a lot more interesting six months to a year from now when more third-party apps and in-call experiences are made available.
Most people may be better off waiting for more apps, in-call experiences, and elements from Facebook’s family of apps to be included.

0 comments:

Post a Comment