How Pandemic Data, Behavioral Science and AI Lead To More Precise Online Choices

The pandemic has generated significant increases in digital engagement generally and ecommerce specifically. In addition to greater sales, this has driven an exponential accrual of personal data. 

Applying this data to enhanced analytical techniques such as machine learning, AI and natural language processing has enabled enterprises selling online to use this information for dramatically improved segmentation, focused not just on preferences, but to identify the psychological and socio-economic influences behind them so that future behavior can be predicted - and possibly modified. The question is at what point optimization becomes manipulation and sparks resentment. JL

Angus Loten reports in the Wall Street Journal:

Since coronavirus, online retailers are leveraging data from a surge in e-commerce to get better at helping customers find what they are looking for, even when they don’t know what that is, (by) building predictive models with a level of accuracy unimaginable a few years ago. Stores create more-robust customer profiles by adding ages and genders, where they live, local weather or seasonal events. Natural-language processing helps hone search results by interpreting buyers’ intentions from the terms and phrases they use. Natural-language processing, computer vision and customer data enable online retailers to better interpret a search query, identify a more precise, smaller assortment of personalized choices unique to each shopper

Wayfair Inc. W 5.92% has 37,173 kinds of coffee mugs for sale. Factor in different colors, sizes or materials, and the range of options rises above 70,000. It’s Jim Miller’s job to help shoppers find the mug they want—along with a set of espresso cups, a waffle iron or other products they didn’t know they wanted.

In the past five years, the company’s success rate has jumped 50%, measured by the number of clicks it takes for a customer to add an item to their carts and how often they buy those items, among other variables, according to Mr. Miller, the Boston-based online retailer’s chief technology officer.

He credits the gains to advances in smart software. Rather than asking customers to browse through the entire catalog of mugs, he says, algorithms, artificial intelligence and troves of data “are doing the work behind the scenes.”

Since the coronavirus outbreak, online retailers like Wayfair, Etsy Inc. ETSY 5.90% and Pinterest Inc. PINS 2.33% are ratcheting up efforts to leverage data from a surge in e-commerce to get better at helping customers find what they are looking for—even when they don’t know what that is.

To do that, these Web-only stores are supercharging search-and-recommendation engines by feeding data into sophisticated algorithms, building predictive models with a level of accuracy unimaginable just a few years ago.

Not all of the capabilities are new—algorithms have been around for decades. But the rapid expansion of computing power and cloud storage in recent years has enabled sellers to gather and crunch data on a massive scale.

Shoppers generate data on retail websites every time they place an item in a virtual cart, hover over product pages, click on product recommendations and ultimately make a purchase. Stores create more-robust customer profiles by adding their shoppers’ ages and genders, where they live, the local weather or seasonal events and holidays—and in some cases data drawn from all over the internet by third-party services.

Two greatly improved tools that most online retailers use to turn that data into sales are computer vision and natural-language processing, says Bob Hetu, a vice president and analyst in tech research firm Gartner Inc.’s retail industry services unit. The former helps to index products in a website’s virtual catalog using visual cues, while the latter aggregates and learns from words that shoppers use when describing products they are looking for. Both rely on algorithms powered by machine learning, a subset of artificial intelligence.

Where standard algorithms generate results based on instructions for very specific input, AI algorithms go further by using the results they produce to then fine-tune the instructions and “learn” how to handle new input, repeating the cycle over and over again.

Gartner predicts that within the next five years, the world’s 10 largest retailers will be using AI models as the backbone of product searches and recommendations—and as a competitive edge.

Companies keep their algorithms a closely guarded secret. Amazon.com Inc.’s AMZN 0.37% AI algorithms scour millions of listings for items matching a buyer’s search query, weighing more than 100 variables, such as past purchases, age, gender, and a long list of criteria known only to company insiders.

Online retailers use computer-vision models to automate the process of assigning keywords, or tags, to identify individual products. The systems are designed to “see” products and label them with a list of attributes, such as chair, teak, Scandinavian, and so on. A decade ago, Mr. Hetu says, online product photos were manually tagged with a handful of descriptive terms.

The problem is two people manually tagging items can look at the same chair, or garment, or bedding and pick out very different attributes, Mr. Hetu says, “or just get it wrong.”

Computer vision, also known as object recognition, recognizes dozens of features from a product photo and compares them with similar items in a store’s database, quickly tagging any overlapping attributes—like a Venn diagram that has furniture as the overall subject and black, leather and contemporary as three circles that intersect at the center.

Mr. Miller, a former vice president of Amazon.com Inc.’s supply-chain operations, says Wayfair trains its computer-vision algorithm with a combination of supplier and customer photos, and photos generated by three-dimensional imagery of a given piece of furniture.

The algorithm captures design features, materials, styles, color, vintage and a vast array of other elements. It then takes that information, along with tags added by professional designers, and applies it to similar items in the store’s extensive line of products. The richer the list of keywords tagged to a product, the more likely an algorithm is to produce relevant search results.

The model improves over time by learning from its successes and from its mistakes, such as when a mislabeled product is spotted and corrected. “You’re taking a very large catalog and parsing it and segregating it into searchable terms, to match what you want,” Mr. Miller says. When a customer enters a store with millions of products, they are going to need as much help as they can get to find the right aisle, he says.

Pinterest Inc., the 10-year-old social-media platform, has trained its object-recognition model on billions of images saved by users from around the Web, says Jeremy King, the site’s senior vice president of engineering. Many of the objects in these photos, known as Pins, are available for purchase.

“The more we know about a Pin and the objects and products inside it and their attributes, we can return more relevant results based on searches,” Mr. King says, adding that the site has enlisted fashion industry pros who make sure clothing styles are being correctly identified.

Mr. King says precise visual-search capabilities enable users to ask “what material is this made from” or “what is the color scheme of this dress,” and get answers that also direct them to similar items for sale.

Pinterest’s visual-search capabilities can also identify incidental objects within a photo, like a vase in the background of a restaurant, and make the items searchable for users who want to buy them, Mr. King says.

Natural-language processing

The other big AI tool that is revolutionizing e-commerce, advanced natural-language processing, helps retailers hone search results by interpreting buyers’ intentions from the terms and phrases they use.

“There’s a paradox of choice that really matters,” Mike Fisher, Etsy’s chief technology officer, says about the online marketplace’s 80 million-plus items for sale. “You don’t want it to be overwhelming,” he says.

Five years ago, Mr. Fisher says, search was purely based on keyword matching. This means it was easy to miss relevant merchandise when customers typed in a search query.

Today, Etsy’s search engines “learn” from a customer’s past queries as well as from other customers’ past queries and purchases. Billions of historical data points are consulted to make links between the terms shoppers have used and the products they eventually find.

This relieves shoppers of the “burden” of having to formulate “the ‘perfect’ search query,” Mr. Fisher says.

The tool also enables Etsy to map out synonyms within a hierarchy of colors. So if a customer searches for a teal rug, say, the algorithm can find products matching the query, but also options in other hues of blue, which would otherwise be buried deeper in the catalog.

Mr. Fisher says the next step will be to train algorithms to combine product and search data over time to determine a shopper’s overall taste. That requires associating an array of elements that aren’t linked in an obvious way, like flannel pajamas and midcentury décor. “The tricky part is going across chairs and shirts or jewelry to detect a similar style,” he says.

Taken together, natural-language processing, computer vision and vast stores of customer data enable online retailers to better interpret a search query, identify a more precise set of relevant products, and pull out a smaller assortment of personalized choices unique to each shopper—all just a few clicks.

Many traditional stores with a heavy online presence are also looking to AI to ratchet up search.

Walmart Inc. WMT 2.03% takes advantage of data from its physical stores and its website to build out a robust profile of customers, says Ravi Jariwala, a company spokesman.

“Because we have a massive data set between our online and offline purchases, the algorithms are incredibly well informed,” he says. According to Mr. Jariwala, most Walmart customers continue to shop in the physical stores, but a growing number are paying online for delivery, or ordering online for in-store pickup. “This is how people are shopping now and we don’t see that changing any time soon,” Mr. Jariwala says.

“So much of our shopping has drifted online and there’s a lack of tolerance for companies that aren’t getting it right,” says Jon Duke, a vice president for retail insights at tech research firm International Data Corp.

“These tools are proving their worth during the crisis,” he says.