Why Economic Markets Don't Work Like They Used To Due To Big Data

Personalized pricing and customized sales pitches, driven by big data-fueled algorithmic marketing is changing the way markets respond to economic incentives.

The question is whether this is really any different from the way merchants haggled for a thousand years, or whether, at scale, it is affecting how markets perform. JL

James Plunkett reports in Medium:

Technology is starting to affect the market economy; the methods by which prices are set and customers are acquired and retained. Big data now allows firms to personalise prices, asking different customers to pay different amounts for the same thing. Consumer-facing companies are running massive, live behavioral experiments, in which they can optimise the design of products and services, maximising the chance that consumers part with their money.

The relationship between the market and the state is one of the oldest questions of public policy and, in the last a year or so, it has re-entered political debate. There’s a growing sense that new technologies mean markets don’t work quite like they used to. This is re-energising longstanding arguments about when, how, and on what basis, the state should intervene.

Although it’s easy to overstate these developments, I do think something real is going on here. Technology is starting to affect important aspects of the market economy, such as the methods by which prices are set and the way customers are acquired and retained. These changes are only just emerging but they still deserve careful attention from policymakers — in the long run, I suspect they’ll require quite a different policy and regulatory regime.

Big data and price discrimination

To unpack this claim, let’s start by looking at prices. In the front-running sectors of our economy, big data now allows firms to personalise their prices, asking different customers to pay different amounts for the same thing.

In its purest form, this marks a break from how prices were typically set in the mass consumer markets of the 20th century. In those decades, we grew used to the idea that the price is a characteristic of the product itself, rather than of the person buying the product. Prices reflect intuitive things like the cost of manufacture and the relative scarcity or abundance of raw materials.

Price discrimination, in which prices also reflect the amount each individual consumer is willing to pay for a product, has long been an idea for the classroom rather than the boardroom. It makes sense in theory but it hasn’t been all that widespread or salient in our day-to-day lives as consumers.

As it turns out, the reasons for this were largely practical.

For one thing, until now, consumer-facing companies simply didn’t know each customer’s magic ‘reserve price’ — the maximum price they would be willing pay. Companies could safely assume that some customers would be willing to pay more than others, but they couldn’t tell who was who. As a result, all customers were charged the same price for each product — in effect, a small and hidden subset of price sensitive people kept prices low for everyone else.

On top of this, even if companies had known which customers were willing to pay more, it would have been impractical and uneconomic for them to act on that information. Imagine changing your price tags, or reprinting your menus, for each new customer that walked in the door. Even if you could rig up a system to do this, the cost and complexity involved wouldn’t justify the benefits.

The result is that price discrimination has long been confined to the margins of consumer markets. Prices do vary sometimes, for example, you might pay more for the same can of beans in Belgravia than you’d pay in Bognor, or in a Tesco Express compared to a Tesco superstore. Prices also tend to vary more in a sector like second hand cars, where haggling survives. And prices vary in indirect ways, such as through well-targeted promotions, or schemes like the Young Person’s Railcard. In most general cases, however, we all the pay the same as each other.

A new era of big data — and new pricing possibilities

In the last 10 years, big data has melted away both of these constraints. Tech-savvy companies can now use data analytics and experimental pricing methods to find the reserve price of specific subgroups of customers. Having done that, they can also act on that information, identifying a specific customer with cookies left on their laptop or phone, and automatically presenting them with an entirely personalised price.

Before we fall into a breathless ‘this changes everything’, it’s important to say these are early days. Opinions differ on the extent to which price discrimination is happening in practice (as this summary shows). And some studies suggest purer forms of price discrimination are not yet widespread.

Even so, differentiated pricing is certainly happening. Long before last week’s story about price discrimination in insurance, online retailers had been found charging different customers different prices for the same item. There’s also evidence of search discrimination, in which customers are shown different search results despite typing the same words into the search box, steering them toward a more profitable choice.

It’s also clear that price discrimination is becoming dramatically more profitable. That’s because big datasets, like a customer’s browsing history, are far better predictors of a person’s price sensitivity than the simplistic demographic data on which companies used to rely. This is spawning an entire industry to advise companies on how to optimise their prices, covering everything from petrol to apps — in their words, these firms are there to help companies ‘align’ their prices to a “customers’ willingness to pay”.

For a simple illustration of the sophistication of modern pricing strategies, just look at how prices now move over time. The chart below shows the price of the board game Monopoly on Amazon over a 10 month period. Prices no longer rise or fall in annual or quarterly increments. They move hourly, like stocks, as an algorithm optimises profit and customer insight over time.

The price of the board game Monopoly on Amazon

Source: http://uk.camelcamelcamel.com/

Price discrimination’s winners and losers

What happens to our lives as consumers when prices vary in these ways? You might worry about a dramatic scenario in which prices fragment, and we each suddenly pay vastly different amounts for the same goods and services.

This extreme scenario — so-called ‘pure price discrimination’ — isn’t really plausible. In practice, competition limits the extent to which companies can raise prices for particular groups of customers. And some economists argue that this effect dominates to such a degree that price discrimination won’t really happen in a big way: some firms will experiment with tailored prices, but then competition will pull them back to an old-school model of competing on one uniform low price.

What does seem plausible, however, is a middle scenario, somewhere far short of perfect price discrimination but still noticeably different to where we are today. Even strong competition won’t save us from this; after all, the whole point of price discrimination is that companies will get better at identifying the customers who are not price sensitive . As a result, that small group of price sensitive people, once hidden among the overall customer population, won’t pull prices down for everyone in the way they once did. Plus, of course, competition is far from perfect, particularly in markets for essential services.

So the direction of travel seems clear: as big data becomes ubiquitous, prices will better reflect our individual willingness to pay.

That means, all else being equal, people who are insensitive to higher prices — the rich, the time-poor, or those with little choice in the moment (think of being stuck at an airport, or in the last minute pre-Christmas rush) — will pay more. Meanwhile, people who are highly sensitive to price — the careful shoppers, the plan-aheaders, the savvy app-users — will more often pay less.

More worryingly, prices will also better reflect the cost of serving individual customers. It will be easier for firms to identify high cost customers — people who might not pay back their debts, for example, or who have costlier needs — and charge them more. This latter effect matters most in insurance, because it weakens the market’s function as a risk-pooling mechanism. It also means prices could vary by characteristics like education level, age, gender, or even ethnicity. And that explains concerns, whether justified yet or not, that discrimination is about to re-enter consumer markets via the backdoor.

The second half of the story: companies can also shape consumer behavior better than ever before

Price discrimination is an interesting issue on its own. But to understand the full implications of big data for our lives as consumers, you need to couple it with a second trend: the growing power of behavioural insights.

If big data helps companies to optimise their prices, it also helps them to optimise the design of their products and services, maximising the chance that consumers will part with their money.

Of course, the idea of nudging consumers is nothing new. Retailers are old-masters of the cleverly-structured decision: just think of the sweets and gum by the till, to encourage impulse purchases, or the 3-for-2 offer that means you leave the store with 3 items when you only went in for one.

What is new, however, is the unprecedented sophistication that big data allows — and therefore its subversive effects. Today’s big, consumer-facing companies, are just as much data companies as they are retail or FMCG manufacturers. They are running, in effect, massive, live behavioural experiments, in which products and services can be tweaked, and the effect on consumer behaviour can be closely monitored and understood.

This all helps to explain an uncomfortable feeling you might have had: that our lives as consumers are now strongly flavoured by frustration and regret.

Think, for example, of that free trial you signed up for in a matter of seconds, that has now become a monthly subscription, and that is really burdensome to escape. Or think of the slow and clumsy switching process that leaves you sticking, grudgingly, to your current deal. Or the refund that’s technically possible but that’s such a hassle to claim that you don’t bother, even though you’re dissatisfied.

These aspects of customer service are no less carefully designed than any other. And they now flavour our lives as consumers just as strongly as those sweet moments, thankfully also still common, of charming customer service or flawless product design.

Price discrimination and behavioural nudges: A dangerous combination

So big data marks a sea change in firms’ capabilities on price strategy and product and service design. And it’s when you combine these 2 capabilities — price optimisation and behavioural nudges — that you start to see some pernicious and significant effects.

A good example is the penalty consumers now pay for inertia. In sectors from energy to telecoms to banking, companies now pursue a harmful business model that could be called ‘bait and squeeze’. This means they compete fiercely with an attractive acquisition price, tempting people to sign up to a contract; then they maximise the chance that people will auto-renew their contract by mistake; and then they hike the price for that disengaged group, charging them more for exactly the same service.

You don’t need to be a cynic to think this is happening; there’s nothing going on here beyond the simple maximisation of profit. And the chart below shows how this feeds through into the numbers, in this case in Britain’s energy market. The gap between competitive acquisition tariffs (the bottom of the grey area) and default Standard Variable Tariffs (the green line) grew to such an unsustainable level that the government felt it had to intervene to cap prices.

Source: Competition and Markets Authority. Based on typical domestic consumption of 3,200 kwh/year for electricity and 13,500 kwh/year for gas. Other costs include transmission, distribution, BSUoS and environmental and social obligations.

Yet although this problem has been most salient in the UK energy market, it goes far further than this. In Britain’s broadband market, for example, prices jump 43% on average at the end of a fixed-term deal — more than they do in energy. In mobile phones, the same effect is achieved by continuing to charge customers for their handset even after it’s paid off. In mortgages, the roughly 1.2 million mortgagors on standard variable rates pay hundreds of pounds more than they did under their fixed-term deal.

Again, none of this is to imply sinister motives; in these industries, bait and squeeze has simply become the equilibrium pricing strategy, making it hard if not impossible for any one company to price in a different way.

When incentives play out in this way, some really egregious outcomes become normal. Take the picture below as an example — it’s from an insurance renewal letter I received a few weeks ago. It proposes a 50% year-on-year price increase. No-one — literally no-one — would choose to pay this amount. (And a 5-minute phone call brought the increase down to 2%). But that, of course, is the point. The company in question doesn’t expect anyone to choose to renew; they expect people to auto-renew without thinking. And it works: around 12.9 million UK households auto-renew their home insurance after one year. Worst of all, vulnerable households are most likely to fall into this trap.

What should be done?

1. The principles at stake

This gives us a good sense of the problem. So how should the state respond to to these trends, both through the government itself and through independent regulators?

To start with, although we should avoid hyperbole, I do think these developments complicate some long-standing principles of consumer policy in interesting ways.

Consider the idea of ‘regret’. It comes up again and again in the examples above: the free trial you regret not cancelling, the exorbitant insurance you regret auto-renewing, the extortionate out-of-bundle mobile phone charges that mean you regret taking out that ‘£20 a month’ contract. What these experiences have in common is the irksome feeling that you didn’t really ‘choose’ the outcome at all, despite being technically free to do so.

Situations like these complicate the orthodox defence of free consumer markets — because they complicate the idea of consumer choice itself.

When a consumer market works well, with low levels of regret, consumer choice is a powerful concept. It lets you argue that each trade that takes place in the market necessarily adds value for the consumer — after all, if the person didn’t value the product they bought more than its price, why did they buy it?

When behavioural tricks come into play, however, that argument goes weak at the knees. Does the fact that you’re paying for a monthly subscription, after a free trial period ended, mean you want that subscription? What if you haven’t switched because your supplier has made it a hassle? Are these purchases adding value to you as a consumer? That’s suddenly a complicated question. And, if the answer’s no, then the ‘choices’ consumers are making, in ostensibly free and competitive markets, aren’t necessarily raising their utility. And that undermines a longstanding premise of regulatory decision-making.

2. The more practical challenge

So these developments should prompt a healthy reappraisal of the more orthodox principles that sit behind consumer policy.

But they also raise far more practical challenges. How do you govern behavioural nudges in otherwise competitive markets? And what do you do when competition itself is the locus of these tricks — in other words, when companies are competing not over ‘who has the best product?’ but over ‘who can most effectively nudge consumers into spending money they later regret?’

Insurance auto-renewal is again a case in point. In recent years, it became increasingly clear that insurance companies were actively nudging their customers to auto-renew by mistake, and then hitting them with massive year-on-year price increases. Auto-renewal letters had begun to sound like adverts for a spa weekend: ‘Sit back and relax — you don’t have to do anything’.

To stop this deception, the FCA now prescribes certain aspects of the auto-renewal letter. They require, for example, firms to show customers the price they paid last year alongside their renewal price (hence my letter above). Now, though, a new battle has started: some companies are gaming the system by showing each price on a different page of the letter, or using different calculation methods so that the figures are hard to compare.

This crystallises an uncomfortable choice now facing regulators:

Option 1: Play the game, setting increasingly detailed rules to stop companies playing misleading behavioural tricks, and then continually amend those rules to stop each new trick.

Option 2: Step back, scrap the detailed rules, and switch to a principles-based approach, requiring companies not to violate general ideas like ‘misleading customers’ or ‘exploiting vulnerable people’ — and then take enforcement action against those that do.

The latter is, overall, the path regulators are taking and, in the long run, it is surely the only viable option. Even so, this shift to a principles-based approach is a big change and one that will be hard to get right.

3. The shape of a new settlement

So, with all this mind, what could a new regulatory settlement ultimately look like? What tests will it need to pass?

For starters, it’s vital not to overreact.

These developments aren’t yet all that widespread, and they might take a long time to go mainstream. Moreover, big data doesn’t just create risks for consumers: it also creates huge opportunities, not least by disrupting incumbents. Were the government to oversteer, that could stop these benefits from playing out, harming consumers in the long-run.

A new approach must therefore not squander the hard-won benefits of today’s regulatory regime — not least the clarity and predictability of the rules. That would create uncertainty, hindering innovation and investment and, ultimately, harming consumers. This also means expectations need to be managed: if the government ends up being held accountable for prices in consumer markets, that would invite ever more significant interventions, and that would be a dangerous path to take.

Second, although this means policy choices should be careful and evidence-based, they should still be bold.

If the government or regulators don’t move quickly enough, and consumer outcomes get worse as technology continues to improve, particularly in essential markets, people could lose faith in the system. As the energy industry has shown, that is the surest way to end up with policy uncertainty. It is better to act quickly and decisively than to wait for the pressure to build.

There’s also no reason to wait. Although there’s uncertainty about how these trends will play out, the direction of travel is clear: big data will become more powerful and ubiquitous; the speed of computers and communications networks will soar and their cost will plummet; mobile devices will take over; retail will continue to shift online. This means we already know roughly what tomorrow’s consumer markets will look like — in short, they’ll look a lot like the tech sector today — so we can prepare for this outcome now.

Moreover, whether markets work better or worse in this likely future, it’s seems clear they’ll work differently in some basic ways. For example, prices will likely move closer to the reference price of individual consumers, meaning they’ll vary more between different groups. Behavioural nudges will become more sophisticated, making it more difficult to claim that a person’s purchase of a product or service necessarily ‘reveals their preference’ for that product or service. Some industries will get locked into harmful equilibriums, in which there’s more money to be made perfecting behavioural tricks than perfecting products, and that will harm consumer outcomes and distort investment.

Put simply, today’s settlement isn’t yet ready for this world. As Martin Wolf put it in a recent op-ed on technology monopolies: “policymakers must get an intellectual grip on what is happening”. To do that, they will need to think boldly and not be overly constrained.

Third, although the details of the response will take time to develop, the rough outlines of a new settlement are starting to emerge. As a starter for ten, my sense is that at least 3 broad shifts will be needed:

1. A greater willingness to intervene, in targeted ways, based on clear rules, to limit extortionate prices for subgroups of consumers. The emphasis here should be on protecting vulnerable people, particularly in markets for essential services.

This entails a greater acceptance that behavioural market failures sometimes justify supply-side interventions, even when a market seems competitive. In other words, if it’s clear a majority of consumers are behaving in a certain way, and have been for many years, policymakers shouldn’t just try another nudge to get them to engage — sometimes bolder action will be needed.

As an example, this would suggest tougher action on an issue like bank overdraft fees — acknowledging that information remedies are unlikely to bite. It also, interestingly, fits with the government’s welcome pledge to ban letting agent fees, which will shift the burden of fees off tenants (who don’t shop around on the basis of fees) onto landlords (who do).

What’s interesting about these new approaches is that they’re both more interventionist and more pro-competition — the state intervenes, based on insights into how consumers are behaving in reality, and this intervention then increases competitive pressure on prices.

2 . A general rebalancing away from highly prescriptive regulatory rules toward broader principles, or outcomes-based regulation, backed by tough enforcement. This avoids an arms race in which regulators prescribe company behaviour in ever more detail to stop ever cleverer behavioural nudges. It also shifts the responsibility of compliance onto industry, and off of the regulator; instead of the regulator having to check that every company has ticked every compliance box, the companies themselves have to weigh up whether they’re confident they’re acting within the broad principles set out by the regulator.

3 . A fuller and stronger set of horizontal institutions to support well-functioning consumer markets. These cross-cutting institutions will become increasingly important as the boundaries blur between sectors (for example, as intermediaries start to bundle services like energy and telecoms together).

There are several elements to this third category, including, for example:

• Strong and cross-cutting consumer advocacy to make sure consumers are well-represented in important decisions, balancing the well-funded lobbying power of industry. This includes filling advocacy gaps like the one in telecoms.
• Rules to make data more open, so that intermediaries can access data on things like tariffs and pricing, as well as rules to put data into consumers’ hands.
• Mechanisms to support closer cooperation between sector regulators, particularly on complex issues like price settlements, where regulators face formidable institutional challenges and where decisions therefore often favour the interests of producers over consumers.
• A strong and consistent landscape of Alternative Dispute Resolution (ADR), fixing today’s ragged patchwork, so that consumers can get easy and quick redress when a company does push things too far.

Fourth and finally, any full response to these trends requires more than a technocratic solution; there are political debates to have too.

Stepping back, the subtle changes we’re seeing to consumer markets are just one small aspect, among many others, of growing unease about how mature market economies function. This sits alongside other concerns, such as the material crisis in living standards and the changing nature of work.

When it comes to consumer policy, there are, of course, unhelpful versions of this political debate: an ‘anti-business’ agenda pitched against a ‘pro-market’ agenda, for example. But a more nuanced conversation could help to surface and test public views on important questions. What outcomes do we expect from well-functioning consumer markets? What are the ethics of practices like price discrimination? What role do we want the government to play and what role falls to independent regulators? These are political questions as much as they are economic or technocratic ones, so they deserve a well-informed and public debate.