The Reason Humanity Needs Experts

To help us make sure the solution is not worse than the problem. JL

Greg Satell reports in Digital Tonto:

The function of experts is not to make us more sure of our judgments, but less so. Ask a group of experts about any significant topic in their field and you are likely to get a wide range of opinions. The purpose of expertise isn’t to make things simpler for us, but to give us better information. We need experts not to give us answers, but to help us ask better questions. That’s how we can find the flaws in existing rules so that we can search for new ones. It’s how we, as non-experts, help those who look in different places consider the evidence in a new light and develop a more effective paradigm.

In a 2015 poll, 30% of Republicans and 19% of Democrats supported bombing Agrabah, the fictional hometown of the Disney character Aladdin. In a similar vein, a 2014 poll found that the less people knew about where Ukraine is located on a map, the more they wanted the U.S. to intervene militarily.

To make matters worse, another study done by researchers at Ohio State University found that when confronted with scientific evidence that conflicted with their pre-existing views, such as the reality of climate change or the safety of vaccines, partisans would not only reject the evidence, but become hostile and question the objectivity of science.

It’s become fashionable in certain circles to dismiss experts. Yet as the Coronavirus crisis has shown us, we ignore expertise at our peril. The reason we need professionals with specialized knowledge, however, isn’t so that we can outsource our decisions to them. Rather, we need experts to help us ask better questions, explore options and to make better judgments.

A World Of Increasing Complexity

It’s no secret that our world is becoming increasingly complex. So it shouldn’t be surprising that many harken for a simpler existence in which we can rely on parochial observations and good old-fashioned common sense. Complexity makes us feel powerless, because it creates a world that is difficult to understand, while simplicity makes us feel more in control.

Yet in Overcomplicated, mathematician Sam Arbesman gives two reasons why complexity is, to a great extent, unavoidable. The first is accretion. We build systems, like the Internet or the laws set down in the US Constitiution, to perform a limited number of tasks. Yet to scale those systems, we need to build on top of them to expand their initial capabilities. As systems become larger, they get more complex.

The second force that leads to complexity is interaction. We may love the simplicity of our iPhones, but don’t want to be restricted to its capabilities alone. So we increase its functionality by connecting it to millions of apps. Those apps, in turn, connect to each other as well as to other systems. Every connection increases complexity.

The simple fact is that to operate in an environment of increasing complexity, we not only need experts, but increased specialization of expertise. To point to just one example, TopTal, an expert freelance service, offers dozens of specialties ranging from mixed reality developers and UX designers to blockchain consultants and Agile coaches.

Foxes And Hedgehogs

Expertise is no panacea. In fact, in a twenty year study of political experts, Philip Tetlock found that that expert predictions were no better than flipping a coin. Further, he found that pundits who specialized in a particular field tended to perform worse than those whose knowledge was more general.

He called this phenomenon “the hedgehog and the fox” after the famous essay by Isaiah Berlin, in which a fox knows many things, but a hedgehog knows one important thing extremely well. Foxes are generalists, while hedgehogs are specialists. What Tetlock found was that, in the contest between them, the fox nearly always wins.

This is so counterintuitive that it hardly seems possible, but it’s true. The reason lies in the confidence of the predictions. Specialists, with their deep knowledge of a particular subject, tend not to incorporate information outside their domain, which makes for a cleaner, more definitive story line. They see things as simpler and are more sure of themselves.

A hedgehog claims unique insight.  A fox does no such thing. So foxes, with their broad-based knowledge, are more conscious of complexity and less confident, but more accurate. Complexity makes us more careful. It makes us look for more information and forces us to check our facts. If we see things as simple and clear, we tend to feel less need to do those things.

The Outsider’s Advantage

Thomas Kuhn explained in The Structure of Scientific Revolutions that, at some point expertise runs its course. As the world changes and evolves, flaws in existing models become more and more evident, eventually becoming untenable. That’s what sets the stage for a paradigm shift. “Failure of existing rules is the prelude to a search for new ones,” he wrote.

Yet new paradigms almost always need to be championed by outsiders or newcomers rather than acknowledged experts. Or, as the physicist Max Planck put it “a new scientific truth does not triumph by convincing its opponents and making them see the light, but rather because its opponents eventually die, and a new generation grows up that is familiar with it.”

In my book Mapping Innovation, I showed how data and real-world experience bear this out. On the innovation platform Innocentive, problems tend not to be solved within the domain in which they arose, but by a practitioner in an adjacent field. For example, when a problem has the chemistry world stuck, it’s usually someone like a biologist or a physicist that solves it.

Yet even in the context of breakthroughs, experts have a role to play. Researchers at Northwestern University analyzed nearly 18 million scientific papers and found that the most highly cited work most often comes from a highly focused team of specialized experts working with an outsider. That combination of deep domain expertise and outside thinking is often what produces major breakthroughs.

Leveraging Expertise

We all like to have opinions. If we are asked whether we should bomb Agrabah or send the Marines into Eastern Ukraine, we want to have our voice heard. Yet our desire to make judgments has nothing to do with our ability to make wise choices. In fact, studies have shown that confidence in our judgments is inversely correlated to their accuracy.

So the function of experts is not to make us more sure of our judgments, but less so. Ask a group of experts about any significant topic in their field and you are likely to get a wide range of opinions, which can be maddening. After all, if the experts can’t agree on a clear answer, then what’s the point of specialized knowledge?

Yet the purpose of expertise isn’t to make things simpler for us, but to give us better information. Any expert in foreign policy can tell us that Agrabah does not, in fact, exist, just as subject matter experts can explain the complicated history of Russian-Ukrainian relations or how Coronavirus is transmitted and what it will take to develop a vaccine.

That’s why we need experts. Not to give us answers, but to help us ask better questions. That’s how we can, as Kuhn put it, find the flaws in existing rules so that we can search for new ones. It’s how we, as non-experts, can help those who are look in different places, consider the evidence in a new light and, hopefully, develop a more effective paradigm.