How Measurement Fails Us

The problem is more philosophical than operational. Too many of the metrics employed in contemporary management are expressions of distrust and disbelief.

Their use is based on a fundamentally negative world view rooted in desire for control grounded in fear of failure.

Measurement is a tool optimally employed to encourage mutually beneficial forward progress. JL

Robert Wachter comments in the New York Times:

Even superb and motivated professionals believe that the boatloads of measures, and the incentives to “look good,” led them to turn away from the essence of their work. We’re hitting the targets, but missing the point.

TWO of our most vital industries, health care and education, have become increasingly subjected to metrics and measurements. Of course, we need to hold professionals accountable. But the focus on numbers has gone too far. We’re hitting the targets, but missing the point.

Through the 20th century, we adopted a hands-off approach, assuming that the pros knew best. Most experts believed that the ideal “products” — healthy patients and well-educated kids — were too strongly influenced by uncontrollable variables (the sickness of the patient, the intellectual capacity of the student) and were too complex to be judged by the measures we use for other industries.

By the early 2000s, as evidence mounted that both fields were producing mediocre outcomes at unsustainable costs, the pressure for measurement became irresistible. In health care, we saw hundreds of thousands of deaths from medical errors, poor coordination of care and backbreaking costs. In education, it became clear that our schools were lagging behind those in other countries.

So in came the consultants and out came the yardsticks. In health care, we applied metrics to outcomes and processes. Did the doctor document that she gave the patient a flu shot? That she counseled the patient about smoking? In education, of course, the preoccupation became student test scores.

All of this began innocently enough. But the measurement fad has spun out of control. There are so many different hospital ratings that more than 1,600 medical centers can now lay claim to being included on a “top 100,” “honor roll,” grade “A” or “best” hospitals list. Burnout rates for doctors top 50 percent, far higher than other professions. A 2013 study found that the electronic health record was a dominant culprit. Another 2013 study found that emergency room doctors clicked a mouse 4,000 times during a 10-hour shift. The computer systems have become the dark force behind quality measures.

Education is experiencing its own version of measurement fatigue. Educators complain that the focus on student test performance comes at the expense of learning. Art, music and physical education have withered, because, really, why bother if they’re not on the test?

At first, the pushback from doctors and teachers was dismissed as whining from entitled and entrenched guilds spoiled by generations of unfettered autonomy. It was natural, went the thinking, that these professionals would resist the scrutiny and discipline of performance assessment. Of course, this interpretation was partly right.

But the objections became harder to dismiss as evidence mounted that even superb and motivated professionals had come to believe that the boatloads of measures, and the incentives to “look good,” had led them to turn away from the essence of their work. In medicine, doctors no longer made eye contact with patients as they clicked away. In education, even parents who favored more testing around Common Core standards worried about the damaging influence of all the exams.

Even some of the measurement behemoths are now voicing second thoughts. Last fall, the Joint Commission, the major accreditor of American hospitals, announced that it was suspending its annual rating of hospitals. At the same time, alarmed by the amount of time that testing robbed from instruction, the Obama administration called for new limits on student testing. Last week, Andy Slavitt, Medicare’s acting administrator, announced the end of a program that tied Medicare payments to a long list of measures related to the use of electronic health records. “We have to get the hearts and minds of physicians back,” said Mr. Slavitt. “I think we’ve lost them.”

Thoughtful and limited assessment can be effective in motivating improvements and innovations, and in weeding out the rare but disproportionately destructive bad apples.

But in creating a measurement and accountability system, we need to tone down the fervor and think harder about the unanticipated consequences.

Measurement cannot go away, but it needs to be scaled back and allowed to mature. We need more targeted measures, ones that have been vetted to ensure that they really matter. In medicine, for example, measuring the rates of certain hospital-acquired infections has led to a greater emphasis on prevention and has most likely saved lives. On the other hand, measuring whether doctors documented that they provided discharge instructions to heart failure or asthma patients at the end of their hospital stay sounds good, but turns out to be an exercise in futile box-checking, and should be jettisoned.

We also need more research on quality measurement and comparing different patient populations. The only way to understand whether a high mortality rate, or dropout rate, represents poor performance is to adequately appreciate all of the factors that contribute to these outcomes — physical and mental, social and environmental — and adjust for them. It’s like adjusting for the degree of difficulty when judging an Olympic diver. We’re getting better at this, but we’re not good enough.

Most important, we need to fully appreciate the burden that measurement places on professionals, and minimize it. In health care, some of this will come through advances in natural language processing, which may ultimately allow us to assess the quality of care by having computers “read” the doctor’s note, obviating the need for all the box-checking. In both fields, simulation, video review and peer coaching hold promise.

Whatever we do, we have to ask our clinicians and teachers whether measurement is working, and truly listen when they tell us that it isn’t. Today, that is precisely what they’re saying.

Avedis Donabedian, a professor at the University of Michigan’s School of Public Health, was a towering figure in the field of quality measurement. He developed what is known as Donabedian’s triad, which states that quality can be measured by looking at outcomes (how the subjects fared), processes (what was done) and structures (how the work was organized). In 2000, shortly before he died, he was asked about his view of quality. What this hard-nosed scientist answered is shocking at first, then somehow seems obvious.

“The secret of quality is love,” he said.

Our businesslike efforts to measure and improve quality are now blocking the altruism, indeed the love, that motivates people to enter the helping professions. While we’re figuring out how to get better, we need to tread more lightly in assessing the work of the professionals who practice in our most human and sacred fields.