People Are Not Looking At the Most Important Vaccine Statistic

That statistic is the number of people who did not have to be hospitalized and who didnt die because they were vaccinated. JL

Kelsey Piper reports in Re/code:

Efficacy continues to be the statistic that people fixate on. (But) vaccines can protect people by ensuring when a vaccinated person gets the disease, they experience a mild case and are not likely to require hospitalization or be at risk of dying.  As long as vaccines can make sure that Covid-19 isn’t very bad, we can use them to chart our way out of the pandemic. But this effect of the vaccines has been largely neglected.a shift from a focus on efficacy to a focus on improved outcomes is essential for good vaccine decision-making. That’s a difference of potentially hundreds of thousands of lives lost.

The Food and Drug Administration is working through thousands of pages of documentation for an emergency authorization of the Johnson & Johnson vaccine, which could come through this month. Most states are gradually expanding vaccine eligibility, production of the Pfizer and Moderna mRNA vaccines is increasing, and new studies show good results for additional vaccine candidates as well.

The news on the Covid-19 vaccine front of late has been quite good — perhaps so good that our perspective on the fight against the pandemic may be getting a little warped.

Case in point: the media coverage and public reception of the Johnson & Johnson (J&J) vaccine. Results from its trials were released last week. According to the company, it should be able to deliver 100 million doses in the first half of this year. But this good news hasn’t been greeted with the enthusiasm that accompanied announcements about the Pfizer and Moderna vaccines.

Perhaps that’s because in its clinical trial, the J&J vaccine had an efficacy number — the percentage of cases prevented entirely — of 66 percent. Compared against the 95 percent efficacy rate for the Pfizer vaccine and the 94.1 rate for Moderna’s, Johnson & Johnson’s vaccine simply doesn’t look as good.

But in another sense, the Johnson & Johnson vaccine trial results were incredibly encouraging. The same trial showed that J&J’s vaccine makes Covid-19 cases much milder, meaning you might still get sick but you are much less likely to be hospitalized or die. Indeed, on that front, the J&J vaccine performs just as well as Pfizer’s and Moderna’s, a fact that seems to have been undersold in news coverage about it.

To put J&J’s effectiveness in another context, think about the flu. Flu vaccines mostly don’t prevent you from getting sick with the flu but instead make the flu much less awful if you catch it and less fatal for at-risk populations. A Covid-19 vaccine that was similar to that — one that made you much less likely to be hospitalized or die, and made the disease milder — would still be enough to help bring the pandemic to an end and give us back our lives.

That’s exactly what we have with the current crop of vaccines, including the three that haven’t been received as well as Pfizer’s and Moderna’s. No one who got two doses in clinical trials for the Moderna and Pfizer vaccines has been hospitalized or died, and the same holds true for the new vaccines from Johnson & Johnson, Oxford/AstraZeneca, and Novavax (which haven’t yet been approved in the US). Each of the vaccines has demonstrated very high protection against severe Covid-19 in trials.

So why aren’t we talking about that?

How “efficacy” became the measure of vaccine success

The vaccines developed against Covid-19 can protect people in three ways. First, they can make the vaccinated person less likely to catch the disease. That’s called “efficacy.” It’s the number that made the headlines when Pfizer and Moderna results came out this fall — 95 percent and 94.1 percent efficacy, respectively. For every 20 people in the control group to develop symptoms of Covid-19, only one in the vaccinated group did.

That’s an incredible achievement, and it’s what’s gotten most of the attention in the reporting on the vaccines. But it has overshadowed discussion of the other two ways a vaccine against Covid-19 can protect people.

The second is that the vaccines seem to make vaccinated people less likely to transmit the disease; new research suggests the Pfizer and Moderna vaccines do this, too, though imperfectly (so you should still be cautious around unvaccinated people).

And the third way the vaccines can protect people is by ensuring that when a vaccinated person does get the disease, they experience a mild case and are not likely to require hospitalization or be at risk of dying.

This last finding is a big deal. As long as vaccines can make sure that Covid-19, even if you get it, isn’t very bad, we can use them to chart our way out of the pandemic. But this effect of the vaccines has been largely neglected.

Now, there’s a reason headlines have focused on the efficacy of preventing all disease rather than severe disease: It’s what the earliest clinical trials were primarily set up to study.

Every clinical trial defines in advance a primary clinical endpoint: the outcome researchers are primarily focused on studying. The study is designed to have a large enough sample size to detect differences in this primary outcome between the trial group and the placebo group. For the vaccine trials, the primary clinical endpoint has largely been symptomatic Covid-19 infection (however mild or severe), or, in some cases, a positive Covid-19 PCR test.

It makes sense for the studies to set that endpoint. Designing a study to measure rare events like hospitalization or death is much harder than designing a study to measure relatively common events like infection. Covid-19 is a scary disease, but less than 1 percent of people who get it will die from it. That means that a study designed with death from Covid-19 as a primary clinical endpoint would need to be enormous, with potentially hundreds of thousands of participants. Pinning down the exact frequency of rare events requires a lot of data, and vaccine developers could only include so many people in their first clinical trials because recruiting and coordinating participants is expensive and time-consuming.

So instead, most studies primarily measure how many cases of Covid-19 the vaccines prevent. They do collect data on hospitalizations and deaths, but it’s not the primary outcome their studies are designed to measure, and that has been reflected in press releases, media coverage, and the popular understanding of how well the vaccines work.

As it happens, that data on hospitalizations and deaths looks really good. Indeed, there have been zero cases of hospitalization or death in clinical trials for all of these vaccines. But we hear about that less partly because the studies weren’t designed with that outcome in mind.

This focus on preventing cases, rather than preventing cases as well as hospitalizations and deaths, may have led to the sentiment that the new, unapproved vaccines are worse, and that it’s okay that the US isn’t expediting their approval and distribution because people would rather get the better vaccines anyway. This line of reasoning would be understandable if some vaccines are 95 percent effective at preventing hospitalization and death and some are only 60 percent effective. But in this instance, it’s a mistake, since all of the vaccines are indistinguishably effective by this metric.

The focus on studying efficacy, instead of studying hospitalization and death, affects major policy calls.

South Africa suspended its rollout of the AstraZeneca vaccine after a new study found that the vaccine had very low — potentially zero — efficacy against the variant strain of the disease that predominates in the country. Obviously, that finding was really bad news for the AstraZeneca vaccine. But less widely broadcast was the fact that no one in the study got seriously ill or was hospitalized from Covid-19 (no one in the placebo group did, either; the study was small and didn’t contain at-risk people).

Now, let’s be clear: While other studies have convincingly demonstrated that the AstraZeneca vaccine protects against hospitalization and death, it seems to work less well against the South African virus variant, and that might well mean it works less well to prevent hospitalization and death as well. One crucial caveat here is that the study population was young; we would not have expected many hospitalizations or any deaths within that age group. So it could be that the AstraZeneca vaccine has very limited usefulness against the new strain. Or it could be that the AstraZeneca vaccine offers very strong protection against severe disease from the new strain. The study was not set up to answer that question, so we simply don’t know.

In that regard, South Africa’s decision to suspend the AstraZeneca vaccine rollout based on the discouraging data makes some sense. But countries that have focused on hospitalization and death statistics, rather than efficacy statistics, have made the opposite call.

“I think it’s important for people to bear in mind that all of them, we think, are effective in delivering a high degree of protection against serious illness and death, which is the most important thing,” UK Prime Minister Boris Johnson said, defending Britain’s continued rollout of the AstraZeneca vaccine.

Countries face tough choices here, and their leaders would be able to make a vastly more informed decision if there had been a study with sufficient statistical power — a lot more participants — to detect whether the vaccine is protecting against hospitalization and death. As it is, the understandable focus on efficacy simply doesn’t give us much information on this essential question.

The communications challenge of vaccine statistics

In the past few weeks, some public health experts have started working to change the conversation on vaccines to focus on the key hospitalization and death numbers. One person leading this effort is Ashish Jha, dean of the Brown University School of Public Health, who tweeted on January 31:

One note about this data: It refers only to the clinical trials. Now that the vaccine is out in the world, we’re getting real-world data. The news is still good: 700,000 people in Israel have been vaccinated with the Pfizer and Moderna vaccines, and there have in fact been a few hospitalizations — 16, to be precise, for a rate of 0.002 percent — and no deaths.

Jha’s message was widely shared, but efficacy still continues to be the statistic that people fixate on.

Communications about severe disease and hospitalization face an uphill battle. Efficacy is much easier to explain — do you get sick or not? For severe disease, on the other hand, many studies have slightly different benchmarks for severity. And while right now there have been zero hospitalizations and zero deaths by anyone vaccinated in the clinical trials for any of the Pfizer, Moderna, Novavax, AstraZeneca, and Johnson & Johnson vaccines, it doesn’t mean those outcomes are guaranteed. (Again, in Israel, 0.002 percent have been hospitalized so far — that’s very, very low, but it’s not zero.)

Furthermore, as the pandemic stretches into its second year, we’re all desperate for certainty. “This study simply didn’t collect enough data to tell us what we hoped to learn” is not the answer any of us are looking for. The efficacy of the vaccines is known; their effectiveness against hospitalization and death is known to be very high but not pinned down for certain, especially not against the South African variant. All of those pose extraordinary challenges for public health officials and the media to communicate.

But a shift from a focus on efficacy to a focus on improved outcomes is essential for good vaccine decision-making. Moderna and Pfizer can’t produce enough mRNA vaccines alone to end the pandemic, so the public embrace of the Johnson & Johnson vaccine — and potentially the AstraZeneca and Novavax vaccines, which we’re waiting to learn more on — might make a huge difference for whether things start to get back to normal in the US in April or in August. That’s a difference of potentially hundreds of thousands of lives lost.