Covid Preprint Increase Shows Medical Research Can Be Sped, Opened Up

Sharing of early research has saved lives and changed the way such analysis is conducted and reported. JL 

Jeffrey Brainard reports in Science:

COVID-19 did show that in a crisis, scholarly communications can be sped up and opened up. Early analyses suggest less progress toward providing “open data,” the sharing of data sets with other scientists to speed the reuse of data and increase transparency of research. Among more than 7000 medRxiv preprints about COVID-19 published during 2020, authors of only about half offered to readily share the underlying data. (But) the pandemic prompted an avalanche of new papers: more than 530,000, released either by journals or as preprints. 

In January 2020, as COVID-19 spread insidiously, research funders and journal publishers recognized their old ways wouldn’t do. They needed to hit the gas pedal to meet the desperate need for information that could help slow the disease.

One major funder, the Wellcome Trust, issued a call for changing business as usual. Authors should put up COVID-19 manuscripts as preprints, it urged, because those are publicly posted shortly after they’re written, before being peer reviewed. Scientists should share their data widely. And publishers should make journal articles open access, or free to read immediately when published.

Dozens of the world’s leading funders, publishers, and scientific societies (including AAAS, publisher of Science) signed Wellcome’s statement. Critics of the tradition-bound world of scientific publishing saw a rare opportunity to tackle long-standing complaints—for example, that journals place many papers behind paywalls and take months to complete peer review. They hoped the pandemic could help birth a new publishing system.

But nearly 2 years later, hopes for a wholesale revolution are fading. Preprints by medical researchers surged, but they remain a small fraction of the literature on COVID-19. Much of that literature is available for free, but access to the underlying data is spotty. COVID-19 journal articles were reviewed faster than previous papers, but not dramatically so, and some ask whether that gain in speed came at the expense of quality. “The overall system demonstrated what could be possible,” says Judy Luther, president of Informed Strategies, a publishing consulting firm.

One thing is clear. The pandemic prompted an avalanche of new papers: more than 530,000, released either by journals or as preprints, according to the Dimensions bibliometric database. That fed the largest 1-year increase in all scholarly articles, and the largest annual total ever. That response is “bonkers,” says Vincent Larivière of the University of Montreal, who studies scholarly publishing. “Everyone had to have their COVID moment and write something.”

Helping drive the explosion was a sudden turn to preprints among medical researchers. Various servers accept papers directly from authors, posting them days later after a few basic checks but no peer review; most later appear in journals. A leading server, medRxiv—itself a newcomer, founded only in 2019—posted about 200 preprints on all topics in January 2020; by May 2020, the monthly tally had swelled to more than 2000, about three-quarters of them about the pandemic coronavirus. Traffic to medRxiv skyrocketed in tandem, with abstracts viewed a dizzying 6.1 million times in July, up from 30,000 in August 2019. Some preprints delivered findings that physicians immediately put to use at patients’ bedsides. News that the corticosteroid dexamethasone reduced deaths in severe illness was published as a preprint in June 2020, but not in a peer-reviewed journal until February. By then, doctors had been giving the steroid for months.

By embracing preprints, medical researchers played fast catch-up to physicists, who have relied on them for decades—the original preprint server, arXiv, began to post physics papers in 1991.

A running concern about preprints in medical science had been their lack of vetting—but the pandemic made clear that peer review was not infallible either. High-profile, peer-reviewed journal articles about COVID-19 were retracted; suspicions arose whether data were fabricated for one that asserted hydroxychloroquine, an antimalarial drug promoted by former President Donald Trump, was not only ineffective, as many studies have shown, but could harm COVID-19 patients. “The border between validated scientific knowledge”—the turf traditionally claimed by peer-reviewed journals—“and not-yet-validated scientific knowledge [in the form of preprints] has become more blurred” during the pandemic, Larivière says.

COVID-19 also accelerated fledgling efforts to create a middle ground for preprints by putting some through a kind of rapid peer review. But those efforts remain small in scale. Rapid Reviews: COVID-19, a journal launched by MIT Press in August 2020, has quickly published online reviews of about 150 COVID-19 preprints.

But early predictions that preprints would dominate the COVID-19 literature—and reshape all scientific publishing after the pandemic—so far appear incorrect. Through May, only about 5% of all peer-reviewed journal articles about COVID-19 papers started out as a preprint, according to an estimate by Ludo Waltman and Stephen Pinfield of the Research on Research Institute, based on the Dimensions and Unpaywall bibliometric databases. (Most COVID-19 preprints have not appeared in peer-reviewed journals, at least so far.) And last year’s preprint surge is subsiding, with medRxiv’s monthly total now at about 600 papers, roughly half about COVID-19 (see graphic, below).

“Many researchers are not used to preprints and don’t get rewarded for them,” Waltman says. “Perhaps funders could have really encouraged or even mandated preprinting. If we really want [more preprints], the different stakeholders need to work together. … Otherwise, it will remain a small-scale phenomenon.”

Complementing the spurt of preprints, editors of traditional journals promised to speed up reviews of COVID-19 manuscripts. They had some success: In January 2020, manuscripts on COVID-19 and related topics were accepted about 130 days after submission, on average. In July 2020, the lag had dropped to about 90 days, according to an analysis by Daniel Hook and colleagues at Digital Science, based on its Dimensions database. (The study examined the 20% of all papers in which these dates were provided.)

Storm surge

Medical research preprints about COVID-19 grew especially fast in the pandemic’s early months.

(GRAPHIC) N. DESAI/SCIENCE; (DATA) MEDRXIV

Did that speed up blunt quality? Although comprehensive evidence is lacking, only about 0.03% of journal articles on COVID-19 have been retracted—slightly less than in all disciplines before the pandemic.

Serge Horbach of Aarhus University in Denmark dove deeper, analyzing the rigor of reviewers’ comments from two journals, The BMJ and eLife, which routinely post the comments with published articles. He examined 30 papers that appeared through late April 2020. Ten were about COVID-19; the rest covered other topics.

He found that reviewers of the COVID-19 papers frequently did not comment on whether the data or experiments were sufficient to support the conclusions, as they did for the other papers. When the reviewers did suggest improvements to COVID-19 articles, they tended to do so gently, inviting the authors to add data or experiments only if it could be done “easily” or “quickly.”

It’s “laudable,” Horbach says, “that peer review, which tends to have a high level of inertia, was able to adapt so quickly in these crisis conditions. But indeed, it does raise questions about the quality of the material” published.

As for Wellcome’s call to make COVID-19 articles and data free for all, the results are mixed. In early 2020, many publishers pledged to exempt COVID-19 papers from their paywalls, and as of last week, 77% of all such articles were free to read, according to the Dimensions database. That’s a decline from a high of 85% in May 2020, but still well above the 50% for newly published scientific literature before the pandemic.

Early analyses suggest less progress toward providing “open data,” the sharing of data sets with other scientists to speed the reuse of data and increase transparency of research. Among more than 7000 medRxiv preprints about COVID-19 published during 2020, authors of only about half offered to readily share the underlying data, according to an analysis by Larivière and colleagues.

Even though the changes were incremental, observers say COVID-19 did show that in a crisis, scholarly communications can be sped up and opened up. Larivière hopes the progress made for COVID-19 publishing can serve as a model, and be replicated for other global crises. One possibility: climate science, where only about 50% of new articles are now open access.

“Many people consider climate change a much more important threat than COVID, over the long term,” Larivière says. “We need to treat it in the same manner.”