Scientific studies that are cited by lots of other papers are more likely to show evidence of bias, according to new research.
Small studies, the first studies to find a particular effect, studies by researchers who are early on in their career, and studies by authors who had previously had to retract papers were also more likely to overestimate the effects they showed.
The authors said the study supported previous research into bias in science, and suggested that work to determine and reduce bias was looking at the right problems.
The study, by scientists at Stanford University and published in the Proceedings of the National Academy of Science, looked at meta-analyses – research that combines the results of smaller studies. The 3,000 meta-analyses it examined encompassed nearly 50,000 studies across 22 fields of science.
Dr Daniele Fanelli, a researcher at the Meta-Research Innovation Centre at Stanford and the lead author on the study, told BuzzFeed News that overall, the findings were reassuring. "Our main conclusion is that you can’t say bias is hampering science as a whole," he said. "For most of these biases the effects are reasonably small, although they’re more than I expected."
"Studies of publication bias suffer from publication bias."
Previous "meta-research", the study of science itself, has been focused on individual fields of science, said Fanelli. "It's a hot field, but everyone's interested in the biases of their own field," he said. "Our main objective was to get an impression of the whole literature." In its present fragmented state, the field itself is ironically prone to problems: "Studies of publication bias suffer from publication bias," said Fanelli. "The real mission was to draw samples at random, from all fields of science – to the extent that's possible – and see how common these biases are."
Part of the reason why highly cited articles might be less reliable, said Fanelli, is that surprising, exciting results tend to be more highly cited – but, of course, one reason that a result might be surprising is that it's wrong, or at least overestimated. Sometimes a study will "find" some dramatic effect just by statistical fluke, and cause great excitement, only for later studies to look at it and find something less impressive. "We do have some evidence that that's true," said Fanelli. "We found consistently that the first studies [of a phenomenon] find something strong and significant, but subsequent studies show something else."
The papers most likely to show bias were small studies with few participants – unsurprisingly, since larger studies have less random "noise" in the data. There are perfectly good reasons to do small studies, said Fanelli: "It might make sense in a preliminary study, before you invest in a large study."
The trouble is that that other scientists – and, especially, science reporters – may treat these small, pathfinder studies as trustworthy. "The responsible, scientifically accurate way to report on science is to be aware of the background literature and put it into that context," he said. "If it's reporting evidence of an exciting new finding, it should be treated with great caution. If it's the fifth big study and they consistently report a phenomenon, you can be surer."
"The responsible way to report on science is to be aware of the background literature and put it into context."
"This is very impressive work," Chris Chambers, a professor of psychology at Cardiff University who is interested in bias and the practice of science, told BuzzFeed News.
It's not surprising that highly cited studies are more likely to be biased, he says. "Bias is like car polish," he says: It makes studies shine. "Journals like shiny cars, scientists like shiny cars. But the problem is a lot of the cars are lemons."
He says that a partial solution to the problem is for scientists to write down their hypotheses in advance, so that studies are judged on the quality of their theory and methods rather than the results they produce. "This is the so-called Registered Reports model of publishing and may hold the key to solving many of these problems," he said. Fanelli said that the move to preregistration of hypotheses is "very exciting, updating the scientific method to the 21st century", but added that the problems of science are diverse, and that preregistration can only be a partial solution.