Eight years ago, a team of researchers launched a project to carefully repeat early but influential lab experiments in cancer research.
They recreated 50 experiments, the type of preliminary research with mice and test tubes that sets the stage for new cancer drugs. The results reported on Tuesday: About half the scientific claims didn’t hold up.
“The truth is we fool ourselves. Most of what we claim is novel or significant is no such thing,” said Dr. Vinay Prasad, a cancer doctor and researcher at the University of California, San Francisco, who was not involved in the project.
It’s a pillar of science that the strongest findings come from experiments that can be repeated with similar results.
In reality, there’s little incentive for researchers to share methods and data so others can verify the work, said Marcia McNutt, president of the National Academy of Sciences. Researchers lose prestige if their results don’t hold up to scrutiny, she said.
And there are built-in rewards for publishing discoveries.
But for cancer patients, it can raise false hopes to read headlines of a mouse study that seems to promise a cure “just around the corner,” Prasad said. “Progress in cancer is always slower than we hope.”
The new study reflects on shortcomings early in the scientific process, not with established treatments. By the time cancer drugs reach the market, they’ve been tested rigorously in large numbers of people to make sure they are safe and they work.
For the project, the researchers tried to repeat experiments from cancer biology papers published from 2010 to 2012 in major journals such as Cell, Science and Nature.
Overall, 54 per cent of the original findings failed to measure up to statistical criteria set ahead of time by the Reproducibility Project, according to the team’s study published online Tuesday by eLife. The nonprofit eLife receives funding from the Howard Hughes Medical Institute, which also supports The Associated Press Health and Science Department.
Among the studies that did not hold up was one that found a certain gut bacteria was tied to colon cancer in humans. Another was for a type of drug that shrunk breast tumors in mice. A third was a mouse study of a potential prostate cancer drug.
A co-author of the prostate cancer study said the research done at Sanford Burnham Prebys research institute has held up to other scrutiny.
“There’s plenty of reproduction in the (scientific) literature of our results,” said Erkki Ruoslahti, who started a company now running human trials on the same compound for metastatic pancreatic cancer.
This is the second major analysis by the Reproducibility Project. In 2015, they found similar problems when they tried to repeat experiments in psychology.
Study co-author Brian Nosek of the Centre for Open Science said it can be wasteful to plow ahead without first doing the work to repeat findings.
Discussion about this post