• Dr. Timothy Smith

Do It Again! Reproducibility Matters

Woman Researching

Photo Source: Max Pixel

How can you trust the science if you cannot repeat the experiment? In order to rely on scientific conclusions, scientists require that experiments must be repeatable with reproducible results. Logic, experimentation, and observation help scientists to understand how the world works. Carefully designed experiments demonstrate the causal relationship between different agents such as the relationship between a bacterium called Mycobacterium Tuberculosis and the deadly disease tuberculosis. Correspondingly, a good experiment only takes science so far as a reliable system for understanding the physical world. The next and crucial step after running tests and drawing conclusions involves other scientists running the same tests to see if they get the same results. In other words, experiments must be reproducible for scientists to believe them. However, an explosion of scientific research over the past hundred years has produced millions of publications describing millions of experiments, but recent analyses have cast doubt on the reproducibility of many published scientific experiments.

A number of institutions and scientific organizations recently reported that many scientific experiments could not be reproduced in other laboratories or clinics. The prestigious journal, Nature, noted in a special edition titled, “Challenges in Irreproducible Research,” that a survey of 1500 scientists found that scientists are concerned about how reproducible scientific experiments are. One specific example revolves around research performed with the dietary supplement resveratrol that comes from some plants, most notably in wine made with red grapes. Researchers, Gordon J. Lithgow, and others had shown in their lab that resveratrol had extended the life of round worms by a staggering 67% over untreated worms. Such results suggested a potent life-extension property of resveratrol. However, following the publication of their findings, other labs around the world could not reproduce their impressive results. Over the next 15 years, the researchers tried to figure out why others could not reproduce their results. They found many subtleties between the labs such as how researchers handled the worms, or if the measured lifespan of a worm started at the point of egg laying or egg hatching. In “A long journey to reproducible results,” the authors admit that they still do not really know why the results from other labs do not match their own. In the area of psychology in an article in the highly respected journal Science titled, “Estimating the reproducibility of psychological science,” many researchers from institutions around the world collaborated to try to reproduce the results from 100 published psychology experiments. Their findings indicated that only 39% of the 100 experiments could be replicated.

Science builds on a foundation of discoveries and conclusions that depend in part on the reproducibility of the experiments supporting scientific breakthroughs. In the training phase of every scientific career, students work in laboratories reproducing experiments with known results under the eye of teachers to learn techniques and proper experimental design. Later, as scientists become researchers, they must design new experiments never done before. Part of scientific publication involves a careful description of the tests that others can follow to repeat them. In recent years, concern among the scientific community has arisen about the lack of reproducibility of published research. Sometimes investigations cannot be repeated because the description does not contain all the necessary details necessary to repeat them. Analogously, an incomplete cooking recipe will not produce the desired result if it does not include specific information or ingredients. Surveys and failed attempts to reproduce experiments has prompted the scientific community to push for better descriptions of tests, and in many cases, labs must now supply essential ingredients to other labs so that experiments can be more faithfully run. To be fair, some tests involve many complicated steps and improvised machines and techniques to produce results. Such complexity at times requires hands on training not conveyed in a publication alone. With that aside, science faces a point of reckoning that must demand reproducibility as an essential part of the scientific process, and the public should demand that the experiments that lead to significant scientific discoveries that affect our lives, health, and public policy should be repeatable and reproducible.

Dr. Timothy Smith

Dr. Smith’s career in scientific and information research spans the areas of bioinformatics, artificial intelligence, toxicology, and chemistry. He has published a number of peer-reviewed scientific papers. He has worked over the past seventeen years developing advanced analytics, machine learning, and knowledge management tools to enable research and support high level decision making. Tim completed his Ph.D. in Toxicology at Cornell University and a Bachelor of Science in chemistry from the University of Washington.

You can buy his book on Amazon in paperback here and in kindle format here.

How to Profit and Protect Yourself from Artificial Intelligence