The February 2023 issue of IEEE Spectrum is here!

Close bar

Study Shows Ensuring Reproducibility in Research Is Needed

The IEEE Computer Society suggests improvements

3 min read
 Illustration of 6 people, some holding large check marks or Xs, looking at a giant clipboard with notes on it.

About 60 percent of IEEE conferences, magazines, and journals have no practices in place to ensure reproducibility of the research they publish. That's according to a study by an ad hoc committee formed by the IEEE Computer Society to investigate the matter and suggest remedies.

Reproducibility—the ability to repeat a line of research and obtain consistent results—can help confirm the validity of scientific discoveries, IEEE Fellow Manish Parashar points out. He is chair of the society's Committee on Open Science and Reproducibility.

"Ensuring the robustness and trustworthiness of science that is done using computing and data is absolutely critical," Parashar says.

The inability to reproduce the results of an experiment can lead to a range of consequences, including the retraction of research and injury to the reputations of the authors and the journal that published it.

For example, after a string of failed attempts to replicate the results of a study on preventing fraud in policy review and insurance claim forms, a request was made in August to the Proceedings of the [U.S.] National Academy of Sciences to retract a 2012 paper on the research. According to the study, when people signed an honesty declaration at the beginning of an insurance policy form, rather than the end, they were less likely to lie about the information they provided. Insurance companies, private organizations, and government agencies then adopted the seemingly inexpensive and effective method to reduce fraud. But other research teams could not confirm the finding, and an anonymous group of scientists then found evidence suggesting that the original experiment used fabricated data.


The goal of the ad hoc committee's study was to ensure that research results IEEE publishes are reproducible and that readers can look at the results and "be confident that they understand the processes used to create those results and they can reproduce them in their labs," Parashar says.

"Ensuring the robustness and trustworthiness of science that is done using computing and data is absolutely critical."

The committee's international membership spans academia and national laboratories as well as representatives from IEEE leadership.

To get a better sense of the issue, the group surveyed more than 100 IEEE journals, magazines, and conferences and analyzed the reproducibility models and practices they use. The findings were published online.

Here are three key recommendations from the report:

  • Researchers should include specific, detailed information about the products they used in their experiment. When naming the software program, for example, authors should include the version and all necessary computer codes that were written. In addition, journals should make submitting the information easier by adding a step in the submission process. The survey found that 22 percent of the society's journals, magazines, and conferences already have infrastructure in place for submitting such information.
  • All researchers should include a clear, specific, and complete description of how the reported results were reached. That includes input data, computational steps, and the conditions under which experiments and analysis were performed.
  • Journals and magazines, as well as scientific societies requesting submissions for their conferences, should develop and disclose policies about achieving reproducibility. Guidelines should include such information as how the papers will be evaluated for reproducibility and criteria code and data must meet.


The report covers roadblocks researchers and journals could face when ensuring reproducibility, such as the allocation of responsibilities and economic issues, as well as ways to overcome them.

Also included is an overview of an ongoing pilot being conducted by IEEE Transactions on Parallel and Distributed Systems. It is offering incentives such as "reproducibility badges" to authors for making code and data available for reuse along with their publications.

The Conversation (1)
Don Carter04 Jan, 2022

Reproducibility—the ability to repeat a line of research and obtain consistent results—can help confirm the validity of scientific discoveries..

Needs to be changed to "- is foundational to science"

without reproducibility, it isn't science.

Get unlimited IEEE Spectrum access

Become an IEEE member and get exclusive access to more stories and resources, including our vast article archive and full PDF downloads
Get access to unlimited IEEE Spectrum content
Network with other technology professionals
Establish a professional profile
Create a group to share and collaborate on projects
Discover IEEE events and activities
Join and participate in discussions
Illustration showing an astronaut performing mechanical repairs to a satellite uses two extra mechanical arms that project from a backpack.

Extra limbs, controlled by wearable electrode patches that read and interpret neural signals from the user, could have innumerable uses, such as assisting on spacewalk missions to repair satellites.

Chris Philpot

What could you do with an extra limb? Consider a surgeon performing a delicate operation, one that needs her expertise and steady hands—all three of them. As her two biological hands manipulate surgical instruments, a third robotic limb that’s attached to her torso plays a supporting role. Or picture a construction worker who is thankful for his extra robotic hand as it braces the heavy beam he’s fastening into place with his other two hands. Imagine wearing an exoskeleton that would let you handle multiple objects simultaneously, like Spiderman’s Dr. Octopus. Or contemplate the out-there music a composer could write for a pianist who has 12 fingers to spread across the keyboard.

Such scenarios may seem like science fiction, but recent progress in robotics and neuroscience makes extra robotic limbs conceivable with today’s technology. Our research groups at Imperial College London and the University of Freiburg, in Germany, together with partners in the European project NIMA, are now working to figure out whether such augmentation can be realized in practice to extend human abilities. The main questions we’re tackling involve both neuroscience and neurotechnology: Is the human brain capable of controlling additional body parts as effectively as it controls biological parts? And if so, what neural signals can be used for this control?

Keep Reading ↓Show less