Validate your experiment here..

A news blurb about “The Reproducibility Initiative” caught my attention a few days ago. Started by cancer researcher Elizabeth Iorns, this program offers scientists a means of validating their experiments at outside laboratories.  Validation tests are performed by one of more than 1000 expert providers (including VT for some tests) through the Science Exchange.  The tests are performed blind so there are no worries about “stealing” ideas or results and are provided on a fee-for-service basis.  Certificates are provided for experiments that are found to have reproducible results.  This type of validation has apparently been endorsed by the likes of Nature and PLOS One.

(image from The Science Exchange)

Is this type of validation the wave of the future in scientific research?  Will all studies be subject to outside review at this level?  What are the benefits?

  • Anomalous results would be highlighted by the verification tests.
  • May help to improve the quality of published research.
  • Gives outsiders an independent verification that data is correct.  The public doesn’t have to solely trust in affiliation, funding source, publication reputation, etc. (Corburn p.67) to assure them.
  • Consortia like Science Exchange give even small organizations access to high-tech research level tests and knowledge without having to own the equipment and expertise themselves.

Are there drawbacks to such rigorous validation?  Probably, yes.

  • It doesn’t solve most issues related to the science-public power dynamic.
  • Requiring validation would be inherently distrustful, not to mention expensive.
  • Core ethical issues of how to practice science are not addressed, rather honesty is forced through policing of results.

Any thoughts??

4 thoughts on “Validate your experiment here..

  1. Fascinating. I didn’t know about this. I wish you could elaborate on your point about the science-public power dynamic. One question: what do you think about turning science into an enterprise that is based on distrust rather than trust? Might that benefit science? Hurt it? Both?

    • In class and through our Street Science readings, I think you have done well to show us that everyone is better served when science relinquishes some of its power and gives the public a place at the table. I’m afraid that this validation concept could reinforce the science-public divide in some ways. It could be taken to imply that only other scientists are capable of “validating” research, leaving the public to watch on the sidelines. I’m not saying that this is the intent of those providing these services, just that it is a danger to be considered.

      Your other question about science becoming an enterprise based on distrust I think pinpoints my biggest qualms about this service. This service could lead to a state where all results were considered suspect until outside verification is performed, which I think would hurt scientific enterprise in the long run. It seems as if an “innocent until proven guilty” philosophy better serves everyone. On the other hand, some form of accountability is sorely needed because as I’ve stated in earlier posts people are prone to wrongdoing.

      To avoid a culture of distrust, I think validation service might best be used by scientists when they know their results are going to be controversial or seem anomalous, but not as a routine practice. It also seems useful in arenas such as journal discussions. Let researchers publish without outside validation. If another researcher can show reason to question the results in a formal discussion, then maybe the next step is to perform validation and settle the matter between the two parties.

  2. This is really interesting. I think it is a good idea, to an extent. For instance, this could help eliminate the possibility that a researcher might lie and fabricate data, because the experiment must be reproducible. However, not all experiments can fall under this. Many take years to complete and the design requirements cost money and lots of time. Who would fund something like this, just to make sure that results are able to be reproduced? It seems like a waste of time to some researchers. It brings up the idea that researchers are not professional enough, and their work must be checked up on.

  3. I think this idea has merit for many endeavors. The costs of bad science are much, much higher than anyone realizes, and can be measured in lost lives, misallocated resources, bad products, wild goose chases, rewarding the guilty and punishing the innocent. The notion that someone might replicate your results would keep us on our toes.

Leave a Reply