I’m not sure how many academics read EngineerBlogs, but I’m curious how many have run into the problem of false results. I personally spent a lot of time trying during my MS trying to reproduce data from various papers only to find there were problems with the results or the set up. I can remember one paper, in particular, that showed an antenna design published by a fairly prestigious group. I used exactly the same software and tried to replicate their design only to find that the results were considerably different than what was shown in the paper. I finally broke down and emailed one of the authors. Their response was that the dimensions given on the design were not correct. The problems are multiplied, of course, if you’re dealing with different software as you may not be able to even get your simulation results to match up. How do you know your differences aren’t a result of the computational algorithms?
I was very interested, therefore, when I heard about an effort to create more reproducibility in science. On Tuesday, ECN published an article about Science Exchange’s “Reproducibility Initiative.” While it appears the initiative is geared toward social and medical sciences, the idea is that Science Exchange will find a lab that can repeat experiments to validate the reproducibility of the results. Ideally, the cost to reproduce the results will be only 20% of the cost of doing the original study.
I think this implies a lot about how the funding/publishing system is broken. What funding agency is really willing to go through and fund several groups to do the same work? Part of the reason there is a push to do so much ‘new research’ and so little effort to verify reproducibility is because there’s no money to do it. It also doesn’t get you very far in terms of original publications, if you can get it published at all: negative results aren’t often viewed as favorable by a lot of reviewers.
Personally, however, I think this is a great idea. It is most certainly in the interest of academic researchers to have governmental agencies fund these validation studies. When you compare it to how much money is probably wasted by other groups attempting to replicate false results, which is a significant chunk, according to the article, it seems that increasing funding by 20% is small change.
The cynical part of me thinks the initiative is doomed to failure because of financial issues. Where should that additional 20% come from? Many corporations will spend money to pay their own people to replicate experiments, but are they willing to pay additional taxes instead of replicating the research themselves? If industry doesn’t want to help pay for it, where should the money come from? Or should we continue to leave the priority to replicate these in the hands of industry and let academic researchers suffer?