From The New York Times:
A chemistry professor at Columbia University who in March retracted two papers and part of a third published in a leading journal is now retracting four additional scientific papers.
The retractions came after the experimental findings of the papers could not be reproduced by other researchers in the same laboratory.
It’s a problem if published experiments are not reproducible — but what kind of problem it is might not be clear yet.
Sometimes experiments aren’t reproducible because they didn’t really happen (i.e., the results are fabricated), but sometimes they aren’t reproducible for reasons that are far less sinister.
Maybe some key feature of the procedure wasn’t made clear enough in the “Materials and Methods” section, or in the laboratory notebooks. (Why? Maybe it was just one of those things the experimenter did as part of the routine, but that didn’t seem relevant to the experiment working. Or, maybe it was part of the routine that the experimenter assumed would be obvious to other scientists doing the same kind of experiment.)
Maybe the original experimenter had better technique in the lab than any of the people trying to reproduce the results in question. Experiments are a pain to get running. Some people are more precise with the mixing and the measuring and the callibrating than others. I’ve written about “magic hands” before. So has the Pluripotentate, who can tell you from experience that scientists are sometimes resistant to following someone else’s protocol exactly — even though this is usually what’s required to be able to replicate their results.
What’s difficult about this particular case is that the papers in question were retracted by one of the authors, the former graduate advisor of another of the authors (who, of course, did the experiments in question). And, the former graduate student-author disputes that the papers ought to be retracted, since she got the experiments reported in those papers to work. (According to an earlier article in the New York Times, she wasn’t informed by her former advisor that there was even a problem with the articles, let alone that they were being retracted.)
So, there’s some disagreement among the authors about whether the experiments in question are problematic or not. From the June 15 NYT article:
The professor, Dalibor Sames, was the senior author of all the papers in question. Another author, Bengu Sezen, a former graduate student of Dr. Sames who received her doctorate last year, performed most of the experiments described in the papers.
In a telephone interview, Dr. Sames said efforts to reproduce the original findings had failed. “We have re-examined Ms. Sezen’s work extremely carefully,” he said. “Each experiment was performed by at least two independent, fully qualified scientists.”
Dr. Sames added: “I would not do anything like this lightly. To retract a paper is very difficult for any scientist.”
From the March 18 NYT article:
A former graduate student at Columbia University whose research is at the center of three papers retracted by a leading chemistry journal is disputing accusations that her experiments cannot be reproduced.
The former student, Bengu Sezen, who finished her doctorate last year and left the university, said in an e-mail message on Thursday that she had not known of any controversy about the papers until a reporter asked her about it.
Dr. Sezen said that before the papers were published, other scientists in the group successfully performed the same experiments, even when she was not in the laboratory. …
Dr. Sezen said in her message that “I was unaware of the situation until I received your e-mail” and that neither Dr. Sames nor anyone else at Columbia had consulted her about the retractions.
Although the reporter’s query did not list the three articles in question, Dr. Sezen noted them in her reply. She said that other researchers in Dr. Sames’s group had reproduced the experiments with the same results reported in the two retracted papers. She listed dates when she said she was out of town while her colleagues performed the experiments. For the third, partly retracted paper, Dr. Sezen said her role was “limited to an intellectual one.”
“I preserve copies of experimental data which supports the original claims of these publications,” she wrote. “I am also prepared to perform the reactions under supervision of Prof. Sames.”
Sames says others in his laboratory haven’t been able to reproduce Sezen’s experiments. Sezen says others in Sames’ lab already have reproduced them — and she’s willing to come back and perform the experiments herself under Sames’ supervision. No word on whether folks in other laboratories have tried to reproduce these experiments yet.
Sezen’s consternation here is understandable. The retraction of these papers seems to cast aspersions on her experimental competence, or on her integrity. But if it’s true that Sames didn’t contact her about the problems, that’s fishy.
Given the importance of reproducibility to the scientific enterprise, maybe we need to start thinking about what sort of burden of proof needs to be met before new findings are reported — and what kind of burden of proof needs to be met before we declare findings irreproducible.
Lots of peer-reviewed papers are wrong for subtle (and sometimes not-so-subtle) reasons. This wrong-ness is part of the progress of science, and doesn’t (in my view) mean a paper should be retracted. When we can’t reproduce someone else’s results, that’s worthy of another publication, not the retraction of the first.
But I guess the fact that this is internal to the same lab changes things a bit. Still, I can’t help but feel that if I found something wrong, I’d publish an erratum or something, rather than retracting the paper. Retraction seems to imply something fishy.
Reading your article I would not want to be one of Dr. Sames’ students as I wouldn’t trust anyone who would get a paper retracted without contacting co-authors especially when it comes to reproducibility of work. I’ve a personal anecdote that demonstrates how the smallest thing can make a project non-reproducible.
In my case a graduate student in our lab had developed a methodology to create vesicles that were being used in an experiment to test pore-forming chemicals. Unfortunately, our technicians could not replicate his findings which caused consternation in the lab and a delay in publication of the results. As a new graduate student my untainted eyes were assigned the task of watching the work being completed to figure out what was going wrong.
In the end the result was incredibly simple. The methodology, as developed by the grad student, called for the vesicle solution to be sonicated for 30 minutes. The grad student had chosen 30 minutes not for any technical reason but rather because that is how long it had initially taken him to assemble the equipment for the second phase of the experiment. However, after doing the experiment several dozen times he had become much quicker at setting up the second phase and when I observed him he completed the work in 15 minutes. The tech, on the other, hand followed the original procedure precisely and waited the full 30 minutes each time. When I pointed out to the grad student that he had only been sonicating for 15 minutes he was astounded, he had taken it for granted that it took him 30 minutes to set up the second stage. Later study based on the finding determined that a 15 minutes increase in sonication resulted in substantially smaller vesicles which had very different properties.
Now imagine a scenario where the tech is following the original protocol alone, she would have failed to get the result time and again, not because the protocol was wrong but because it had changed in an idiosyncratic way, unique to my colleague. Once the change was identified the work once again became fully reproducible.
So the problem arises, how do we ensure reproducibility? The answer seems to be that if the published protocols fail one must try to contact the person who conducted the original work. Dr Sames failed this simple second step and I must agree with you that the fact that Dr. Sames did not contact the original worker is a very bad sign.
Oh, my goodness! There is almost certainly something sinister, even socio-pathological behind a story like this. It sounds like desperate absolution of responsibility on behalf of the mentor or organizer of the study and the institution represented since he/she was senior author and ultimately responsible. A resignation would be more sincere and a penalty to those in the host institution linked at all levels to such a system. A recent study I believe showed that even retracted papers continue to be cited for long periods of time anyway, frequency depending on the perceived reputation of the research group. And even count, both the original and retraction on the pub list of the author’s to an unaware evaluator This is another symptom of the dysfunctional anti-scientific socio-political distribution of support, credit and information system that is in desperate need of reformation. In respect to biological sciences, the most novel and innovative conceptual science is terribly difficult to reproduce in the short term and only uncovered by the most dedicated, persistent and skilled of researchers. The reason behind that great discoveries are not recognized for years and years later. Documentation of duplicative trivia of which the great majority of current activity in the biological sciences in the scramble for publication numbers has grown to at present is very easy to replicate. But—-
[Despite everything, I believe that people are really good at heart–Anne Frank]
It occurs to me that one relatively easy way around this problem would be supplied by having computer-connected cameras filming all activity in the lab. Hard drive space is cheap these days.
Unfortunately, I’m not quite sure what happens to the archived footage. See, I’d want to be on hand and control any video that I appear in. I strongly suspect others would feel the same way. This becomes a problem when multiple people then are working on different experiments in the same lab.
We’re talking convenience-store-surveilance-tape-type footage, not The Real World: Sames Lab, right? Because those reality-show edits can be pretty cruel.
Pingback: What are honest scientists to do about a master of deception? | Adventures in Ethics and Science