Independent confirmation and open inquiry (investigation? examination?): Purdue University and the Rusi Taleyarkhan case.

My recent post on the feasibility (or not) of professionalizing peer review, and of trying to make replication of new results part of the process, prompted quite a discussion in the comments. Lots of people noted that replication is hard (and indeed, this is something I’ve noted before), and few were convinced that full-time reviewers would have the expertise or the objectivity to do a better job at reviewing scientific manuscripts than the reviewers working under the existing system.
To the extent that building a body of reliable scientific knowledge matters, though, we have to take a hard look at the existing system and ask whether it’s doing the job. Do the institutional structures in which scientific work is conducted encourage a reasonable level of skepticism and objectivity? Is reproducibility as important in practice as it is in the scientist’s imagination of the endeavor? And is this a milieu where scientists hold each other accountable for honesty, or where the assumption is that everyone lies?

The allegations around nuclear engineer Rusi Taleyarkhan — and Purdue University’s responses to these allegations — provide vivid illustrations of the sorts of problems a good system should minimize. The larger question is whether they are problems that are minimized by the current institutional structures.

This article in ScienceNOW gives a nice overview of the situation. In a 2002 paper in Science, Dr. Rusi Taleyarkhan reported his dicovery of “sonofusion”, a process in which “sound waves can collapse bubbles in a manner that causes atoms to fuse, releasing energy”. Remember how cold fusion was going to maybe solve all our energy problems? This was another one of those findings that seemed like it might have that sort of big practical payoff.
Minimally, the finding Taleyarkhan reported challenged the body of knowledge fusion experts accepted. If it was true, the understanding of fusion would require significant revision. Or, possibly this was a too-good-to-be-true result that wouldn’t hold up under scrutiny. There is an ideal in scientific practice that the community should meet new results with organized skepticism, not taking them on faith or on authority, but checking them out to see if the reported results can be reproduced. In this case, people made the usual noises about how an extraordinary claim of this sort required extraordinary proof.

The finding would be more credible if it could be replicated — especially by some scientist other than Taleyarkhan, someone who hadn’t already staked his reputation on “sonofusion” being a real phenomenon. The search for independent confirmation was not immediately successful, however — so much so that the lack of replication (among other things) got the attention of Rep. Brad Miller (yes, that Brad Miller) and the House Committee on Science and Technology. Here’s how it’s described in the letter prepared by the staff of the Investigations and Oversight Subcommittee of the House Committee on Science and Technology:

Except for the claimed confirmation in the Purdue laboratory, no other researchers have been able to independently replicate Dr. Taleyarkhan’s experiments, including researchers from three universities working under a grant from the Defense Advanced Research Projects Agency (DARPA). (p. 1)

More about that “claimed confirmation in the Purdue laboratory” in just a moment. Later in the letter:

That publication [of the issue of Science in 2002 with the paper by Taleyarkhan et al. announcing the “sonofusion” finding] noted in an accompanying editorial the concerns of other researchers at Oak Ridge National Laboratory — where the original research had taken place — because they had not been able to replicate the experiment. Several other scientists openly pointed to problems with the underlying data. The controversy only increased when the three peer reviewers of the article for Science broke their silence and said they had advised that the article not be published because potential sources of error had not been ruled out. Since that time, no independent confirmation of sonofusion has occurred. In 2005, a private consortium funded by Impulse Devices, Inc., was formed to confirm Dr. Taleyarkhan’s work. After spending $4 million, it abandoned the approach. (p. 4)

Stipulate that replication of results is hard. It takes time, money, and expertise. But if serious scientists in a field apply lots of time, money, and expertise to trying to replicate a result and they can’t replicate it, what ought we to conclude? Should we assume they’re a bunch of bumblers, or should we look more skeptically at the results that are resisting replication?

Of course, we can note that established scientists in a given field have a stake in things. Even if there are not professional jealousies at work, or a desire not to share the limelight with some up-and-coming scientist, the established scientists have likely staked their careers on research that rests on the current understanding of how things are in that field. In Kuhnian terms, they see the world through the lens of their paradigm, so they will likely be resistant (if not downright dismissive) of a finding that challenges that paradigm. So perhaps the graybeards are not as objective and open-minded in evaluating new and surprising results as we might hope.

In that case, who can give the new finding a fair hearing? A newish researcher in the field might not be so wedded to the dominant paradigm — but would she have sufficient expertise to be able to reproduce the finding or to demonstrate persuasively that it is irreproducible?

Indeed, Taleyarkhan claims that his results have been reproduced, pointing to an “independent confirmation” in a lab at Purdue. However, given that the purported replication was performed by a postdoc and a grad student of Taleyarkhan — in his lab — the “independence” of this “confirmation” is in some question. And, according to the Investigations and Oversight Subcommittee letter, it gets worse:

The two publications at the center of this dispute were the May 2005 “Confirmatory Experiments for Nuclear Emission During Acoustic Cavitation,” published in the journal Nuclear Engineering and Design (NED) and the paper entitled “Bubble Dynamics and Tritium Emission During Bubble Fusion Experiments” presented at the 11th International Topical Meeting on Nuclear Reactor Thermal-Hydraulics (NURETH-11) in October 2005. Both were presented as “independent” confirmation of work done previously at Oak Ridge National Laboratory by Dr. Taleyarkhan. The research was partially funded by the Defense Advanced Research Projects Agency (DARPA). In August of 2005, one of the stduents listed as a co-author on both of those papers came to the head of the School of Nuclear Engineering and said he had nothing to do with the papers and wasn’t even assigned to the laboratory at the time it was done. He also alleged that Dr. Taleyarkhan had added his name to the NED paper a week before it was submitted for publication and he was not aware that his name was on the second paper until a week before the conference.

As a result of this allegation and prodding from other faculty members, in February of 2006 the head of the nuclear engineering school — who had already raised concerns with Purdue officials about the research and these publications — established an informal “Fact-Finding Committee” within the school to determine the process by which these particular papers came to be published. When that committee reported back, included in its report was a statement by the graduate student that made clear that he had made no intellectual contribution to the paper, and his name had been added to the paper by Dr. Taleyarkhan, his laboratory supervisor, just before it was submitted to the journal.

The student’s first-hand statement to the Fact-Finding Committee was presented in its report as was the information that the other co-author, a post-doctoral researcher assigned to Dr. Taleyarkhan, refused to state who had written the final draft of the article, saying it would compromise the claim of independence. (pp. 4-5)

Regular readers of this blog will have noticed clues that I view dishonesty about who is responsible for a scientific paper and for the piece of research it purports to describe as a very bad thing. If it is the case that Taleyarkhan added the name of the grad student who did none of the research and wrote none of the papers to those papers, that in itself would be problematic. If he did it to support a (false) claim that his own controversial findings had been independently replicated, that is far worse. Maybe the postdoc whose name is also on these papers has some other good reason to keep quiet about who actually drafted the manuscrcripts, but it’s hard not to suspect that Taleyarkhan himself might have done it — in which case “independence” is pretty much off the table.

Indeed, Purdue’s Inquiry Committee flagged the authorship issue as a big concern around Taleyarkhan’s work. Here’s how the findings are described by the Investigations and Oversight Subcommittee letter:

1. Dr. Taleyarkhan diplayed “what might be characterized most favorably as a severe lack of judgment” regarding his involvement with the “independent confirmation” experiment performed by a post-doctoral student and a master’s student working with him. Although the committee inexplicably failed to address the specific allegation that Dr. Taleyarkhan actually wrote the draft submitted for publication, it did find that he suggested “the inclusion of various technical details, text used to communicate with journal editors, and rebuttal of points made by referees.” The “sum total of this involvement,” the committee determined, “undermines the claim of independent confirmation.”
2. The committee could not understand why a graduate student, who said he had nothing to do with either of the papers or the underlying research, was listed as a co-author of both.
3. Dr. Taleyarkhan’s claim that the measurements by the post-doctoral researcher constituted an independent confirmation of his earlier results did not include these critical elements: a) a different critical gamma-ray detector with different calibration curve than was used in the earlier experiment; b) researchers not affiliated with Purdue and “certainly not individuals having close relationships to Dr. Taleyarkhan;” and c) minimal involvement by Dr. Taleyarkhan.
4. Dr. Taleyarkhan placed junior scientists in “precarious positions” in order to promote his research program and “abused his position as a senior scientist.”
5. It is “highly doubtful” that other claims by Dr. Taleyarkhan of independent confirmation of his results will be accepted by the scientific community and were “representative of poor judgment” by Dr. Taleyarkhan.

(pp. 7-8)

If I were a scientist in Taleyarkhan’s field, I don’t think I’d trust him. No matter what the facts behind his original findings, the reports of these efforts to manufacture “independent confirmation” undercut his credibility rather severely.

Despite having a “Policy on Integrity in Research” that defined research misconduct as “fabrication, falsification, plagiarism, or other practices that seriously deviate from those that are commonly accepted within the scientific and academic community for proposing, conducting, or reporting research,” however, Purdue’s Inquiry Committee in the case decided that there was insufficient evidence of research misconduct by Taleyarkhan to warrant a full investigation.

Let that roll around in your head a moment. The committee found that Taleyarkhan misrepresented his own participation and that of his grad student and postdoc in these papers that he was hyping as independent confirmation of his earlier work — but this does not seriously deviate from commonly accepted practices?

Did the committee here get confused about the distinction between what some scientists get away with and what scientists as a community understand that they ought to do in order to build a reliable body of knowledge and a community of scientists whose work they can depend upon?

The Investigations and Oversight Subcommitteeraises a set of concerns about whether Purdue correctly followed its own procedures for investigating and responding to the allegations of misconduct against Taleyarkhan. I’m not an administrator, so I’m not going to try to untangle the policies to make a judgment on whether the letter of the law was followed. However, I have a few comments to make about the spirit of the investigation.

The situation, at least as described by the Investigations and Oversight Subcommittee, is one where nuclear engineering faculty had voiced concerns about Taleyarkhan’s results for some time. There are reports that a dean actually put pressure on some of Taleyarkhan’s colleagues to withdraw a paper they had co-authored that challenged the plausibillity his findings, and that this dean directed them to stay clear of fusion (Taleyarkhan’s “turf”) henceforth. Meanwhile, when the purported “independent confirmation” was published, the university announced it in a press release, even though the fact that the authors were working in Taleyarkhan’s lab should have raised some question about its independence.

At various stages of Purdue’s response to concerns about Taleyarkhan, the administration apparently spun it as a case of conflicting personalities and clashing egos. It would not be surprising if personalities did become part of the issue, but this would not mean that the concerns raised by scientists with macroscopic egos were concerns with no substance. Dismissing complaints as solely based in personal disagreements would be a pretty good way to cripple trust within the school of nuclear engineering. If the administration won’t take our scientific concerns seriously — if they want us to shut up about them so they can bask in the glow of favorable press releases — the clearly their interests depart from our own. If we want to build reliable knowledge in our field, and to identify persons and behaviors acting as impediments to that, the administration isn’t going to be much help, no matter what their official policy on integrity in research might say.

The confusion about whether the official process was followed is really part of this alienation of the faculty from the administration in this case. The unclarity about the authority and responsibility of the “fact-finding” and “examination” committees, the question of whether the “examination” was performing what the policy described as an “inquiry”, and of whether an official “investigation” had been launched, are all tied to uncertainty about how committed the institution was to rooting out bad behavior rather than covering it up, excusing or minimizing it, or simply hoping it would go away. They make faculty fearful that administrators regard this sort of situation as personel matters rather than matters of the honesty and integrity that is essential for productive interactions between members of a discipline.

If truth matters more than public relations, the faculty need to see more evidence of this institutional commitment.

Given that Purdue is launching a further inquiry into the concerns around Taleyarkhan’s work, the administration still has the opportunity to align its actions with its commitments. As noted by the Investigations and Oversight Subcommittee:

It will be a challenge for Purdue to regain the confidence and cooperation of its nuclear engineering faculty and adequately investigate these allegations. Six of the school’s senior faculty members objected in writing to the manner in which the previous inquiry was conducted and its conclusions. Some have indicated that they do not want to participate in another proceeding.

Unfortunately, all three of the members of the new committee that Purdue named to the new committee were members of previous committees, and the staff member also is the same as before. We were told that these people were appointed because they were already familiar with the issues and the research which would save committee time. We acknowledge that inquiries of this type take a great deal of time and commitment from the research university and its staff. But appointing new, independent and disinterested members would add credibility to the process.

In the same way that Taleyarkhan may be biased toward seeing confirmations of his results, that the graybeards in his field may be biased against accepting results that require them to revise their fundamental understanding of their subject matter, and that grad students and postdocs may be biased toward interpreting results in ways that please the boss, the folks on the committee who have already dealt with allegations against Taleyarkhan may well be biased in any future inquiry or investigation by their previous findings. If the truth matters — and it should — bringing on new people without a personal stake in what the facts end up showing is essential.