Manuscript review and the limits of your expertise.

At his lounge, the Lab Lemming poses an excellent hypothetical question about manuscript review:

Suppose you are reviewing a paper. Also assume, that like most papers these days, that it has multiple authors, each of whom applies his expertise to the problem at hand. And finally, assume that you are an expert in some, but not all of the fields used to solve the particular problem being reported in this paper.
What do you do if one of the key points in the paper that is not your area of expertise seems fishy. For example, if the paper is on your field area, what if some of the lab results seem fishy. Or if you are an analyst, what if the experimental setup seems odd.
Assuming that you are a successful researcher, you probably have long-time collaborators who are experts in these fields. So, what is the best way of accessing their expertise, given that some sort of confidence generally surrounds papers in review.

Let’s break down some of the relevant interests at work here:

Scientists, collectively, want peer review to function as a quality control mechanism to ensure that papers that get published in journals meet scientific standards — claims are supported with data, methodologies are spelled out explicitly enough that other researchers can figure out how the data were collected, analyzed, and so forth, some account is taken of how the reported findings fit into the existing body of knowledge in this field, etc. If there’s a problem with the paper, whether it’s an oversight, a result of sloppiness or an honest error, or actual flim-flam, it’s better for the tribe of science that the problem is identified — and addressed — prior to publication.
An individual scientist acting as a peer reviewer has an interest, therefore, in being as thorough as possible in screening manuscripts and flagging potential problems. Often this will involve drawing on one’s own area of expertise, but sometimes, as noted in the scenario above, one’s spidey-sense will be set to tingling by part of the paper that is outside one’s immediate expertise.
In the interests of being thorough and making the best informed evaluation possible, one might want to turn to a colleague with the relevant expertise for advice on the troubling bit of the paper. However, generally the peer review of manuscripts is explicitly confidential — in agreeing to review the manuscript, you are promising the journal editor (and by extension the authors of the paper) not to disclose the contents of the manuscript to anyone else. Among other things, this is supposed to prevent the leakage of information that might help competing researchers from scooping the authors of the manuscript before it has been published.
Now, there might be ways to ask experts about the bits of the manuscript that bug you by couching your questions in hypothetical terms. However, especially in the circumstances where your expert friends are working in closely related areas, the chances are good that they may be actual or potential competitors of the manuscript authors — and that they may be able to figure out, from your hypothetical description, just what these authors are up to. This is to say, your efforts to avoid a meaningful violation of confidentiality may fail — and because you aren’t up on the stuff you want to consult your expert friends about, you may be ill equipped to get meaningful feedback without such a violation.
Here are the alternative options that occur to me:
1. Hit the literature. You could ask your expert friends for recommendations of papers that describe the kind of experimental set up, or data analysis strategy, or whatever, that is bugging you in the manuscript you’re reviewing. From those recommendations, you can use the references to collect more papers. Perusing a reasonable handful of papers that deal with the issue you’re pondering may give you a better sense of whether what’s presented in the manuscript is normal or weird. If it still strikes you as weird, you can flag it as such in your review (listing the articles you’re using as your “baseline” for normal).
2. Note your concerns, and the limits of your expertise. You can communicate your concerns as part of your formal review of the manuscript, or you can do it in an email to the journal editor in advance of your formal review. Either way, it’s not inappropriate to pass on to the journal editor names and email address of the experts you would turn to for advice. The editor can then investigate whether these experts would be appropriate additional reviewers.
When the peer review process works well, the authors of the manuscript will address the concerns raised in the reviews (including yours). Flagging a piece of the paper that concern you — even if you don’t feel you have the basis to fully evaluate whether your concerns are well-grounded — ought to open up a discussion and present an opportunity for the authors to provide more explanation. If you’re open with the journal editor about the limits of your own expertise, the journal editor can take responsibility for drawing on the advice of additional experts in vetting the manuscript.
If and when the manuscript is published, you might shoot an email to the expert friend you were tempted to consult and have a chat about the published paper. Use it as an opportunity for your friend to educate you about the details of his or her expertise as applied to the research described in the paper. If he or she sees a glaring problem in the paper, a letter to the journal editor may be in order.