Chad Orzel takes a commenter to task for fetishizing peer review:
Saying that only peer-reviewed articles (or peer-reviewable articles) count as science only reinforces the already pervasive notion that science is something beyond the reach of “normal” people. In essence, it’s saying that only scientists can do science, and that science is the exclusive province of geeks and nerds.
That attitude is, I think, actively harmful to our society. It’s part of why we have a hard time getting students to study math and science, and finding people to teach math and science. We shouldn’t be restricting science to refereed journals, we should be trying to spread it as widely as possible.
Peer review and refereed journals are a good check on science, but they do not define the essence of science. Science is, at its core, a matter of attitude and procedure. The essence of science is looking at the world and saying “Huh. I wonder why that happens?” And then taking a systematic approach to figuring it out.
I see what Chad is saying — and to the extent that science can be said to have an “essence” think he’s hit on a nice way to describe it. But I’m going to speak up for peer review here.
It’s worth noting that “peer review” can encompass different things.
Peer review describes the formal process through which manuscripts that have been submitted to journal editors are then sent to reviewers with relevant expertise for their evaluation. These reviewers then reply to the journal editors with their evaluation of the manuscript — whether it should be accepted, resubmitted after revision, or rejected — and their comments on particular aspects of the manuscript (this conclusion would be more solid if it were supported by this kind of analysis of the data, that data looks more equivocal than the authors seem to think it is, this part of the materials and methods is confusingly written, the introduction could be much more concise, etc., etc.). The editor passes on the feedback to the author, the author responds to that feedback (either by making changes in the manuscript or by presenting the editor with a persuasive argument that what a reviewer is asking for is off base or unreasonable), and eventually the parties end up with a version of the paper deemed good enough for publication (or the author gives up, or tries to get a more favorable hearing from another journal).
This flavor of peer review is very much focused on making sure that papers published in scientific journals meet a certain standard of quality or acceptability to the other scientists who will be reading those papers. There’s a lot of room for disagreement about what sort of quality is produced here, about how conservative reviewers can be when faced with new ideas or approaches, about how often reviewer judgments can be overturned by the judgment of editors (and whether that is on balance a good thing or a bad thing). As we’ve discussed before, the quality control here does not typically include reviewers actually trying to replicate the experiments described in the manuscripts they are reviewing.
Still, there’s something about peer review that a great many scientists think is important, at least when they want to be able to consult the literature in their discipline. If you want to see how your results fit with the results that others are reporting in similar lines of research, or if you’re looking for promising instrumental or theoretical approaches to a tenacious scientific puzzle, it’s good to have some reason to trust what’s reported in the literature. Otherwise, you have to do all the verification yourself.
And this is where a sort of peer review becomes important to the essence of science Chad describes.
The scientist, looking at the world and trying to figure out some bit of it, is engaged in theorizing and observing, in developing hunches and then testing those hunches. The scientist wants to end up with a clearer understanding of how that bit of the world is behaving, and of what could explain that behavior.
And ultimately, the scientist relies on others to get that clearer understanding.
To really trust our observations, they need to be observations that others could make as well. To really buy our own explanations for what we observe, we need to be ready to put those explanations out for the inspection of others who might find some flaw in them, some untested assumption that doesn’t hold up to close scrutiny.
Science may be characterized by an attitude toward the world, an attitude that gets us asking particular kinds of questions, but the systematic approach to answering these questions requires the participation of other people working with the same basic assumptions about how we can engage with the world to understand it better. Those other people are peers, and their participation is a kind of review.
In short, I agree with Chad that there is a lot of scientific thinking and activity beyond that whose products are reported in scientific journals after a formal peer review process. But I also think that it is vital to recognize that even the less formal, more everyday scientific thinking is inherently a team sport where verifiability by others is a criterion of success.
Not to mention that peer review is a process to which, in principle, anyone can submit, and as I understand it the review process is anonymous not only in that the author(s) do not know the identity of the reviewers, but also in that the reviewer does not know the identity of the author(s). Granted that my own experience is in astronomy, a field with a long tradition of hobbyist involvement, and granted also that page charges can sometimes keep people from being able to publish in peer-reviewed journals, but in principle if you would like to join the scientific discussion, and you do work that can stand up to scrutiny, nothing is preventing you from submitting it.
Hogwash. I could, if I so desired, go off on my own and do science. I could preform experiments to get every individual fact. Would I still be doing science? Indeed I would be by your own description even: [quote]The scientist, looking at the world and trying to figure out some bit of it, is engaged in theorizing and observing, in developing hunches and then testing those hunches. The scientist wants to end up with a clearer understanding of how that bit of the world is behaving, and of what could explain that behavior.[/quote]
The fact that no one goes off by themselves and does this is because 1.) social reasons and 2.) you’d never make much progress.
The fact that the way we [i]do[/i] science is in teams doesn’t require that science [i]be[/i] a team activity.
Modern science needs peer review like modern society needs laws. It doesn’t make science perfect, but it keeps standards high enough that by and large, the work of others can be trusted and cited confidently. Without peer review in this day and age, there would be no initial check in place – made by people with suitable expertise – against flaws in reasoning, poorly worded arguments, mistakes in methodology, failure to cite past literature, and fraud. Broadly speaking, every paper I’ve submitted – and every one I’ve reviewed – is improved by the crucible of peer review. Last but not least, without peer review the pseudoscientific hokum of Intelligent Design and Creation Science might actually be treated, at least initially, as valid science.
If it hasn’t been subject to peer review, it may as well not exist as far as the collective enterprise of generating scientific knowledge is concerned.
Grad – I partially agree with your last statement (“The fact that the way we [i]do[/i] science is in teams doesn’t require that science [i]be[/i] a team activity.”), but want to note one thing –
We don’t need others to do science, but if we want to do GOOD science, the yes, we must engage in a team activity.
There are no “value-free” facts or inquiries, and every observation is theory-laden (that is, every observation presupposes some theoretical framework in order to make sense). Every choice of experiment or method, interpretation of data, every hunch we follow, is loaded with bias and value assumptions. Whether they are wide-spread psychological biases, biases of our own quirky personal history, biases inherent in language, biases from our philosophical commitments, or from some other source, we are always limited to some degree, and every statement about the world we make is, in some way, skewed.
If, as I believe, one of the goals of science is to get as unbiased a perspective on the world as humanly possible, a perspective based on what is, not on what we see, then in order to do science well, we need other people. We need them as a check on our bias, and as sources of different perspectives on the same world.
One of the reasons science is such a powerful tool is that by working as a community of people who strive for rigor and the most objective methods possible, check up on each other’s work, test hypotheses and throwing away those that don’t hold up, eventually enough of the bias gets cleared away that we can talk about objective fact. But this is something that can only be done in a community.
I’m with Grad on this one. The fact that knowledge generation is a social acitivity is a contingent fact about steps that have been taken to increase the reliability of the process. There’s nothing about knowledge itself, or it’s validation, that requires results to be filtered through a “group mind.”
The journal Nature didn’t routinely use anonymous peer review until 1967 – see http://www.nature.com/nature/history/timeline_1960s.html.
Einstein had exactly one paper subject to peer review (it was rejected) – http://scitation.aip.org/journals/doc/PHTOAD-ft/vol_58/iss_9/43_1.shtml.
More generally, anonymous peer review only started to become widespread in science in the 1930s, and only became near-universal in the 1960s. These facts are extremely inconvenient for people who believe anonymous peer review is necessary for science.
Grad would have fit right in several hundred years ago, when many folks did secret science, and then had long battles over priority.
It is not science until it has met criticism by peers.
Michael Nielsen – if I understand her correctly, I believe that our good blogmistress’s point is that the specific method – anonymous review procedures – used by journals today is only one sense in which communities engage in peer review, and that method itself is not necessary. Other methods for peer review of your work include presenting or posting it in a public forum, and providing a means for people to comment on and discuss it.
Consequently, I don’t find those facts inconvenient at all.
No they aren’t — unless it’s the “anonymous” part we’re arguing over? I’m all for at least allowing reviewers to choose whether or not to remain anonymous.
But as an argument against the value and utility of peer review, “Einstein didn’t need it, nyah nyah” is bogus. Einstein et al didn’t have to make themselves heard over the roar of today’s flood of information, nor were they exactly doing run-of-the-mill work. The Richard Feynmans of the world might be fine without peer review, but normal science (in the Kuhn sense) would grind to a halt without it.
If “grind to a halt” is too strong, dilute it as per James F above. I find it difficult to believe that, in the face of both the increasing scale and complexity of the problems we’re tackling and the ever-increasing data deluge, we could build reliable knowledge at anything resembling a reasonable speed without peer review.
(Caveat: “peer review” need not be performed only in the context of publication in journals. The trust-network-based filtering that goes on at FriendFeed is a kind of peer review, and there’s no reason to think it couldn’t be harnessed and formalized in the pursuit of alternative outlets for scientific information.)
Mea culpa. I was unclear in both who and what I was addressing. I quite agree with the thrust of the post. I agree that some form of peer review – basically, having skeptical friends and colleagues go over stuff – is absolutely essential to science. I think this is true even for the Einsteins and Feynmans. My earlier comment was meant to address simply the formalized system of anonymous peer review, which wasn’t the subject.
The essence (one of the essences?) of doing good science is admitting that you can’t really trust your own judgement. I of course think my work is brilliant, but I also think (with another bit of my brain) that my judgement of my own work is very biased because I’m unable to set aside my wishes and preconceptions. That’s why we need peer review*
*Those of us who can’t recognize that they shouldn’t trust their own judgement need it too, even though they think they don’t.
After reviewing hundredes of manuscripts for dozens of scientific journals throughout my career, I can only imagine the disastous outcome caused to the scientific endeavor by some of the rubbish that is regularly being submitted for publication. Peer review is the best and only filter science has to stop this rubbish from contaminating the scientific record. That is not to say that some junk does manage to filter in, but, thanks to peer review, it is not enough to destroy the scientific process.
Grad says, “The fact that the way we do science is in teams doesn’t require that science be a team activity.”
In some sense, perhaps, but science for limited creatures like us requires cooperation if we are to make anything like the progress that we’ve made. Escaping this would require more than determination or cleverness. A creature that could do advanced science from the beginning all by itself, although possible in principle, would be nothing like a human. As a philosopher, I have no interest in a conception of science fit only for gods.
As a philosopher, I _am_ interested in a conception of science “fit only for gods.” As in many areas of human endeavor, such conceptions can play the very valuable role of articulating aspirational goals. If the goal we aspire to is truth, we’re already playing the aspirational game.
“as I understand it … the reviewer does not know the identity of the author(s).”
This is rarely true. Generally, the author’s name is known to the reviewers.
As with most things, the benefits and potential abuses of peer review should be borne in mind. [personally, I think the process should be double blind; reviewers AND authors should be anonymous].
I will point out that scientific results are meaningless until they’re published — you can happily keep them to yourself, but then who will know, who will benefit, who can use them to develop further results, who can learn from them? Once they’re published, they’re subject to plenty of peer review. Everyone who reads it does her own evaluation; groups get together and publish reviews and counterarguments.
Peer review happens, whether we want it to or not, and whether it be formal or not.
I’ll also point out how many “proofs” of Fermat’s last theorem came along, and were shot down by reviewers, before a real proof survived scrutiny. Without review, one retains one’s errors and compounds them.