Friday Sprog Blogging: peer review.

Unlike some of my dear readers, the elder Free-Ride offspring, upon reading yesterday’s post, immediately recognized it as an April Fool’s Day joke. (This recognition was accompanied by only the barest hint of a smile. A mother’s fine, dry wit is, apparently, an acquired taste.)
Although that post was bogus, some of its content seemed worth discussing.
Dr. Free-Ride: Do you know what peer review is?
Elder offspring: No.
Dr. Free-Ride: Do you have a guess? Do you know what a “peer” is?

Elder offspring: Yes, a friend, or anyone you work with.
Dr. Free-Ride: OK, and do you know what “review” is?
Elder offspring: Yeah, it’s your opinion on something.
Dr. Free-Ride: Oh, like a movie review is your opinion on the movie. Do you have a hunch from that what “peer review” might mean?
Elder offspring: Review on your friends?
Dr. Free-Ride: So, when scientists are publishing what they’ve found out for other scientists, before it gets published, it usually undergoes something called peer review.
Elder offspring: They talk about what they found out with their friends?
Dr. Free-Ride: It’s kind of like that, but it’s also kind of not like that. When you think about science, what makes scientific claims different from other kinds of claims?
Elder offspring: They’re true. Usually.
Dr. Free-Ride: They at least are reaching for truth, right?
Elder offspring: Yeah.
Dr. Free-Ride: But to get their claims closer to truth, scientists can’t just have the kind of interactions you might have with a friend who asks, “Hey, do I look good in this outfit?” or “Do you like this picture I just drew?” In cases like that, you might be tempted to tell your friends something that they want to hear.
Elder offspring: Uh huh.
Dr. Free-Ride: We hope scientists aren’t just telling their scientific friends something they want to hear about whether their results are good ones. So what scientists do is they try really hard to figure out what’s going on in the system they’re studying, and they write up what they did to study it, and they write up what their conclusions are, and they send it to the journal editor. And then the journal editor sends that out to other scientists, and often the other scientists don’t even get the name of the person whose findings they’re reviewing. They read these findings and they write a peer review, which basically says, here’s what I think of what they wrote. Then those comments go back to the author. Sometimes they say, “This seems important! This should get published!” Other times they say, “Well, I’m not sure. I’m not convinced by this part of it.” Or, “Maybe you need to do another experiment to make sure what you’re seeing isn’t due to this other thing that I think could be the reason for the effect you’re seeing.” And then the scientist who wrote the original paper (actually, it’s usually a group of scientists) look at the reviews they’ve gotten back, usually from three different reviewers, and they’re supposed to respond to those somehow. Either they make the changes the reviewers ask for, or they explain why they don’t need to make those changes, or whatever. Then they send their response to the journal editor, and maybe it goes back to the reviewers and then back to the authors, or maybe the editor decides that the authors have answered all the questions well enough and it their findings get published.
Elder offspring: Wow.
Dr. Free-Ride: What do you think about this? Does it sound like a good way to do things, or a bad way to do things, or what?
Elder offspring: Well, I can tell you some of the cons.
Dr. Free-Ride: Tell me.
Elder offspring: You could lose friendship with your friends.
Dr. Free-Ride: With your scientific friends? But you might not know that they they were the ones reviewing your paper. These reviews usually come back without names on them.
Elder offspring: They do? Well, that’s a pro.
Dr. Free-Ride: Although sometimes the worry is that while you don’t know who’s reviewing your paper, they may know who you are, or at least be able to figure it out from your paper. Some people think that’s a little bit unfair. Are there other cons you can think of?
Elder offspring: Well, if you’re a reviewer and you criticize someone, if they don’t know who you are, maybe they could tell from your style of writing. Or maybe they would just go thinking that it was someone who has something against them personally. So what you write in your review could cause more trouble than you intended to.
Dr. Free-Ride: So authors could try to get revenge, is what you’re saying? Maybe when they were reviewers?
Elder offspring: Yeah.
Dr. Free-Ride: Do you think scientists ultimately would rather get their work published, or do you think they’d rather have work that makes claims about the world that are true?
Elder offspring: Work that makes claims about the world that’s true.
Dr. Free-Ride: But maybe it’s hard to tell the difference when you’ve been working on a project for so long.
Elder offspring: Yeah.
Dr. Free-Ride: Can you think of other ways scientists could help each other to figure out if the picture of the world they’re building is a true one?
Elder offspring: Hmm. Ask other reliable people.
Dr. Free-Ride: OK, but how do you tell who those are?
Elder offspring: Like, if they run everything, they would probably — hopefully — be reliable.
Dr. Free-Ride: “Run everything” like what? What do you mean?
Elder offspring: Like run the lab or something.
Dr. Free-Ride: Oh, I see. So the person who’s in charge of the lab is probably reliable.
Elder offspring: Yeah.
Dr. Free-Ride: Sometimes the person who runs the lab brings in funding and stuff, but isn’t necessarily the one who knows how to make the equipment work to get the experiments to go.
Elder offspring: Hmm.
Dr. Free-Ride: And sometimes there’s research that has so many different parts that there’s no one person who knows how to make them all work. In cases like that, the group of scientists working together hopes that each of the scientists in that group is reliable. But how do we know?
Elder offspring: I don’t know.
Dr. Free-Ride: Maybe we don’t know. Maybe we have to go with the best evidence we’ve got, and keep looking for more evidence.
Elder offspring: I guess.
Dr. Free-Ride: Well, now that you know that articles that get published in scientific journals go through this whole process where they get comments from other scientists before they get published, does that change your opinion of scientific papers.
Elder offspring: Mmm … it depends.
Dr. Free-Ride: Do you think scientists in general try to be more trustworthy than people in other fields?
Elder offspring: Yes.
Dr. Free-Ride: Do you think they succeed more often?
Elder offspring: I don’t know.
Dr. Free-Ride: Do you think there’s a danger they could fool themselves into thinking they succeed more often than they do?
Elder offspring: Yeah, I think so.
Dr. Free-Ride: In your experience, is it easy to get fooled or hard to get fooled? Or is it context-dependent?
Elder offspring: It’s context dependent.
Dr. Free-Ride: What kinds of things could make it harder to fool you?
Elder offspring: Like, if you have a lie detector.
Dr. Free-Ride: Did you know, in my Psych 101 class in college, I got to be the student hooked up to the polygraph? They had me draw a number, then they asked me for all the numbers one to ten if that was the number I drew, and I was supposed to say “no”, and they watched what the machine output was.
Elder offspring: So one of those times you said “no” it was a lie?
Dr. Free-Ride: Yes. But the thing is, I was able to fool the machine — and the class — into thinking I had picked a different number.
Elder offspring: You beat the lie-detector?
Dr. Free-Ride: Yup. I think sometimes it works as a lie-detector by fooling people into believing it always works as a lie detector. Maybe sometimes trusting the technology too much can make you easier to fool.
Elder offspring: Yeah.
Dr. Free-Ride: Are there other particular things that make you more cautious about being fooled?
Elder offspring: I don’t know.
Dr. Free-Ride: Any special places? Any special days on the calendar?
Elder offspring: Yeah, April Fool’s Day.