There’s a recent paper on blogs as a channel of scientific communication that has been making the rounds. Other bloggers have discussed the paper and its methodology in some detail (including but not limited to Bora and DrugMonkey and Dr. Isis), so I’m not going to do that. Rather, I want to pull back and “get meta” with the blogospheric discussion of the paper, and especially the suggestion that it might be out of bounds for science bloggers (some of whom write the blogs that provided the data for the paper in question) to mount such a vigorous critique of a paper that was, as it turns out, authored by a graduate student.
So, let’s consider the situation more generally:
Let’s say you are a graduate student. You have undertaken a research study (which maybe is central to your dissertation project, or perhaps is peripheral to your dissertation project). You think the research poses an interesting and important question, and that it makes a contribution to working out an empirically grounded answer to that question. Probably your research won’t be the last word on it, but you think it’s an important step forward (maybe because it puts forward a reasonable methodological strategy for tackling the question, or it at least tries to get relevant data where previous discussion have been data-free).
You write the research up and you submit a manuscript to a journal. Maybe it’s a high prestige journal, or maybe it’s a more “low key” journal. Maybe your graduate advisor (or another mentor) has given you a lot of feedback as you did the research and wrote the manuscript; on the other hand, maybe you’ve been feeling your way through the process pretty much on your own.
The journal editor sends you a bunch of comments from the people who refereed your manuscript.
You read these comments, make whatever revisions you’re inclined to make (or whatever revisions the communique from the journal editor makes it sound like you need to make to get the manuscript published, or whatever revisions your graduate advisor or other mentor makes it sound like you ought to make before the manuscript is published), and you send that revised manuscript back to the journal editor.
Then, the journal publishes your manuscript.
At this stage, what do you want?
Do you want people to read your article and discuss it?
If so, does it matter to you if this discussion is in private spaces (where you don’t know how it’s being discussed, or even that it’s being discussed at all)? Or in traditional scholarly public spaces (like print journals or conference sessions or letters to the editor of the journal where your article was published)? Or in public spaces like listservs and blogs (where you — or others who have read your published article — might happen on them via a search engine)? Why or why not?
Do you want the people having these discussions to point out what they see as the limitations of your data, or the shortcomings in your methodology? Why or why not?
Do you want people who might discuss your paper, positively or negatively, to take special account of the fact that you are a graduate student? Why or why not?
The same question can (and perhaps should) be asked of a post-doc. In a way, the post-doc has more at stake. Graduate students have more room to be controversial and/or stupid. At least there is a sense of engagement. Post-docs on the other hand have the potential to do some real damage to their careers.
I don’t think I can even begin to answers those questions at a meta-level, but I’ll try to answer them from my own experience. Mind you, my experience is in trying to get published (journalistically) rather than academically, but I think there is significant overlap in the post-publishing process.
I think it really all comes back to education as process rather than product. Opening your work to wide-spread scrutiny will give you more eyes to find errors. Assuming you’re interested in the topic you’re writing about, this will only lead to making your next paper stronger. I think the preferred forum for all this to play out is on blogs/twitter etc, as they are the most open. You will actually be able to respond/discuss the problems they inevitably find, rather than be forced to read them from afar.
As an example from my own current experience, I put interview transcripts and the first draft of an article on my blog. The interviewees shared links with people they know, and a significant amount of discussion has spawned out of it. One of the biggest selfish benefits of this is that I’ve been introduced to a suite of ideas – ideas I hadn’t even considered. The second draft of my article will take some of these opinions into consideration, and the final product will be vastly improved.
As for special consideration for being a grad student… I’m not really sure what form that would be able to take. Respect and constructive discussion is beneficial for anyone, not just grad students.
I don’t give a single flying fuck whether people read or discuss my article. I want them to fucking cite it in the peer reviewed literature.
Catharine @1, I have a feeling that how much room grad students are given to “be stupid” in the literature is a field-related issue. For example, a number of faculty my grad program in philosophy warned grad students off publishing “too soon” and being saddled, for their whole career, with authorship of an unfortunate (or badly-defended) view.
This is tough. As a graduate student still learning the ropes of science in many ways, I do sometimes shrink under criticism. When it’s coming from my mentor, my course instructors, or my peers, I can take it because I know that A) it’s intended to help me do better work in the future, and B) these people don’t actually hate my guts. Depending on how critical comments were made in the blogosphere, I could see myself being unsure of whether points A and B still applied.
Still, I think any criticism that’s based on the content of my work is worth hearing, even if I am upset by the tone with which it’s delivered, because it provides an opportunity for improvement. If the criticism refers to an experimental method, or a statistical analysis, or something else objective, it’s pretty easy to say, “I was unaware of that, and I’ll take it into consideration next time.” For something more subjective, though, it’s hard. I’ve had to deal with meeting much more senior scientists who (GASP!) disagree with me about scientific models or philosophical/political ideas that are close to my heart, and while it is tough to take sometimes (especially if those senior scientists get… heated… during discussion), ultimately I see that as an opportunity for growth, too. I may decide to retain my opinions in the end, but seeing things from another perspective is useful.
I do think that some criticism can be so personal as to become inappropriate, though. I haven’t read all the comments on this particular article, so I can’t say whether I think any of them have crossed that line.
My god! Of course you want people to discuss it! That’s how science works. We publish our results in journals, people discuss them, people discuss the interpretations and implications of them, people test the predictions and try to replicate them. The worst thing for a scientific result is to be ignored.
As to whether it matters if you’re a graduate student or not… it doesn’t matter at all. Hopefully, the graduate student’s friends and mentors will make sure that the student takes comments and discussion in the correct light and keeps them from getting discouraged, but that has absolutely nothing to do with who the author is, whether the author is wimpy graduate student whose feelings you don’t want to hurt or angry senior PI who you are scared of.
This is why impact is measured as “impact factor” – how much your work is discussed. Note that a citation saying “X says Y is true, but we think X is wrong” is a citation. As my old advisor used to say, I don’t care if you trash my work… just spell my name right.
Science is about open discussion. That’s what makes the scientific method work. Anything else is just masturbation. Look, this isn’t an f’ing carebears party. We’re trying to discover things here. That means that the scientific record is for discussion. “Don’t argue with me” is not science.
At this stage, what do you want?
The question might or might be interesting – but is completely irrelevant. You may be a grizzled veteran or a budding graduate student getting into the pool for the very first time – but that, too, is completely irrelevant. You’ve published it and you don’t have a say in how people react to your work anymore than any author, artist, musician, programmer or anyone publishing things for others to see.
To put it this way: if the graduate student had published their first novel – “Burning Bunsens, Burning Passions”, a steamy pageturner about forbidden love in a chemistry lab, say – then they would have no say in how critics and the public received it, right? They’d have no right to direct how people discuss it, and have no cause to argue that they should not face criticism because it’s their first book.
The work sucks or soars, and people will say so. The author has no voice over and above any other, whether the work is a novel or a piece of research.
A couple years ago, there was a commentary published in Nature Geoscience that argued that blogging about research articles was problematic: http://www.nature.com/ngeo/journal/v1/n4/abs/ngeo174.html . (Unfortunately, it’s behind a paywall.) Chris Rowan and I both discussed it on our respective blogs.
(Chris here: http://scienceblogs.com/highlyallochthonous/2008/04/bloggers_and_blogging_in_natur.php ; me here: http://shearsensibility.blogspot.com/2008/04/is-it-unethical-to-blog-about-peer.html .)
This is really field-dependent. I can’t imagine anyone who does basic science caring about lay opinions of his or her work, but in the public health sciences, lay perception of work makes a huge difference to funding priorities.
If people think that conversations they are seeing on blogs are the exact same conversations that are happening in labs and journal clubs, they are deluded. At least here they are easily accesible.
CPP #3: I agree that getting lots of citations is the desired result. But before other people can cite my paper, they have to know about it. So if people are reading and discussing my paper, that’s a good thing.
I want people to read and discuss my work. Although, as CPP so eloquently points out, I want that more as a means-to-an-end of being cited or otherwise building a professional reputation, than an end in itself.
Discussion is particularly important to get a good grasp on different views of strengths/weaknesses of my work. Up until the point I’ve figured out what the limitations are, criticism (particularly of the “what would really improve this study is…” variety) is hugely beneficial.
In this particular case, it would appear that Bora was being a laudable scientific citizen in his original peer-review, and that he was going above and beyond at working toward a solid scientific record in his subsequent write up. However, from my view, it would seem that others have yet to add anything that constitutes an original insight into the limitations of the paper.
In short, if we are going meta, it’s been mostly unoriginal analysis of unoriginal analysis… and none of it particularly constructive (with the exception of Bora’s input).
Given the relative lack of focus on improving the work, and the fact that Dr. Isis and DM’s tones could strike the unaware as condescending and abrasive, I am sympathetic to those that view this as academic bullying. I wouldn’t want my work discussed like that (well, not if I didn’t know Dr. Isis and DM well enough to be able to accept them as well-intentioned… but if your friends *have* to use the fact that you are their friend to defend your actions… something about your actions might not be optimal).
Do grad students really get the chance to publish something unfortunate?
Even if my mentors had allowed something bad to get submitted, peer review would have smacked it down HARD. Just as it smacks down some stuff I submit as a senior PI type person.
How does something (like the article in question) get through peer review with major holes in it?
One publishes one’s work with the expectation that it will be discussed. One does not get to ask the question you are asking here.
Having said that, what is the evidence that people do not want this paper discussed by bloggers? I’ve read the blog posts you cite and some others you did not cite, and some of the comments, and I have not seen anyone making the suggestion that the blogs should keep their mitts off this paper. Please provided documentation of this. (Not because I do not believe you, I simply would like to see what people have said out of interest.)
This is another serious look at that paper, from a perspective of someone who has experience in social/qualitative research.
It’s probably good practice for the students in question. It takes a rather thick hide to survive in science. That said, whether the criticisms are done respectfully — that is, as thoroughly as needed, but without too much ‘viciousness’ — is the responsibility of the blogger.
I’d suspect that the criticisms on public blogs are mild and gentle compared to how some of these papers are discussed in private. I’ve called papers and hypotheses “fucking stupid” in private with my friends, but I generally refrain from even that tone in public writing. Except for creationists with an agenda. But even there being aggressive detracts from your argument. I imagine most other bloggers have similar guidelines. And honestly, we can’t be -that- much worse than many reviewers!
Furthermore, personally I’d kind of -like- people to actually react to anything I write publicly. Anything meant for viewing by others — that is, pretty much anything besides a private diary or something — should be intended for some sort of reaction on the part of the reader, and reactions go both ways…
It doesn’t really matter whether you want people to or not.
One of the many (assumed/given) key activities of scientific research is openness. With properly public discussion you get much wider distribution and more input (some of it criticism), and this is a Good Thing past the immediate short term emotional involvement with a particular paper. Whether you like it or not…
One thing I was hoping for, but did not happen (yet) was for the author (and/or advisor) to come and respond in the comments of my blog post, to enlighten us a little bit about the thought processes behind the study, the chosen methodology, etc. and to engage with the critique.
I had to laugh at #3 response, since obviously the commenter considers number of citations as the ultimate indicator of excellence of his/her cited scientific work. This, of course, is the falacy of simply counting citations as an indicator of excellence. The scientific literature is flooded with examples of papers that received great number of citations only to be exposed as poorly designed, wrongly analyzed or completely fabricated.
I see a huge upside to blogging on and discussion of a published scientific paper’s merits, especially when certain problems with the published work could be exposed early.
I do not like when such discussions lead to personal attacks on the author(s), as is evident in the post by Dr. Isis.
Does anyone know if the graduate student in question is even aware of the conversations her paper has initiated, and whether she is contributing in any way?
If I were her, I suppose I would be mortified, because the criticism has been (understandably, IMO) strong. That said, I have felt some unease at the strength of the reviews given she is a grad student… and some frustration with the journal for publishing something with a heterogeneous n of 11. I think Coturnix’s review does a good job of implicating the journal has some responsibility here, since he has records of voicing concerns with the manuscript that were not addressed.
And if I were her, while right now I may want to crawl under a rock, I would like to think that in a year or two I would have a renewed perspective and an appreciation that so many people care enough about this field and their stake in it that they would talk so much about her paper.
I want people to read and discuss my work- everyone likes to hear only the good comments about their work, but I would like to hear the bad too – what can I do to improve? Was there something I missed that I should have caught? Even better if I can see those comments – if I never hear the negative comments, then I can neither respond to them (if they aren’t valid) nor do anything about them (if they are valid). I’d much rather that happen in public – where I can find out, than in a journal club somewhere where I’ll never hear about useful critiques of my work. (Not that journal clubs are bad – I just don’t have legions of spies reporting to me…)
One thing I notice about cranks is that they tend to work alone – no-one points out to them early on that they’re making a basic mistake. Then ten-years later, they’re too invested in their work to give it up for being wrong – it must be the basic theory that’s wrong. I don’t want to go down that road!
@qaz #6: what you say is field dependent. Mathematicians easily reach consensus on right/wrong issues (not so much on interesting) and you can’t publish something wrong once the referee notices. On the other hand, if you write something difficult it might well take the community 10+ years to fully digest it.
That’s why for us citation number is not really relevant – we’re measured by letters (and invitations, grants, etc: ultimately it’s always peer review).
It’s also a field were as a grad student you can be as silly as you want – all the blame goes to your advisor (who is not supposed to co-sign, even if they found the problem for you and taught you everything you need to know to solve it).
Hi Janet
It’s always a writer’s fault if the reader misunderstands them. Or so I’m told.
I don’t mind being used as a discussion starter but I think my position isn’t quite what you are implying.
To channel physioprof I am the motherfucking ninja of robust peer review. I love it. I’ve learnt shitloads from good peer-reviewers. Grad students should be peer-reviewed the hell out of- it’s part of their training. Post publication peer review on blogs is awesome- especially when it’s done by people like DM, Janet or Bora.
I thought DMs original post was excellent. I did not mean to give the impression that I questioned its relevance. My fear was that the repetition of posting was in danger of moving beyond a legitimate and robust peer-review and into something more like an academic blood sport.
Prosaica #22 –
No, it is not field dependent. I’ve worked in mathematics and the same thing applies. While it is true that in mathematics, it is possible to lay out the entire logic in the publication, there are many publications in mathematics where the implications are discussed for many years after publication. In particular, the generality (and sometimes even the validity) has been/can be questioned after publication.
Science (I’ll include mathematics here, but not literary criticism) is based on a self-correction process by which replication, open discussion, and community checking gets at a truth that can’t be reliably gotten in any other way.
Obviously it is field dependent, and obviously what you want is sort of irrelevant. But I do think I personally would have drawn the line at the photo implying the author was taking a dump when they wrote the paper, given that they are a grad student ;-).
Sharon
“I personally would have drawn the line at the photo implying the author was taking a dump when they wrote the paper, given that they are a grad student.”
This is just another sign that science has become a blood sport and many of its practitioners today are seeing themselves as its gladiators. It also goes into the issue of incivility on the bloggosphere. Even more disturbing is the fact that the blogger who described the author of this study as “taking a dump” while writing it, has declared herself “a fighter for women equality in science” yet, she easily disparages a female colleague who also happened to be a graduate student. This fake defender of women equality in science would be the first one to launch an all out attack on all “white male scientists” if even one of them would dare describing a female author of a scientific paper as sitting on the toilet seat while writing the paper.
BTW, some of the best ideas may emerge, while one is sitting in the restroom, the only place one really knows what one does! 😉
Once something is published, it is fair game. So if whiny ignored reviewers want to debase themselves by whinging to cyberspace, there is nothing unethical about them doing so.