Category Archives: Methodology

Limits of ethical recycling.

Posted on by

In the “Ethics in Science” course I regularly teach, we spend some time discussing case studies to explore some of the situations students may encounter in their scientific training or careers where they will want to be able to make good ethical decisions.

A couple of these cases touch on the question of “recycling” pieces of old grant proposals or journal articles — say, the background and literature review.

There seem to be cases where the right thing to do is pretty straightforward. For example, helping yourself to the background section someone else had written for her own grant proposal would be wrong. This would amount to misappropriating someone else’s words and ideas without her permission and without giving her credit. (Plagiarism anyone?) Plus, it would be weaseling out of one’s own duty to actually read the relevant literature, develop a view about what it’s saying, and communicate clearly why it matters in motivating the research being proposed.

Similarly, reusing one’s own background section seems pretty clearly within the bounds of ethical behavior. You did the intellectual labor yourself, and especially in the case where you are revising and resubmitting your own proposal, there’s no compelling reason for you to reinvent that particular wheel (unless, if course, reviewer comments indicate that the background section requires serious revision, the literature cited ought to take account of important recent developments that were missing in the first round, etc.).

Between these two extremes, my students happened upon a situation that seemed less clear-cut. How acceptable is it to recycle the background section (or experimental protocol, for that matter) from an old grant proposal you wrote in collaboration with someone else? Does it make a difference whether that old grant proposal was actually funded? Does it matter whether you are “more powerful” or “less powerful” (however you want to cash that out) within the collaboration? Does it require explicit permission from the person with whom you collaborated on the original proposal? Does it require clear citation of the intellectual contribution of the person with whom you collaborated on the original proposal, even if she is not officially a collaborator on the new proposal?

And, in your experience, does this kind of recycling make more sense than just sitting down and writing something new?

A question for the trainees: How involved do you want the boss to get with your results?

Posted on by

This question follows on the heels of my recent discussion of the Bengü Sezen misconduct investigations, plus a conversation via Twitter that I recapped in the last post.

The background issue is that people — even scientists, who are supposed always to be following the evidence wherever it might lead — can run into trouble really scrutinizing the results of someone they trust (however that trust came about). Indeed, in the Sezen case, her graduate advisor at Columbia University, Dalibor Sames, seemed to trust Sezen and her scientific prowess so much that he discounted the results of other graduate students in his lab who could not replicate Sezen’s results (which turned out to have been faked).

Really, it’s the two faces of the PI’s trust here: trusting one trainee so much that her results couldn’t be wrong, and using that trust to ignore the empirical evidence presented by other trainees (who apparently didn’t get the same level of presumptive trust). As it played out, at least three of those other trainees whose evidence Sames chose not to trust left the graduate program before earning their degrees.

The situation suggests to me that PIs would be prudent to establish environments in their research groups where researchers don’t take scrutiny of their results, data, methods, etc., personally — and where the scrutiny is applied to each member’s results, data, methods, etc. (since anyone can make mistakes). But how do things play out when they rubber hits the road?

So, here’s the question I’d like to ask the scientific trainees. (PIs: I’ve posed the complementary question to you in the post that went up right before this one!)

In his or her capacity as PI, your advisor’s scientific credibility (and likely his or her name) is tied to all the results that come out of the research group — whether they are experimental measurements, analyses of measurements, modeling results, or whatever else it is that scientists of your stripe regard as results. Moreover, in his or her capacity as a trainer of new scientists, the boss has something like a responsibility to make sure you know how to generate reliable results — and that you know how to tell them from results that aren’t reliable. What does your PI do to ensure that the results you generate are reliable? Do you feel like it’s enough (both in terms of quality control and in terms of training you well)? Do you feel like it’s too much?

Commenting note: You may feel more comfortable commenting with a pseudonym for this particular discussion, and that’s completely fine with me. However, please pick a unique ‘nym and keep it for the duration of this discussion, so we’re not in the position of trying to sort out which “Anonymous” is which. Also, if you’re a regular commenter who wants to go pseudonymous for this discussion, you’ll probably want to enter something other than your regular email address in the commenting form — otherwise, your Gravatar may give your other identity away!

A question for the PIs: How involved do you get in your trainees’ results?

Posted on by

In the wake of this post that touched on recently released documents detailing investigations into Bengü Sezen’s scientific misconduct, and that noted that a C & E News article described Sezen as a “master of deception”, I had an interesting chat on the Twitters:

@UnstableIsotope (website) tweeted:

@geernst @docfreeride I scoff at the idea that Sezen was a master at deception. She lied a lot but plenty of opportunities to get caught.

@geernst (website) tweeted back:

@UnstableIsotope Maybe evasion is a more accurate word.

@UnstableIsotope:

@geernst I’d agree she was a master of evasion. But she was caught be other group members but sounds like advisor didn’t want to believe it.

@docfreeride (that’s me!):

@UnstableIsotope @geernst Possible that she was master of deception only in environment where people didn’t guard against being deceived?

@UnstableIsotope:

@docfreeride @geernst I agree ppl didn’t expect deception, my read suggests she was caught by group members but protected by advisor.

@UnstableIsotope:

@docfreeride @geernst The advisor certainly didn’t expect deception and didn’t encourage but didn’t want to believe evidence

@docfreeride:

@UnstableIsotope @geernst Not wanting to believe the evidence strikes me as a bad fit with “being a scientist”.

@UnstableIsotope:

@docfreeride @geernst Yes, but it is human. Not wanting to believe your amazing results are not amazing seems like a normal response to me.

@geernst:

@docfreeride @UnstableIsotope I agree. Difficult to separate scientific objectivity from personal feelings in those circumstances.

@docfreeride:

@geernst @UnstableIsotope But isn’t this exactly the argument for not taking scrutiny of your results, data, methods personally?

@UnstableIsotope:

@docfreeride @geernst Definitely YES. I look forward to people repeating my experiments. I’m nervous if I have the only result.

@geernst:

@docfreeride @UnstableIsotope Couldn’t agree more.

This conversation prompted a question I’d like to ask the PIs. (Trainees: I’m going to pose the complementary question to you in the very next post!)

In your capacity as PI, your scientific credibility (and likely your name) is tied to all the results that come out of your research group — whether they are experimental measurements, analyses of measurements, modeling results, or whatever else it is that scientists of your stripe regard as results. What do you do to ensure that the results generated by your trainees are reliable?

Now, it may be the case that what you see as the appropriate level of involvement/quality control/”let me get up in your grill while you repeat that measurement for me” would still not have been enough to deter — or to detect — a brazen liar. If you want to talk about that in the comments, feel free.

Commenting note: You may feel more comfortable commenting with a pseudonym for this particular discussion, and that’s completely fine with me. However, please pick a unique ‘nym and keep it for the duration of this discussion, so we’re not in the position of trying to sort out which “Anonymous” is which. Also, if you’re a regular commenter who wants to go pseudonymous for this discussion, you’ll probably want to enter something other than your regular email address in the commenting form — otherwise, your Gravatar may give your other identity away!

What are honest scientists to do about a master of deception?

Posted on by

A new story posted at Chemical & Engineering News updates us on the fraud case of Bengü Sezen (who we discussed here, here, and here at much earlier stages of the saga).

William G. Schultz notes that documents released (PDF) by the Department of Health and Human Services (which houses the Office of Research Integrity) detail some really brazen misconduct on Sezen’s part in her doctoral dissertation at Columbia University and in at least three published papers.

From the article:

The documents—an investigative report from Columbia and HHS’s subsequent oversight findings—show a massive and sustained effort by Sezen over the course of more than a decade to dope experiments, manipulate and falsify NMR and elemental analysis research data, and create fictitious people and organizations to vouch for the reproducibility of her results. …

A notice in the Nov. 29, 2010, Federal Register states that Sezen falsified, fabricated, and plagiarized research data in three papers and in her doctoral thesis. Some six papers that Sezen had coauthored with Columbia chemistry professor Dalibor Sames have been withdrawn by Sames because Sezen’s results could not be replicated. …

By the time Sezen received a Ph.D. degree in chemistry in 2005, under the supervision of Sames, her fraudulent activity had reached a crescendo, according to the reports. Specifically, the reports detail how Sezen logged into NMR spectrometry equipment under the name of at least one former Sames group member, then merged NMR data and used correction fluid to create fake spectra showing her desired reaction products.

Apparently, her results were not reproducible because those trying to reproduce them lacked her “hand skills” with Liquid Paper.

Needless to say, this kind of behavior is tremendously detrimental to scientific communities trying to build a body of reliable knowledge about the world. Scientists are at risk of relying on published papers that are based in wishes (and lies) rather than actual empirical evidence, which can lead them down scientific blind alleys and waste their time and money. Journal editors devoted resources to moving her (made-up) papers through peer review, and then had to devote more resources to dealing with their retractions. Columbia University and the U.S. government got to spend a bunch of money investigating Sezen’s wrongdoing — the latter expenditures unlikely to endear scientific communities to an already skeptical public. Even within the research lab where Sezen, as a grad student, was concocting her fraudulent results, her labmates apparently wasted a lot of time trying to reproduce her results, questioning their own abilities when they couldn’t.

And to my eye, one of the big problems in this case is that Sezen seems to have been the kind of person who projected confidence while lying her pants off:

The documents paint a picture of Sezen as a master of deception, a woman very much at ease with manipulating colleagues and supervisors alike to hide her fraudulent activity; a practiced liar who would defend the integrity of her research results in the face of all evidence to the contrary. Columbia has moved to revoke her Ph.D.

Worse, the reports document the toll on other young scientists who worked with Sezen: “Members of the [redacted] expended considerable time attempting to reproduce Respondent’s results. The Committee found that the wasted time and effort, and the onus of not being able to reproduce the work, had a severe negative impact on the graduate careers of three (3) of those students, two of whom [redacted] were asked to leave the [redacted] and one of whom decided to leave after her second year.”

In this matter, the reports echo sources from inside the Sames lab who spoke with C&EN under conditions of anonymity when the case first became public in 2006. These sources described Sezen as Sames’ “golden child,” a brilliant student favored by a mentor who believed that her intellect and laboratory acumen provoked the envy of others in his research group. They said it was hard to avoid the conclusion that Sames retaliated when other members of his group questioned the validity of Sezen’s work.

What I find striking here is that Sezen’s vigorous defense of her’s own personal integrity was sufficient, at least for awhile, to convince her mentor that those questioning the results were in the wrong — not just incompetent to reproduce the work, but jealous and looking to cause trouble. And, it’s deeply disappointing that this judgment may have been connected to the departure of those fellow graduate students who raised questions from their graduate program.

How could this have been avoided?

Maybe a useful strategy would have been to treat questions about the scientific work (including its reproducibility) first and foremost as questions about the scientific work.

Getting results that others cannot reproduce is not prima facie evidence that you’re a cheater-pants. It may just mean that there was something weird going on with the equipment, or the reagents, or some other component of the experimental system when you did the experiment that yielded the exciting but hard to replicate results. Or, it may mean that the folks trying to replicate the results haven’t quite mastered the technique (which, in the case that they are your colleagues in the lab, could be addressed by working with them on their technique). Or, it may mean that there’s some other important variable in the system that you haven’t identified as important and so have not worked out (or fully described) how to control.

In this case, of course, it’s looking like the main reason that Sezen’s results were not reproducible was that she made them up. But casting the failure to replicate presumptively as one scientist’s mad skillz and unimpeachable integrity against another’s didn’t help get to the bottom of the scientific facts. It made the argument personal rather than putting the scientists involved on the same team in figuring out what was really going on with the scientific systems being studied.

Of all of the Mertonian norms imputed to the Tribe of Science, organized skepticism is probably the one nearest and dearest to most scientists’ basic understanding of how they get the knowledge-building job done. Figuring out what’s going on with particular phenomena in the world can be hard, not least because lining up solid evidence to support your conclusions requires identifying evidence that others trying to repeat your work can reliably obtain themselves. This is more than just a matter of making sure your results are robust. Rather, you want others to be able to reproduce your work so that you know you haven’t fooled yourself.

Organized skepticism, in other words, should start at home.

There is a risk of being too skeptical of your own results, and there are chances to overlook something important as noise because it doesn’t fit with what you expect to observe. However, the scientist who refuses to entertain the possibility that her work could be wrong — indeed, who regards questions about the details of her work as a personal affront — should raise a red flag for the rest of her scientific community, no matter what her career stage or her track record of brilliance to date.

In a world where every scientist’s findings are recognized as being susceptible to error, the first response to questions about findings might be to go back to the phenomena together, helping each other to locate potential sources of error and to avoid them. In such a world, the master of deception trying to ride personal reputation (or good initial impressions) to avoid scrutiny of his or her work will have a much harder time getting traction.

Scientific knowledge and “what everyone knows”.

Posted on by

Those of you who read the excellent blog White Coat Underground have probably had occasion to read PalMD’s explanation of the Quack Miranda Warning, the disclaimer found on various websites and advertisements that reads, “These statements have not been evaluated by the Food and Drug Administration. This product is not intended to diagnose, treat, cure or prevent any disease.” When found on a website that seems actually to be offering diagnosis, treatment, cure, or prevention, PalMD notes, this language seems like a warning that the big change that will be effected is that your wallet will be lightened.

In response to this, Lawrence comments:

This statement may be on every quack website but is on every legitamate website and label as well. Take vitamin C for example. Everyone knows that it can help treat & cure diseases. Vitamin C has been used for centuries to cure disease by eating various foods that are high in it. Even doctors tell you it is good to take when you are sick because it helps your body fight off the disease. So the fact that this statement is required to be on even the most obviously beneficial vitamins pretty much means that the FDA requires a companies to lie to the public and that they have failed in their one duty to encouraging truth in health. Once I realized this, it totally discredits everything the FDA says.

Sure if something is not approved by a big organization whose existance is supposed to safeguard health it makes it easier for the little con artest to step in at every opportunity, but that doesn’t mean that the big con artests arn’t doing the same thing

PalMD’s reply is succinct:

“Everyone knows…”

A phrase deadly to science.

I’m going to add my (less succinct) two cents.

There are plenty of things that people take to be something everyone knows. (The “everyone” is tricky, because there are enough people on the planet that it’s usually (always?) possible to find someone who doesn’t know X.). And, I’m happy to grant that, for some values of X, there are indeed many people who believe X.

But belief is not the same as knowledge.

What “everyone knows” about celebrities should help us notice the difference. Richard Gere? Jamie Lee Curtis? Even in the event that everyone has heard the same rumors, the extent of what we actually know is that there are rumors. Our propensity to believe rumors is why the team at Snopes will never want for material.

This is not to say that we have to do all of our epistemic labor ourselves. Indeed, we frequently rely on the testimony of others to help us know more than we could all by ourselves, But, this division of labor introduces risks if we accept as authoritative the testimony of someone who is mistaken — or who is trying to sell us snake-oil. Plus, when we’re accepting the testimony of someone who knows X on the basis of someone else’s testimony, our connection to the actual coming-to-know of X (through a mode other than someone else’s say-so) becomes more attenuated.

At least within the realm of science, the non-testimony route to knowledge involves gathering empirical evidence under conditions that are either controlled or at least well characterized. Ideally, the effects that are observed are both repeatable in relevantly similar conditions and observable by others. Science, in its methodology, strives to ground knowledge claims in observational evidence that anyone could come to know (assuming a standard set of properly functioning sense organs). Part of how we know that we know X is that the evidence in support of X can be inspected by others. At this basic level, we don’t have to take anyone else’s word for X; the testimony of our senses (and the fact that others who are pointing their sense organs at the same bits of the world and seeing the same things) gives us the support for our beliefs that we need.

Claims without something like empirical support might inspire belief, but they don’t pass scientific muster. To the extent that an agency like the FDA is committed to evaluating claims in a scientific framework, this means that they want to evaluate the details of the experiments used to generate the empirical data that are being counted as support for those claims. In other contexts, folks may be expecting, or settling for, other standards of evidence. In scientific contexts, including biomedical ones, scientific rules of evidence are what you get.

Why then, one might ask, might a physician suggest vitamin C to a patient with a cold if there isn’t sufficient scientific evidence to say we know vitamin C cures cold?

There are a few possibilities here. One is that the physician judges (on the basis of a reasonable body of empirical evidence) that taking vitamin C is unlikely to do harm to the patient with a cold. If the physician’s clinical experience is that cold patients will feel better with some intervention than with no intervention, recommending vitamin C may seem like the most benign therapeutic option.

It’s also possible that some of these physicians accept the testimony of someone else who tells the there is good reason to believe that vitamin C cures colds. Being human, physicians sometimes get burned by testimony that turns out to be unreliable.

It’s even possible that some physicians are not so clear on scientific rules of evidence, and that they make recommendations on the basis of beliefs that haven’t been rigorously tested. The more high profile of these physicians are the kinds of folks about whom PalMD frequently blogs.

Friday Sprog Blogging: science fair experimental design.

Posted on by

The elder Free-Ride offspring is thinking about a project studying the behavior of Snowflake Free-Ride, the rabbit in residence at Casa Free-Ride. While finding interesting questions to ask about the bunny is pretty easy, working out reasonable ways to get data that might help answer those questions is somewhat harder:

Elder offspring: I want to see whether Snowflake finds food with her eyes or her nose.

Dr. Free-Ride: What are your thoughts on how to do that?

Elder offspring: Well, we need a room …

Dr. Free-Ride: … OK. Tell me more.

Elder offspring: We need a room with a fan up at the top.

Dr. Free-Ride: Why do we need a fan up at the top?

Elder offspring: To blow away the smells.

Dr. Free-Ride: Hmm. So you’re looking for some mechanism to mask smells and see if she can still find the food.

Elder offspring: Yes.

Dr. Free-Ride: I guess I’m not totally convinced a fan is the best way to mask a smell. Also, I worry that it might freak her out.

Elder offspring: Oh.

Dr. Free-Ride: Well, your hypothesis is that she’s either finding the food by smell or by sight. So how would you tell if sight is what she’s using?

Elder offspring: We start the fan and put the food there and if she can find it … We may also need to use a clothespin, like in those cartoons —

Dr. Free-Ride: We’re totally not putting a clothespin on the rabbit’s nose, smart aleck!

Elder offspring: (snickering) I know.

Dr. Free-Ride: Let’s back up a little bit. We’re talking about two possible ways you think the rabbit could locate food — one is by vision, one is by smell. Masking smell means we have to figure out a way to get the volatile stuff that the nose detects away from her. But my own hunch is that masking sight might be easier. Do you have any thoughts on how to mask —

Elder offspring: Blindfolds.

Dr. Free-Ride: Uh, no. You’ll have to be more clever, since you can’t blindfold the bunny.

Elder offspring: Put her in a dark room.

Dr. Free-Ride: I don’t know how good her night vision is. (Or how good your night vision is if you’re in the dark room trying to observe her.)

Elder offspring: If we hear munching …

Dr. Free-Ride: Isn’t she always munching on something?

Elder offspring: We’d use a food where the munching sounds like crunching.

Dr. Free-Ride: Aside from utter darkness, can you think of any other way to mask visual contact with the food?

Elder offspring: What if we surround a carrot by things that are visually distracting?

Dr. Free-Ride: Does that really test whether she’s using vision to find the carrot, or whether she can pick it out visually amongst a bunch of visually distracting things? Maybe you need to think about whether there’s some way to disguise it looking like a carrot, but it would still be there for her to smell.

Elder offspring: How about we put it behind a curtain or something?

Dr. Free-Ride: Ah, a barrier that keeps her from seeing it. Then, with the carrot out of sight but in smelling range, you’d see if she reacted like, “Where’s the carrot. GIMME THE CARROT!”

Elder offspring: Yeah.

Dr. Free-Ride: OK, that seems like a key part of your experimental design: how exactly are you going to mask the carrot’s visibility but not its smell?

Elder offspring: Invisibility cloak!

Dr. Free-Ride: You don’t get to use things that don’t exist in your science fair project. Unless you can successfully invent them, in which case — if you can successfully invent an invisibility cloak, I submit to you that that would probably be a more impressive science fair project than this information on rabbit behavior that you obtain using the invisibility cloak.

Elder offspring: Yeah, OK.

Dr. Free-Ride: Hey, is it going to be a problem that you have exactly one rabbit to study?

Elder offspring: Nah.

Dr. Free-Ride: What’s that going to do to the conclusions you can draw.

Elder offspring: I probably can’t say that all rabbits are like this based on the behavior of this one rabbit. But, she’s a pretty typical rabbit.

Dr. Free-Ride: How do you know she’s pretty typical?

Elder offspring: Because, she’s a breed [New Zealand white] that’s raised for lab use, and they want typical animals for lab use.

Dr. Free-Ride: Which means you would be surprised if she were very weird, as rabbits go?

Elder offspring: Yes.

Dr. Free-Ride: Of course, she’s been living with you for almost a year now. That might be enough to make a rabbit weird.

Elder offspring: Hey!

Dr. Free-Ride: I’m just saying. So, back to your experimental design, since Snowflake is a smart rabbit — she learns stuff — what if you make a curtain or some other barrier and she starts associating it with carrots?

Elder offspring: Maybe sometimes we could just put a rock behind it instead of a carrot.

Dr. Free-Ride: Good call — something that isn’t edible and doesn’t smell like a treat.

Dispatch from PSA 2010: Symposium session on ClimateGate.

Posted on by

The Philosophy of Science Association Biennial Meeting included a symposium session on the release of hacked e-mails from the Climate Research Unit at the University of East Anglia. Given that we’ve had occasion to discuss ClimateGate here before, i thought I’d share my notes from this session.

Symposium: The CRU E-mails: Perspectives from Philosophy of Science.

Naomi Oreskes (UC San Diego), gave a talk called “Why We Resist the Results of Climate Science.”

She mentioned the attention brought to the discovery of errors in the IPCC report, noting that while mistakes are obviously to be avoided, it would be amazing for there to be a report that ran thousands of pages that did not have some mistakes. (Try to find a bound dissertation — generally only in the low hundreds of pages — without at least one typo.) The public’s assumption, though, was that these mistakes, once revealed, were smoking guns — a sign that something improper must have occurred.

Oreskes noted the boundary scientists of all sorts (including climate scientists) have tried to maintain between the policy-relevant and the policy-prescriptive. This is a difficult boundary to police, though, as climate science has an inescapable moral dimension. To the extent that climate change is driven by consumption (especially but not exclusively the burning of fossil fuels), we have a situation where the people reaping the benefits are not the ones who will be paying for that benefit (since people in the developed world will have the means to respond to the effects of climate change and those in the developing world will not). The situation seems to violate our expectations of intergenerational equity (since future generations will have to cope with the consequences of the consumption of past and current generations), as well as of inter-specific equity (since the species likely to go extinct in response to climate change are not the ones contributing the most to climate change).

The moral dimension of climate change, though, doesn’t make this a scientific issue about which the public feels a sense of clarity. Rather, the moral issues are such that Americans feel like their way of life is on trial. Those creating the harmful effects have done something wrong, even if it was accidental.

And this is where the collision occurs: Americans believe they are good; climate science seems to be telling them that they are bad. (To the extent that people strongly equate capitalism with democracy and the American way of life, that’s an issue too, given that consumption and growth are part of the problem.)

The big question Oreskes left us with, then, is how else to frame the need for changes in behavior, so that such a need would not make Americans so defensive that they would reflexively reject the science. I’m not sure the session ended with a clear answer to that question.

* * * * *

Wendy S. Parker (Ohio University) gave a talk titled “The Context of Climate Science: Norms, Pressures, and Progress.” A particular issue she took up was the ideal of transparency and how it came up in the context of climate scientists interactions with each other and with the public.

Parker noted that there had been numerous requests for access to raw data by people climate scientists did not recognize as part of the climate science community. The CRU denied many such requests, and the ClimateGate emails made it clear that the scientists generally didn’t want to cooperate with these requests.

Here, Parker observed that while we tend to look favorably on transparency, we probably need to say more about what transparency should amount to. Are we talking about making something available and open to scrutiny (i.e., making “transparency” roughly the opposite of “secrecy”)? Are we talking about making something understandable or usable, perhaps by providing fully explained nontechnical accounts of scientific methods and findings for the media (i.e., making “transparency” roughly the opposite of “opacity”)?

What exactly do we imagine ought to be made available? Research methods? Raw and/or processed data? Computer code? Lab notebooks? E-mail correspondence?

To whom ought the materials to be made available? Other members of one’s scientific community seems like a good bet, but how about members of the public at large? (Or, for that matter, members of industry or of political lobbying groups?)

And, for that matter, why do we value transparency? What makes it important? Is it primarily a matter of ensuring the quality of the shared body of scientific knowledge, and of improving the rate of scientific progress? Or, do we care about transparency as a matter of democratic accountability? As Parker noted, these values might be in conflict. (As well, she mentioned, transparency might conflict with other social values, like the privacy of human subjects.)

Here, if the public imputed nefarious motives to the climate researchers, the scientists themselves viewed some of the requests for access to their raw data as attempts by people with political motivations to obstruct the progress (or acceptance) of their research. It was not that the scientists feared that bad science would be revealed if the data were shared, but rather that they worried that yahoos from outside the scientific community were going to waste their time, or worse to cherry pick the shared data to make allegations that the scientists to which would then have to respond, wasting even more time.

In the numerous investigations that followed on the heels of the leak of stolen CRU e-mails, about the strongest charge against the involved climate scientists that stood was that they failed to display “the proper degree of openness”, and that they seemed to have a ethos of minimal compliance (or occasionally non-compliance) with regard to Freedom of Information Act (FOIA) requests. They were chided that the requirements of FOIA must not be seen as impositions, but as part of their social contract with the public (and something likely to make their scientific knowledge better).

Compliance, of course, takes resources (one of the most important of these being time), so it’s not free. Indeed, it’s hard not to imagine that at least some FOIA requests to climate scientists had “unintended consequences” (in terms of the expenditure of tim and other resources) on climate scientists that were precisely what the requesters intended.

However, as Parker noted, FOIA originated with the intent of giving citizens access to the workings of their government — imposing it on science and scientists is a relatively new move. It is true that many scientists (although not all) conduct publicly funded research, and thereby incur some obligations to the public. But there’s a question of how far this should go — ought every bit of data generated with the aid of any government grant to be FOIA-able?

Parker discussed the ways that FOIA seems to demand an openness that doesn’t quite fit with the career reward structures currently operating within science. Yet ClimateGate and its aftermath, and the heightened public scrutiny of, and demands for openness from, climate scientists in particular, seem to be driving (or at least putting significant pressure upon) the standards for data and code sharing in climate science.

I got to ask one of the questions right after Parker’s talk. I wondered whether the level of public scrutiny on climate scientists might be enough to drive them into the arms of the “open science” camp — which would, of course, require some serious rethinking of the scientific reward structures and the valorization of competition over cooperation. As we’ve discussed on this blog on many occasions, institutional and cultural change is hard. If openness from climate scientists is important enough to the public, though, could the public decide that it’s worthwhile to put up the resources necessary to support this kind of change in climate science?

I guess it would require a public willing to pay for the goodies it demands.

* * * * *

The next talk, by Kristin Shrader-Frechette (University of Notre Dame), was titled “Scientifically Legitimate Ways to Cook and Trim Data: The Hacked and Leaked Climate Emails.”

Shrader-Frechette discussed what statisticians (among others) have to say about conditions in which it is acceptable to leave out some of your data (and indeed, arguably misleading to leave it in rather than omitting it). There was maybe not as much unanimity here as one might like.

There’s general agreement that data trimming in order to make your results fit some predetermined theory is unacceptable. There’s less agreement about how to deal with outliers. Some say that deleting them is probably OK (although you’d want to be open that you have done so). On the other hand, many of the low probability/high consequence events that science would like to get a handle on are themselves outliers.

So when and how to trim data is one of those topics where it looks like scientists are well advised to keep talking to their scientific peers, the better not to mess it up.

Of the details in the leaked CRU e-mails, one that was frequently identified as a smoking gun indicating scientific shenanigans was the discussion of the “trick” to “hide the decline” in the reconstruction of climatic temperatures using proxy data from tree-rings. Shrader-Frechette noted that what was being “hidden” was not a decline in temperatures (as measured instrumentally) but rather in the temperatures reconstructed from one particular proxy — and that other proxies the climate scientists were using didn’t show this decline.

The particular incident raises a more general methodological question: scientifically speaking, is it better to include the data from proxies (once you have reason to believe it’s bad data) in your graphs? Is including it (or leaving it out) best seen as scrupulous honesty or as dishonesty?

And, does the answer differ if the graph is intended for use in an academic, bench-science presentation or a policy presentation (where it would be a very bad thing to confuse your non-expert audience)?

As she closed her talk, Shrader-Frechette noted that welfare-affecting science cannot be treated merely as pure science. She also mentioned that while FOIA applies to government-funded science, it does not apply to industry-funded science — which means that the “transparency” available to the public is pretty asymmetrical (and that industry scientists are unlikely to have to devote their time to responding to requests from yahoos for their raw data).

* * * * *

Finally, James McAllister (University of Leiden) gave a talk titled “Errors, Blunders, and the Construction of Climate Change Facts.” He spoke of four epistemic gaps climate scientists have to bridge: between distinct proxy data sources, between proxy and instrumental data, between historical time series (constructed of instrumental and proxy data) and predictive scenarios, and between predictive scenarios and reality. These epistemic gaps can be understood in the context of the two broad projects climate science undertakes: the reconstruction of past climate variation, and the forecast of the future.

As you might expect, various climate scientists have had different views about which kinds of proxy data are most reliable, and about how the different sorts of proxies ought to be used in reconstructions of past climate variation. The leaked CRU e-mails include discussions where climate scientists dedicate themselves to finding the “common denominator” in this diversity of expert opinion — not just because such a common denominator might be expected to be closer to the objective reality of things, but also because finding common ground in the diversity of opinion could be expected to enhance the core group’s credibility. Another effect, of course, is that the common denominator is also denied to outsiders, undermining their credibility (and effectively excluding them as outliers).

McAllister noted that the emails simultaneously revealed signs of internal disagreement, and of a reaching for balance. Some of the scientists argued for “wise use” of proxies and voiced judgments about how to use various types of data.

The data, of course, cannot actually speak for themselves.

As the climate scientists worked to formulate scenario-based forecasts that public policy makers would be able to use, they needed to grapple with the problems of how to handle the link between their reconstructions of past climate trends and their forecasts. They also had to figure out how to handle the link between their forecasts and reality. The e-mails indicate that some of the scientists were pretty resistant to this latter linkage — one asserted that they were “NOT supposed to be working with the assumption that these scenarios are realistic,” rather using them as internally consistent “what if?” storylines.

One thing the e-mails don’t seem to contain is any explicit discussion of what would count as an ad hoc hypothesis and why avoiding ad hoc hypotheses would be a good thing. This doesn’t mean that the climate scientists didn’t avoid them, just that it was not a methodological issue they felt they needed to be discussing with each other.

This was a really interesting set of talks, and I’m still mulling over some of the issues they raised for me. When those ideas are more than half-baked, I’ll probably write something about them here.

More is better? Received widsom in the tribe of science.

Posted on by

Because it’s turning out to be that kind of semester, I’m late to the party in responding to this essay (PDF) by Scott E. Kern bemoaning the fact that more cancer researchers at Johns Hopkins aren’t passionate enough to be visible in the lab on a Sunday afternoon. But I’m sure as shooting going to respond.

First, make sure you read the thoughtful responses from Derek Lowe, Rebecca Monatgue, and Chemjobber.

Kern’s piece describes a survey he’s been conducting (apparently over the course of 25 years) in which he seemingly counts the other people in evidence in his cancer center on Saturdays and Sundays, and interviews them with “open-ended, gentle questions, such as ‘Why are YOU here? Nobody else is here!'” He also deigns to talk to the folks found working at the center 9 to 5 on weekdays to record “their insights about early morning, evening and weekend research.” Disappointingly, Kern doesn’t share even preliminary results from his survey. However, he does share plenty of disdain for the trainees and PIs who are not bustling through the center on weekends waiting for their important research to be interrupted by a guy with a clipboard conducting a survey.

Kern diagnoses the absence of all the researchers who might have been doing research as an indication of their lack of passion for scientific research. He tracks the amount of money (in terms of facilities and overhead, salaries and benefits) that is being thrown away in this horrific weekend under-utilization of resources. He suggests that the researchers who have escaped the lab on a weekend are falling down on their moral duty to cure cancer as soon as humanly possible.

Sigh.

The unsupported assumptions in Kern’s piece are numerous (and far from novel). Do we know that having each research scientist devote more hours in the lab increases the rate of scientific returns? Or might there plausibly be a point of diminishing returns, where additional lab-hours produce no appreciable return? Where’s the economic calculation to consider the potential damage to the scientists from putting in 80 hours a week (to their health, their personal relationships, their experience of a life outside of work, maybe even their enthusiasm for science)? After all, lots of resources are invested in educating and training researchers — enough so that one wouldn’t want to break them on the basis of an (unsupported) hypothesis offered in the pages of Cancer Biology & Therapy.

And while Kern is doing economic calculations, he might want to consider the impact on facilities of research activity proceeding full-tilt, 24/7. Without some downtime, equipment and facilities might wear out faster than they would otherwise.

Nowhere here does Kern consider the option of hiring more researchers to work 40 hour weeks, instead of shaming the existing research workforce into spending 60, 80, 100 hours a week in the lab.

They might still end up bringing work home (if they ever get a chance to go home).

Kern might dismiss this suggestion on purely economic grounds — organizations are more likely to want to pay for fewer employees (with benefits) who can work more hours than to pay to have the same number of hours of work done my more employees. He might also dismiss it on the basis that the people who really have the passion needed to do the research to cure cancer will not prioritize anything else in their lives above doing that research and finding that cure.

If that is so, it’s not clear how the problem is solved by browbeating researchers without this passion into working more hours because they owe it to cancer patients. Indeed, Kern might consider, in light of the relative dearth of researchers with such passion (as he defines it), the necessity of making use of the research talents and efforts of people who don’t want to spend 60 hours a week in the lab. Kern’s piece suggests he’d have a preference for keeping such people out of the research ranks, but by his own account there would hardly be enough researchers left in that case to keep research moving forward.

Might not these conditions prompt us to reconsider whether the received wisdom of scientific mentors is always so wise? Wouldn’t this be a reasonable place to reevaluate the strategy for accomplishing the grand scientific goal?

And Kern does not even consider a pertinent competing hypothesis, that people often have important insights into how to move research forward in the moments when they step back and allow their minds to wander. Perhaps less time away from one’s project means fewer of these insights.

The part of Kern’s piece that I find most worrisome is the cudgel he wields near the end:

During the survey period, off-site laypersons offer comments on my observations. “Don’t the people with families have a right to a career in cancer research also?” I choose not to answer. How would I? Do the patients have a duty to provide this “right”, perhaps by entering suspended animation? Should I note that examining other measures of passion, such as breadth of reading and fund of knowledge, may raise the same concern and that “time” is likely only a surrogate measure? Should I note that productive scientists with adorable family lives may have “earned” their positions rather than acquiring them as a “right”? Which of the other professions can adopt a country-club mentality, restricting their activities largely to a 35–40 hour week? Don’t people with families have a right to be police? Lawyers? Astronauts? Entrepreneurs?

How dare researchers go home to their families until they have cured cancer?

Indeed, Kern’s framing here warrants an examination of just what cancer patients can demand from researchers (or any other members of society), and on what basis. But that is a topic so meaty that it will require it’s own post.

Besides which, I have a pile of work I brought home that I have to start plowing through.

Is objectivity an ethical duty? (More on the Hauser case.)

Posted on by

Today the Chronicle of Higher Education has an article that bears on the allegation of shenanigans in the research lab of Marc D. Hauser. As the article draws heavily on documents given to the Chronicle by anonymous sources, rather than on official documents from Harvard’s inquiry into allegations of misconduct in the Hauser lab, we are going to take them with a large grain of salt. However, I think the Chronicle story raises some interesting questions about the intersection of scientific methodology and ethics.

From the article:

Continue reading→

More on strategies to accomplish training.

Posted on by

Earlier this week, I mentioned that I had powered through some online training courses that I needed to complete by the (rapidly approaching) beginning of my academic term. In that post, I voiced my worries about how well I’d be able to retain the material I took in (and, one hopes, absorbed to at least some extent) in one long sitting at my computer.

As it happens, I am spending today and tomorrow at full-day training sessions (about nine hours per day, including breaks) covering related material at much greater depth and breadth. Obviously, this affords me the opportunity to compare the two modes of content delivery.

One thing I’ve noticed is that I seem to have retained substantial chunks of the material presented in the online training. (Sure, retaining it for two days is maybe not a huge accomplishment, but these have been subtle details — and I’m pretty sure I have students who can forget material more rapidly than this once the quiz on the material is behind them.)

It’s possible, though, that my retention of that material will be better because I’m using it in this live training. I’ll really have no way to tell which bits of the overlapping material stick in my head because of the online training and which stick because of the live training since I’m doing both in rapid succession. (Too many variables!)

The live training has so far been more interactive during the presentation of material, with speakers taking questions and asking us questions. (They’ve also distributed clicker-like devices that we’ll be using during the presentations after lunch.) There haven’t been any quizzes on the material (yet), but there will be breakout groups in which our active participation is required.

We’ve also been presented with gigantic binders containing handouts with slides for each of the presentations (complete with space for our own notes), related articles, and extensive listings of additional resources (including online resources). These binders have been adding to my sense of actively engaging with the information rather than just having the information wash over me. Plus, my binder will now be my first stop if I need to look up a piece of information from this training, which I personally will find easier than digging through my Firefox bookmarks.

A disadvantage of this training is that it eats up two calendar days set far in advance by the trainers, in a particular location far enough from most of the participants’ home bases that they need to book lodging for a couple nights. As well, owing to the A/V needs of the presenters and the aforementioned gigantic binders, the cost per participant of the training session is significant.

Why, you might ask, am I doing both of these overlapping training programs in rapid succession?

Strictly speaking, the live training sessions I’m doing today and tomorrow are not required of me. However, given responsibilities that stem from my committee appointments, this training is a really good idea. It will help me do my job better, and I’m bringing home resources I can share with other committee members who can benefit from them. The training may be taking up eighteen hours of my life right now, but I anticipate what I’m learning may save me at least that many hours of spinning my wheels just in the coming semester.

The online training was something I was required to take, but it strikes me as the minimal amount of information adequate to prepare someone for my committee duties. Plus, the online training is being required of a larger population at my university than just members of my committee, so we committee members are also doing the online training to ensure that we understand how well it’s working for the other people taking it.

One thing I’m thinking in light of this week of training is that my committee might want to find a way to offer periodic opportunities for live training on campus (at least as a companion to the online training if not as a substitutable alternative). If we want the people who are partaking of the training to have more than a minimal grasp of the material on which they’re being trained, recognizing different learning styles and building in more open-ended interactivity might bring about better results.