Category Archives: More Science

Credibility, bias, and the perils of having too much fun.

Posted on by

If you’re a regular reader of this blog (or, you know, attentive at all to the world around you), you will have noticed that scientific knowledge is built by human beings, creatures that, even on the job, resemble other humans more closely than they do Mr. Spock or his Vulcan conspecifics. When an experiment yields really informative results, most human scientists don’t cooly raise an eyebrow and murmur “Fascinating.” Instead, you’re likely to see a reactions somewhere on the continuum between big smiles, shouts of delight, and full-on end zone happy-dance. You can observe human scientists displaying similar emotional responses in other kinds of scientific situations, too — say, for example, when they find the fatal flaw in a competitor’s conclusion or experimental strategy.

Many scientists enjoy doing science. (If this weren’t so, the rest of us would have to feel pretty bad for making them do such thankless work to build knowledge that we’re not willing or able to build ourselves but from which we benefit nonetheless.) At least some scientists are enjoying more than just the careful work of forming hypotheses, making observations, comparing outcomes and predictions, and contributing to a more reliable account of the world and its workings. Sometimes the enjoyment comes from playing a particular kind of role in the scientific conversation.

Some scientists delight in the role of advancer or supporter of the new piece of knowledge that will change how we understand our world in some fundamental way. Other scientists delight in the role of curmudgeon, shooting down overly-bold claims. Some scientists relish being contrarians. Others find comfort in being upholders of consensus.

In light of this, we should probably consider whether having one of these human predilections like enjoying being a contrarian (or a consensus-supporter, for that matter) is a potential source of bias against which scientists should guard.

The basic problem is nothing new: what we observe, and how we interpret what we observe, can be influenced by what we expect to see — and, sometimes, by what we want to see. Obviously, scientists don’t always see what they want to see, else people’s grad school lab experiences would be deliriously happy rather than soul-crushingly frustrating. But sometimes what there is to see is ambiguous, and the person making the observation has to make a call. And frequently, with a finite set of data, there are multiple conclusions — not all of them compatible with each other — that can be drawn.

These are moments when our expectations and our ‘druthers might creep in as the tie-breaker.

At the scale of the larger community of science and the body of knowledge it produces, this may not be such a big deal. (As we’ve noted before, objectivity requires teamwork). Given a sufficiently diverse scientific community, there will be loads of other scientists who are likely to have different expectations and ‘druthers. In trying to take someone else’s result and use it to build more knowledge, the thought is that something like replication of the earlier result happens, and biases that may have colored the earlier result will be identified and corrected. (Especially since scientists are in competition for scarce goods like jobs, grants, and Nobel Prizes, you might start with the assumption that there’s no reason not to identify problems with the existing knowledge base. Of course, actual conditions on the ground for scientists can make things more complicated.)

But even given the rigorous assessment she can expect from the larger scientific community, each scientist would also like, individually, to be as unbiased as possible. One of the advantages of engaging with lots of other scientists, with different biases than your own, is you get better at noticing your own biases and keeping them on a shorter leash — putting you in a better place to make objective knowledge.

So, what if you discover that you take a lot of pleasure in being a naysayer or contrarian? Is coming to such self-awareness the kind of thing that should make you extra careful in coming to contrarian conclusions about the data? If you actually come to the awareness that you dig being a contrarian, does it put you in a better position to take corrective action than you would if you enjoyed being a contrarian but didn’t realize that being contrarian was what was bringing you the enjoyment?

(That’s right, a philosopher of science just made something like an argument that scientists might benefit — as scientists, not just as human beings — from self-reflection. Go figure.)

What kind of corrective action do I have in mind for scientists who discover that they may have a tilt, whether towards contrarianism or consensus-supporting? I’m thinking of a kind of scientific buddy-system, for example matching scientists with contrarian leanings to scientists who are made happier by consensus-supporting. Such a pairing would be useful for each scientist in the pair as far as vetting their evidence and conclusions: Here’s the scientist you have to convince! Here’s the colleague whose objections you need to understand and engage with before this goes any further!

After all, one of the things serious scientists are after is a good grip on how things actually are. An explanation that a scientist with different default assumptions than yours can’t easily dismiss is an explanation worth taking seriously. If, on the other hand, your “buddy” can dismiss your explanation, it would be good to know why so you can address its weaknesses (or even, if it is warranted, change your conclusions).

Such a buddy-system would probably only be workable with scientists who are serious about intellectual honesty and getting knowledge that is objective as possible. Among other things, this means you wouldn’t want to be paired with a scientist for whom having an open mind would be at odds with the conditions of his employment.

_____
An ancestor version of this post was published on my other blog.

Individual misconduct or institutional failing: “The Newsroom” and science.

Posted on by

I’ve been watching The Newsroom*, and in its second season, the storyline is treading on territory where journalism bears some striking similarities to science. Indeed, the most recent episode (first aired Sunday, August 25, 2013) raises questions about trust and accountability — both at the individual and the community levels — for which I think science and journalism may converge.

I’m not going to dig too deeply into the details of the show, but it’s possible that the ones I touch on here reach the level of spoilers. If you prefer to stay spoiler-free, you might want to stop reading here and come back after you’ve caught up on the show.

The central characters in The Newsroom are producing a cable news show, trying hard to get the news right but also working within the constraints set by their corporate masters (e.g., they need to get good ratings). A producer on the show, on loan to the New York-based team from the D.C. bureau, gets a lead for a fairly shocking story. He and some other members of the team try to find evidence to support the claims of this shocking story. As they’re doing this, they purposely keep other members of the production team out of the loop — not to deceive them or cut them out of the glory if, eventually, they’re able to break the story, but to enable these folks to look critically at the story once all the facts are assembled, to try to poke holes in it.** And, it’s worth noting, the folks actually in the loop, looking for information that bears on the reliability of the shocking claims in the story, are shown to be diligent about considering ways they could be wrong, identifying alternate explanations for details that seem to be support for the story, etc.

The production team looks at all the multiple sources of information they have. They look for reasons to doubt the story. They ultimately decide to air the story.

But, it turns out the story is wrong.

Worse is why key pieces of “evidence” supporting the story are unreliable. One of the interviewees is apparently honest but unreliable. One source of leaked information is false, because the person who leaked it has a grudge against a member of the production team. And, it turns out that the producer on loan from the D.C. bureau has doctored a taped interview that is the lynchpin of the story to make it appear that the interviewee said something he didn’t say.

The producer on loan from the D.C. bureau is fired. He proceeds to sue the network for wrongful termination, claiming it was an institutional failure that led to the airing of the now-retracted big story.

The parallels to scientific knowledge-building are clear.

Scientists with a hypothesis try to amass evidence that will make it clear whether the hypothesis is correct or incorrect. Rather than getting lulled into a false sense of security by observations that seem to fit the hypothesis, scientists try to find evidence that would rule out the hypothesis. They recognize that part of their job as knowledge-builders is to exercise organized skepticism — directed at their own scientific claims as well as at the claims of other scientists. And, given how vulnerable we are to our own unconscious biases, scientists rely on teamwork to effectively weed out the “evidence” that doesn’t actually provide strong support for their claims.

Some seemingly solid evidence turns out to be faulty. Measuring devices can become unreliable, or you get stuck with a bad batch of reagent, or your collaborator sends you a sample from the wrong cell line.

And sometimes a scientist who is sure in his heart he knows what the truth is doctors the evidence to “show” that truth.

Fabricating or falsifying evidence is, without question, a crime against scientific knowledge-building. But does the community that is taken in by the fraudster bear a significant share of the blame for believing him?

Generally, I think, the scientific community will say, “No.” A scientist is presumed by other members of his community to be honest unless there’s good reason to think otherwise. Otherwise, each scientist would have to replicate every observation reported by every other scientist ever before granting it any credibility. There aren’t enough grant dollars or hours in the day for that to be a plausible way to build scientific knowledge.

But, the community of science is supposed to ensure that findings reported to the public are thoroughly scrutinized for errors, not presented as more certain than the evidence warrants. The public trusts scientists to do this vetting because members of the public generally don’t know how to do this vetting themselves. Among other things, this means that a scientific fraudster, once caught, doesn’t just burn his own credibility — he can end up burning the credibility of the entire scientific community that was taken in by his lies.

Given how hard it can be to distinguish made-up data from real data, maybe that’s not fair. Still, if the scientific community is asking for the public’s trust, that community needs to be accountable to the public — and to find ways to prevent violations of trust within the community, or at least to deal effectively with those violations of trust when they happen.

In The Newsroom, after the big story unravels, as the video-doctoring producer is fired, the executive producer of the news show says, “People will never trust us again.” It’s not just the video-doctoring producer that viewers won’t trust, but the production team who didn’t catch the problem before presenting the story as reliable. Where the episodes to date leave us, it’s uncertain whether the production team will be able to win back the trust of the public — and what it might take to win back that trust.

I think it’s a reasonable question for the scientific community, too. In the face of incidents where individual scientists break trust, what does it take for the larger community of scientific knowledge-builders to win the trust of the public?

_____
* I’m not sure it’s a great show, but I have a weakness for the cadence of Aaron Sorkin’s dialogue.

** In the show, the folks who try to poke holes in the story presented with all the evidence that seems to support it are called the “red team,” and one of the characters claims its function is analogous to that of red blood cells. This … doesn’t actually make much sense, biologically. I’m putting a pin in that, but you are welcome to critique or suggest improvements to this analogy in the comments.

How far does the tether of your expertise extend?

Posted on by

Talking about science in the public sphere is tricky, even with someone with a lot of training in a science.

On the one hand, there’s a sense that it would be a very good thing if the general level of understanding of science was significantly higher than it is at present — if you could count on the people in your neighborhood to have a basic grasp of where scientific knowledge comes from, as well as of the big pieces of scientific knowledge directly relevant to the project of getting through their world safely and successfully.

But there seem to be a good many people in our neighborhood who don’t have this relationship with science. (Here, depending on your ‘druthers, you can fill in an explanation in terms of inadequately inspiring science teachers and/or curricula, or kids too distracted by TV or adolescence or whatever to engage with those teachers and/or curricula.) This means that, if these folks aren’t going to go it alone and try to evaluate putative scientific claims they encounter themselves, they need to get help from scientific experts.

But who’s an expert?

It’s well and good to say that a journalism major who never quite finished his degree is less of an authority on matters cosmological than a NASA scientist, but what should we say about engineers or medical doctors with “concerns” about evolutionary theory? Is a social scientist who spent time as an officer on a nuclear submarine an expert on nuclear power? Is an actor or talk show host with an autistic child an expert on the aetiology of autism? How important is all that specialization research scientists do? To some extent, doesn’t all science follow the same rules, thus equipping any scientist to weigh in intelligently about it?

Rather than give you a general answer to that question, I thought it best to lay out the competence I personally am comfortable claiming, in my capacity as a trained scientist.

As someone trained in a science, I am qualified:

  1. to say an awful lot about the research projects I have completed (although perhaps a bit less about them when they were still underway).
  2. to say something about the more or less settled knowledge, and about the live debates, in my research area (assuming, of course, that I have kept up with the literature and professional meetings where discussions of research in this area take place).
  3. to say something about the more or less settled (as opposed to “frontier”) knowledge for my field more generally (again, assuming I have kept up with the literature and the meetings).
  4. perhaps, to weigh in on frontier knowledge in research areas other than my own, if I have been very diligent about keeping up with the literature and the meetings and about communicating with colleagues working in these areas.
  5. to evaluate scientific arguments in areas of science other than my own for logical structure and persuasiveness (though I must be careful to acknowledge that there may be premises of these arguments — pieces of theory or factual claims from observations or experiments that I’m not familiar with — that I’m not qualified to evaluate).
  6. to recognize, and be wary of, logical fallacies and other less obvious pseudo-scientific moves (e.g., I should call shenanigans on claims that weaknesses in theory T1 necessarily count as support for alternative theory T2).
  7. to recognize that experts in fields of science other than my own generally know what the heck they’re talking about.
  8. to trust scientists in fields other than my own to rein in scientists in those fields who don’t know what they are talking about.
  9. to face up to the reality that, as much as I may know about the little piece of the universe I’ve been studying, I don’t know everything (which is part of why it takes a really big community to do science).

This list of my qualifications is an expression of my comfort level more than anything else. I would argue that it’s not elitist — good training and hard work can make a scientist out of almost anyone. But, it recognizes that with as much as there is to know, you can’t be an expert on everything. Knowing how far the tether of your expertise extends — and owning up to that when people look to you as an expert — is part of being a responsible scientist.

_______
An ancestor version of this post was published on my other blog.

“There comes a time when you have to run out of patience.”

Posted on by

In this post, I’m sharing an excellent short film called “A Chemical Imbalance,” which includes a number of brief interviews with chemists (most of them women, most at the University of Edinburgh) about the current situation for women in chemistry (and science, technology, engineering, and mathematics, or STEM, more generally) in the UK. Here’s the film:


A Chemical Imbalance
(I’m including my transcription of the film below.)

Some of the things I really appreciate about this film:

  • We get personal impressions, from women of different generations, about what it’s been like for them to be in chemistry in the UK.
  • We get numbers to quantify the gender disparity in academic chemistry in the UK, as well as to identify where in the career pipeline the disparity becomes worse. We also get numbers about how women chemists are paid relative to their male counterparts, and about relative rates of tenure that can’t be blamed on choices about childbearing and/or childrearing. There’s not just the perception of gender disparities in academic chemistry — the numbers demonstrate that the disparities are real.
  • Lurking beneath the surface is a conversation the interviewees might have had (but didn’t in the final cut) about what they count as compromises with respect to parenting and with respect to careers. My sense is that they would not all agree, and that they might not be as accepting of their colleagues’ alternative ways of striking a balance as we might hope.
  • Interviewees in the film also discuss research on unconscious gender bias, which provides a possible causal mechanism for the disparities other than people consciously discriminating against women. If people aren’t consciously discriminating, our intuition is that people aren’t culpable (because they can’t help what their unconscious is up to). However, whether due to conscious choices or unconscious bias, the effects are demonstrably real, which raises the question: what do we do about it?
  • The interviewees seem pretty hesitant about “positive discrimination” in favor of women as a good way to address the gender disparity — one said she wouldn’t want to think she got her career achievements because she’s a woman, rather than because she’s very good at what she does. And yet, they seem to realize that we may have to do something beyond hoping that people’s individual evaluations become less biased. The bias is there (to the extent that, unconsciously, males are being judged as better because they’re men). It’s a systemic problem. How can we put the burden on individuals to somehow magically overcome systemic problems?
  • We see a range of opinions from very smart women who have been describing inequalities and voicing the importance of making things in STEM more equitable about whether they’d describe themselves as feminists. (One of them says, near the end, that if people don’t like the word, we need to find another one so we don’t get sidetracked from actually pursuing equality.)
  • We see a sense of urgency. Despite how much has gotten better, there are plenty of elements that still need to improve. The interviewees give the impression that we ought to be able to find effective ways to address the systemic problems, if only we can find the will to do so within the scientific community.

How important is it to find more effective ways to address gender disparities in STEM? The statistic in the film that hit me hardest is that, at our present rate of progress, it will take another 70 years to achieve gender parity. I don’t have that kind of time, and I don’t think my daughters ought to wait that long, either. To quote Prof. Lesley Yellowlees,

I’ve often heard myself say we have to be patient, but there comes a time when you have to run out of patience, because if we don’t run out of patience and we don’t start demanding more from the system, demanding that culture change to happen faster than it’s happening at present, then I think we not only do ourselves a disservice, but we do the generations both past and the ones to come a huge disservice as well.

It’s been a long time since I’ve seen 13 minutes packed so effectively with so much to think about.

* * * * *
Transcript of “A Chemical Imbalance”:

Dr. Perdita Barran, Reader in Biophysical Chemistry, University of Edinburgh: I’m not sure why it is Edinburgh has such a high number of female faculty, and indeed, female postdoctoral researchers and female research fellows. One of the greatest things about this department is, because they’re are such a high proportion of female faculty — it ranges between 20 and even up to 30 percent at a few times — it becomes less important and we are less prone to the gender bias, because you don’t need to do it. You just think of scientists as scientists, you don’t think of them in terms of their gender.

Prof. Eleanor Campbell FRSC FRS, Professor of Physical Chemistry, Head of School of Chemistry, University of Edinburgh: It’s very difficult to put your finger on it, but I do feel a different atmosphere in a place where you have a significant percentage of women. That’s not to say that women can’t be confrontational and egoistical, of course they can. But on the whole, there is a difference in atmosphere.

Text on screen: 1892 Women are finally allowed to attend The University of Edinburgh as undergraduates.

Text on screen: By 1914, over 1000 women hold degrees.

Prof. Steve Chapman FRSE FRSC, Principal & Vice Chancellor, Heriot-Watt University: There’s still not enough women representation in STEM at all levels, but it gets worse the higher you go up, and when you go to management levels, I think, there is a serious disparity.

Prof. Eleanor Campbell: Yeah, the leaky pipeline is a sort of worrying tendency to lose women at various stages on the career path. [Graph on the screen about “Women in STEM, UK average”.] Here we [discussing the chemistry line on the graph] have roughly 50-50 in terms of male/female numbers at the undergraduate level. It [the proportion of women] drops a little bit at postgraduate level, and then it dives going to postdocs and onward, and that is extremely worrying. We’re losing a lot of very, very talented people.

Text on screen: Women in STEM, UK average
Undergraduate 33%
Professor 9%
(2011 UKRC & HESA)

Dr. Elaine Murray MSP, Shadow Minister for Housing & Transport, Scottish Parliament: I feel that I did — 25 years ago I made the choice between remaining in science and my family. You know, 52% of women who’ve been trained in STEM come out of it. I’m one of them.

Prof. Anita Jones, Professor of Molecular Photophysics, University of Edinburgh: On the whole, women still do take more responsibility for the looking after children and so on. But again, I think there are things that can be put in place, improved child care facilities and so on, that can help with that, and can help to achieve an acceptable compromise between the two.

Dr. Marjorie Harding, Honorary Fellow, University of Edinburgh: The division of responsibilities between husband and wife has changed a lot over the years. When I first had children, it was quite clear that it was my responsibility to cope with the home, everything that was happening there, and the children’s things, and not to expect him to have time available for that sort of thing.

Dr. Carole Morrison, Senior Lecturer in Structural Chemistry, University of Edinburgh: When the children were small, because I was working part time, I felt that I was incredibly fortunate. I was able to witness all of their little milestones. But it’s meant that my career has progressed much slower than it would have done otherwise. But, you know, life is all about compromises. I wasn’t prepared to compromise on raising my children.

Dr. Alison Hulme, Senior Lecturer in Organic Chemistry, University of Edinburgh: I don’t go out of my way to let people know that I only work at 80%, for the very fact that I don’t want them to view me as any less serious about my intentions in research.

Dr. Perdita Barran: I really understood feminism when I had children and also wanted to work. Then it really hits you how hard it is actually to be a female in science.

Text on screen: 1928 Dr. Christina Miller produces the first ever sample of pure phosphorus trioxide.
In the same year British women achieve suffrage.

Text on screen: 1949 Dr. Miller becomes the first female chemist elected to The Royal Society of Edinburgh.

Prof. Steve Chapman: Do I consider myself to be a feminist?

Prof. Anita Jones: Well, that’s an interesting question.

Dr. Perdita Barran: Uh, yeah!

Dr. Marjorie Harding: No.

Dr. Carole Morrison: No, definitely not.

Prof. Eleanor Campbell: No, I’ve never thought of myself as a feminist.

Dr. Alison Hulme: I think that people don’t want to be labeled with the tag of being a feminist because it has certain connotations associated with it that are not necessarily very positive.

Dr. Elaine Murray: I’m of an age when women were considered to be feminists, you know, most of us in the 1970s. There are battles still to be fought, but I think we had a greater consciousness of the need to define ourselves as feminists, and I would still do so. But, there’s been progress, but I think the young women still need to be aware that there’s a lot to be done. All the battles weren’t won.

Text on screen: 1970 The UK Parliament passes The Equal Pay Act.
Over 40 years later, women still earn on average 14.9% less that their male counterparts, and they get promoted less.

Prof. Polly Arnold FRSE FRSC, Crum Brown Chair of Chemistry, University of Edinburgh: The Yale study on subconscious bias was a real shocker. I realized that it was an American study, so the subjects were all American, but I don’t feel that it’s necessarily any different in the UK.

Prof. Steve Chapman: It was a very simple study, but a very telling study. They sent out CVs to people in North American institutions and the only difference in the CV was the name at the top — a male name or a female name. The contents of the CVs were identical. And when the people were asked to comment on the CVs, there was something like a 40% preference for the CV if it had a male name associated with it. Now those people I don’t think were actively trying to discriminate against women, but they were, and they were doing it subconsciously. It scared me, because of course I would go around saying, ‘I’m not prejudiced at all,’ but I read that and I thought, if I saw those CVs, would I react differently?

Dr. Janet Lovett, Royal Society University Research Fellow, University of Edinburgh: You hear the kind of results from the Yale study and unfortunately you’re not that surprised by them. And I think … I think it’s hard to explain why you’re not that surprised by them. There is an endemic sexism to most high-powered careers, I would say.

Prof. Polly Arnold: When I was a junior academic in a previous job, I was given the opportunity to go on a course to help women get promoted. The senior management at the university had looked at the data, and they’d realized that the female academics were winning lots of international prizes, being very successful internationally, but they weren’t getting promoted internally, so what we needed was a course to help us do this. And to this day, I still don’t understand how they didn’t realize that it was them that needed the course.

Dr. Elaine Murray: I think a lot of it isn’t really about legislation or regulation, it’s actually cultural change, which is more difficult to affect. And, you know, the recognition that this is part of an equality agenda, really, that we need to have that conversation which is not just about individuals, its about the experience of women in general.

Text on screen: Women without children are still 23% less likely to achieve tenure than men with children.

Prof. Anita Jones: I’m not really in favor of positive discrimination. I don’t think, as a women, I would have wanted to feel that I got a job, or a fellowship, or a grant, or whatever, because I was a woman rather than because I was very good at what I do.

Prof. Steve Chapman: I think we have to be careful. I was looking at the ratio of women in some of the things that we’re doing in my own institution, and accidentally you can heavily dominate things with males without actually thinking about it. Does that mean we have to have quotas for women? No. But does it mean we have to be pro-active in making sure we’re bringing it to the attention of women that they should be involved, and that they add value? Yes.

Dr. Elaine Murray: I was always an advocator of positive discrimination in politics, in order to address the issue of the underrepresentation of women. Now, a lot of younger women now don’t see that as important, and yet if you present them some of the issues that women face to get on, they do realize things aren’t quite as easy.

Text on screen: 2012 The School of Chemistry receives the Athena Swan Gold Award, recognising a significant progression and achievement in promoting gender equality.

Prof. Steve Chapman: We shouldn’t underestimate the signal that Athena Gold sends out. It sends out the message that this school is committed to the Athena Agenda, which isn’t actually just about women. It’s about creating an environment in which all people can thrive.

Prof. Eleanor Campbell: I think it is extremely important that the men in the department have a similar view when it comes to supporting young academics, graduate students, postdocs, regardless of their gender. I think that’s extremely important. And, I mean, certainly here, our champion for our Athena Swan activities is a male, and I deliberately wanted to have a younger male doing that job, to make it clear that it wasn’t just about women, that it was about really improving conditions for everybody.

Dr. Elaine Murray: I know, for example, in the Scottish government, equalities is somehow lumped in with health, but it’s not. You know, health is such a big portfolio that equalities is going to get pretty much lost in the end, and I think probably there’s a need for equalities issues to take a higher profile at a governmental level. And I think also it’s still about challenging the media, about the sort of stereotypes which surround women more generally, and still in science.

Text on screen: 2012 Prof. Lesley Yellowlees becomes the first female President of The Royal Society of Chemistry.

Prof. Lesley Yellowlees MBE FRSE FRSC, Professor of Inorganic Electrochemistry, Vice Principal & Head of the College of Science & Engineering, University of Edinburgh, President of The Royal Society of Chemistry: I’ve often heard myself say we have to be patient, but there comes a time when you have to run out of patience, because if we don’t run out of patience and we don’t start demanding more from the system, demanding that culture change to happen faster than it’s happening at present, then I think we not only do ourselves a disservice, but we do the generations both past and the ones to come a huge disservice as well.

Text on screen: At our current rate of progress it will take 70 years before we achieve parity between the sexes.

Prof. Polly Arnold: If we’re unwilling to define ourselves as feminists, we need to replace the word with something more palatable. The concept of equality is no less relevant today.

Ethical and practical issues for uBiome to keep working on.

Posted on by

Earlier this week, the Scientific American Guest Blog hosted a post by Jessica Richman and Zachary Apte, two members of the team at uBiome, a crowdfunded citizen science start-up. Back in February, as uBiome was in the middle of its crowdfunding drive, a number of bloggers (including me) voiced worries that some of the ethical issues of the uBiome project might require more serious attention. Partly in response to those critiques, Richman’s and Apte’s post talks about their perspectives on Institutional Review Boards (IRBs) and how in their present configuration they seem suboptimal for commercial citizen science initiatives.

Their post provides food for thought, but there are some broader issues about which I think the uBiome team should think a little harder.

Ethics takes more than simply meeting legal requirements.

Consulting with lawyers to ensure that your project isn’t breaking any laws is a good idea, but it’s not enough. Meeting legal requirements is not sufficient to meet your ethical obligations (which are well and truly obligations even when they lack the force of law).

Now, it’s the case that there is often something like the force of law deployed to encourage researchers (among others) not to ignore their ethical obligations. If you accept federal research funds, for example, you are entering into a contract one of whose conditions is forking within federal guidelines for ethical use of animal or human subjects. If you don’t want the government to enforce this agreement, you can certainly opt out of taking the federal funds.

However, opting out of federal funding does not remove your ethical duties to animals or human subjects. It may remove the government’s involvement in making you live up to your ethical obligations, but the ethical obligations are still there.

This is a tremendously important point — especially in light of a long history of human subjects research in which researchers have often not even recognized their ethical obligations to human subjects, let alone had a good plan for living up to them.

Here, it is important to seek good ethical advice (as distinct from legal advice), from an array of ethicists, including some who see potential problems with your plans. If none of the ethicists you consult see anything to worry about, you probably need to ask a few more! Take the potential problems they identify seriously. Think through ways to manage the project to avoid those problems. Figure out a way to make things right if a worst case scenario should play out.

In a lot of ways, problems that uBiome encountered with the reception of its plan seemed to flow from a lack of good — and challenging — ethical advice. There are plenty of other people and organizations doing citizen science projects that are similar enough to uBiome (from the point of view of interactions with potential subjects/participants), and many of these have experience working with IRBs. Finding them and asking for their guidance could have helped the uBiome team foresee some of the issues with which they’re dealing now, somewhat late in the game.

There are more detailed discussions of the chasm between what satisfies the law and what’s ethical at The Broken Spoke and Drugmonkey. You should, as they say, click through and read the whole thing.

Some frustrations with IRBs may be based on a misunderstanding of how they work.

An Institutional Review Board, or IRB, is a body that examines scientific protocols to determine whether they meet ethical requirements in their engagement of human subjects (including humans who provide tissue or other material to a study). The requirement for independent ethical evaluation of experimental protocols was first articulated in the World Medical Association’s Declaration of Helsinki, which states:

The research protocol must be submitted for consideration, comment, guidance and approval to a research ethics committee before the study begins. This committee must be independent of the researcher, the sponsor and any other undue influence. It must take into consideration the laws and regulations of the country or countries in which the research is to be performed as well as applicable international norms and standards but these must not be allowed to reduce or eliminate any of the protections for research subjects set forth in this Declaration. The committee must have the right to monitor ongoing studies. The researcher must provide monitoring information to the committee, especially information about any serious adverse events. No change to the protocol may be made without consideration and approval by the committee.

(Bold emphasis added.)

In their guest post, Richman and Apte assert, “IRBs are usually associated with an academic institution, and are provided free of charge to members of that institution.”

It may appear that the services of an IRB are “free” to those affiliated with the institution, but they aren’t really. Surely it costs the institution money to run the IRB — to hire a coordinator, to provide ethics training resources for IRB members and to faculty, staff, and students involved in human subjects research, to (ideally) give release time to faculty and staff on the IRB so they can actually devote the time required to consider protocols, comment upon them, provide guidance to PIs, and so forth.

Administrative costs are part of institutional overhead, and there’s a reasonable expectation that researchers whose protocols come before the IRB will take a turn serving on the IRB at some point. So IRBs most certainly aren’t free.

Now, given that the uBiome team was told they couldn’t seek approval from the IRBs at any institutions where they plausibly could claim an affiliation, and given the expense of seeking approval from a private-sector IRB, I can understand why they might have been hesitant to put money down for IRB approval up front. They started with no money for their proposed project. If the project itself ended up being a no-go due to insufficient funding, spending money on IRB approval would seem pointless.

However, it’s worth making it clear that expense is not in itself a sufficient reason to do without ethical oversight. IRB oversight costs money (even in an academic institution where those costs are invisible to PIs because they’re bundled into institutional overhead). Research in general costs money. If you can’t swing the costs (including those of proper ethical oversight), you can’t do the research. That’s how it goes.

Richman and Apte go on:

[W]e wanted to go even further, and get IRB approval once we were funded — in case we wanted to publish, and to ensure that our customers were well-informed of the risks and benefits of participation. It seemed the right thing to do.

So, we decided to wait until after crowdfunding and, if the project was successful, submit for IRB approval at that point.

Getting IRB approval at some point in the process is better than getting none at all. However, some of the worries people (including me) were expressing while uBiome was at the crowdfunding stage of the process (before IRB approval) were focused on how the lines between citizen scientist, human subject, and customer were getting blurred.

Did donors to the drive believe that, by virtue of their donations, they were guaranteed to be enrolled in the study (as sample providers)? Did they have a reasonable picture of the potential benefits of their participation? Did they have a reasonable picture of the potential risks of their participation?

These are not questions we leave to PIs. To assess them objectively, we put these questions before a neutral third-party … the IRB.

If the expense of formal IRB consideration of the uBiome protocol was prohibitive during the crowdfunding stage, it surely would have gone some way to meeting ethical duties if the uBiome team had vetted the language in their crowdfunding drive with independent folks attentive to human subjects protection issues. That the ethical questions raised by their fundraising drive were so glaringly obvious to so many of us suggests that skipping this step was not a good call.


We next arrive at the issue of the for-profit IRB. Richman and Apte write:

Some might criticize the fact that we are using a private firm, one not connected with a prestigious academic institution. We beg to differ. This is the same institution that works with academic IRBs that need to coordinate multi-site studies, as well as private firms such as 23andme and pharmaceutical companies doing clinical trials. We agree that it’s kind of weird to pay for ethical review, but that is the current system, and the only option available to us.

I don’t think paying for IRB review is the ethical issue. If one were paying for IRB approval, that would be an ethical issue, and there are some well known rubber-stamp-y private IRBs out there.

Carl Elliott details some of the pitfalls of the for-profit IRB in his book White Coat, Black Hat. The most obvious of these is that, in a competition for clients, a for-profit IRB might well feel a pressure to forego asking the hard questions, to be less ethically rigorous (and more rubber-stamp-y) — else clients seeking approval would take their business to a competing IRB they saw as more likely to grant that approval with less hassle.

Market forces may provide good solutions to some problems, but it’s not clear that the problem of how to make research more ethical is one of them. Also, it’s worth noting that being a citizen science project does not in and of itself preclude review by an academic IRB – plenty of citizen science projects run by academic scientists do just that. It’s uBiome’s status as a private-sector citizen science project that led to the need to find another IRB.

That said, if folks with concerns knew which private IRB the uBiome team used (something they don’t disclose in their guest post), those folks could inspect the IRB’s track record for rigor and make a judgment from that.

Richman and Apte cite as further problems with IRBs, at least as currently constituted, lack of uniformity across committees and lack of transparency. The lack of uniformity is by design, the thought being that local control of committees should make them more responsive to local concerns (including those of potential subjects). Indeed, when research is conducted by collaborators from multiple institutions, one of the marks of good ethical design is when different local IRBs are comfortable approving the protocol. As well, at least part of the lack of transparency is aimed at human subjects protection — for example, ensuring that the privacy of human subjects is not compromised in the release of approved research protocols.

This is not to say that there is no reasonable discussion to have about striving for more IRB transparency, and more consistency between IRBs. However, such a discussion should center ethical considerations, not convenience or expediency.

Focusing on tone rather than substance makes it look like you don’t appreciate the substance of the critique.

Richman and Apte write the following of the worries bloggers raised with uBiome:

Some of the posts threw us off quite a bit as they seemed to be personal attacks rather than reasoned criticisms of our approach. …

We thought it was a bit… much, shall we say, to compare us to the Nazis (yes, that happened, read the posts) or to the Tuskegee Experiment because we funded our project without first paying thousands of dollars for IRB approval for a project that had not (and might never have) happened.

I have read all of the linked posts (here, here, here, here, here, here, here, and here) that Richman and Apte point to in leveling this complaint about tone. I don’t read them as comparing the uBiome team to Nazis or the researchers who oversaw the Tuskegee Syphilis Experiment.

I’m willing to stipulate that the tone of some of these posts was not at all cuddly. It may have made members of the uBiome team feel defensive.

However, addressing the actual ethical worries raised in these posts would have done a lot more for uBiome’s efforts to earn the public’s trust than adopting a defensive posture did.

Make no mistake, harsh language or not, the posts critical of uBiome were written by a bunch of people who know an awful lot about the ins and outs of ethical interactions with human subjects. These are also people who recognize from their professional lives that, while hard questions can feel like personal attacks, they still need to be answered. They are raising ethical concerns not to be pains, but because they think protecting human subjects matters — as does protecting the collective reputation of those who do human subjects research and/or citizen science.

Trust is easier to break than to build, which means one project’s ethical problems could be enough to sour the public on even the carefully designed projects of researchers who have taken much more care thinking through the ethical dimensions of their work. Addressing potential problems in advance seems like a better policy than hoping they’ll be no big deal.

And losing focus on the potential problems because you don’t like the way in which they were pointed out seems downright foolish.

Much of uBiome’s response to the hard questions raised about the ethics of their project has focused on tone, or on meeting examples that provide historical context for our ethical guidelines for human subject research with the protestation, “We’re not like that!” If nothing else, this suggests that the uBiome team hasn’t understood the point the examples are meant to convey, nor the patterns that they illuminate in terms of ethical pitfalls into which even non-evil scientists can fall if they’re not careful.

And it is not at all clear that the uBiome team’s tone in blog comments and on social media like Twitter has done much to help its case.

What is still lacking, amidst all their complaints about the tone of the critiques, is a clear account of how basic ethical questions (such as how uBiome will ensure that the joint roles of customer, citizen science participant, and human subject don’t lead to a compromise of autonomy or privacy) are being answered in uBiome’s research protocol.

A conversation on the substance of the critiques would be more productive here than one about who said something mean to whom.

Which brings me to my last issue:

New models of scientific funding, subject recruitment, and outreach that involve the internet are better served by teams that understand how the internet works.

Let’s say you’re trying to fund a project, recruit participants, build general understanding, enthusiasm, support, and trust. Let’s say that your efforts involve websites where you put out information and social media use where you amplify some of that information or push links to your websites or favorable media coverage.

People looking at the information you’ve put out there are going to draw conclusions based on the information you’ve made public. They may also draw speculative conclusions from the gaps — the information you haven’t made public.

You cannot, however, count on them to base their conclusions on information to which they’re not privy, including what’s in you’re heart.

There may be all sorts of good efforts happening behind the scenes to get rigorous ethical oversight off the ground. If it’s invisible to the public, there’s no reason the public should assume it’s happening.

If you want people to draw more accurate conclusions about what you’re doing, and about what potential problems might arise (and how you’re preparing to face them if they do), a good way to go is to make more information public.

Also, recognize that you’re involved in a conversation that is being conducted publicly. Among other things, this means it’s unreasonable to expect people with concern to take it to private email in order to get further information from you. You’re the one with a project that relies on cultivating public support and trust; you need to put the relevant information out there!

(What relevant information? Certainly the information relevant to responding to concerns and critiques articulated in the above-linked blog posts would be a good place to start — which is yet another reason why it’s good to be able to get past tone and understand substance.)

In a world where people email privately to get the information that might dispel their worries, those people are the only ones whose worries are addressed. The rest of the public that’s watching (but not necessarily tweeting, blogging, or commenting) doesn’t get that information (especially if you ask the people you email not to share the content of that email publicly). You may have fully lost their trust with nary a sign in your inboxes.

Maybe you wish the dynamics of the internet were different. Some days I do, too. But unless you’re going to fix the internet prior to embarking on your brave new world of crowdfunded citizen science, paying some attention to the dynamics as they are now will help you use it productively, rather than to create misunderstandings and distrust that then require remediation.

That could clear the way to a much more interesting and productive conversation between uBiome, other researchers, and the larger public.

When we target chemophobia, are we punching down?

Posted on by

Over at Pharyngula, Chris Clarke challenges those in the chemical know on their use of “dihydrogen monoxide” jokes. He writes:

Doing what I do for a living, I often find myself reading things on Facebook, Twitter, or those increasingly archaic sites called “blogs” in which the writer expresses concern about industrial effluent in our air, water, consumer products or food. Sometimes the concerns are well-founded, as in the example of pipeline breaks releasing volatile organic chemicals into your backyard. Sometimes, as in the case of concern over chemtrails or toxic vaccines, the concerns are ill-informed and spurious.

And often enough, the educational system in the United States being the way it’s been since the Reagan administration, those concerns are couched in terms that would not be used by a person with a solid grounding in science. People sometimes miss the point of dose-dependency, of acute versus chronic exposure, of the difference between parts per million and parts per trillion. Sometimes their unfamiliarity with the basic facts of chemistry causes them to make patently ridiculous alarmist statements and then double down on them when corrected.

And more times than I can count, if said statements are in a public venue like a comment thread, someone will pipe up by repeating a particular increasingly stale joke. Say it’s a discussion of contaminants in tap water allegedly stemming from hydraulic fracturing for natural gas extraction. Said wit will respond with something like:

“You know what else might be coming out of your tap? DIHYDROGEN MONOXIDE!”

Two hydrogens, one oxygen … what’s coming out of your tap here is water. Hilarious! Or perhaps not.

Clarke argues that those in the chemical know whip out the dihydrogen monoxide joke to have a laugh at the expense of someone who doesn’t have enough chemical knowledge to understand whether conditions they find alarming really ought to alarm them. However, how it usually goes down is that other chemically literate people in earshot laugh while the target of the joke ends up with no better chemical understanding of things.

Really, all the target of the joke learns is that the teller of the joke has knowledge and is willing to use it to make someone else look dumb.

Clarke explains:

Ignorance of science is an evil that for the most part is foisted upon the ignorant. The dihydrogen monoxide joke depends for its humor on ridiculing the victims of that state of affairs, while offering no solution (pun sort of intended) to the ignorance it mocks. It’s like the phrase “chemophobia.” It’s a clan marker for the Smarter Than You tribe.

The dihydrogen monoxide joke punches down, in other words. It mocks people for not having had access to a good education. And the fact that many of its practitioners use it in order to belittle utterly valid environmental concerns, in the style of (for instance) Penn Jillette, makes it all the worse — even if those concerns aren’t always expressed in phraseology a chemist would find beyond reproach, or with math that necessarily works out on close examination.

There’s a weird way in which punching down with the dihydrogen monoxide joke is the evil twin of the “deficit model” in science communication.

The deficit model assumes that the focus in science communication to audiences of non-scientists should be squarely on filling in gaps in their scientific knowledge, teaching people facts and theories that they didn’t already know, as if that is the main thing they must want from science. (It’s worth noting that the deficit model seems to assume a pretty unidirectional flow of information, from the science communicator to the non-scientist.)

The dihydrogen monoxide joke, used the way Clarke describes, identifies a gap in understanding and then, instead of trying to fill it, points and laughs. If the deficit model naïvely assumes that filling gaps in knowledge will make the public cool with science, this kind of deployment of the dihydrogen monoxide joke seems unlikely to provoke any warm feelings towards science or scientists from the person with a gappy understanding.

What’s more, this kind of joking misses an opportunity to engage with what they’re really worried about and why. Are they scared of chemicals per se? Of being at the mercy of others who have information about which chemicals can hurt us (and in which amounts) and/or who have more knowledge about or control of where those chemicals are in our environment? Do they not trust scientists at all, or are they primarily concerned about whether they can trust scientists in the employ of multinational corporations?

Do their concerns have more to do with the information and understanding our policymakers have with regard to chemicals in our world — particularly about whether these policymakers have enough to keep us relatively safe, or about whether they have the political will to do so?

Actually having a conversation and listening to what people are worried about could help. It might turn out that people with the relevant scientific knowledge to laugh at the dihydrogen monoxide joke and those without share a lot of the same concerns.

Andrew Bissette notes that there are instances where the dihydrogen monoxide joke isn’t punching down but punching up, where educated people who should know better use large platforms to take advantage of the ignorant. So perhaps it’s not the case that we need a permanent moratorium on the joke so much as more careful thought about what we hope to accomplish with it.

Let’s return to Chris Clarke’s claim that the term “chemophobia” is “a clan marker for the Smarter Than You tribe.”

Lots of chemists in the blogosphere regularly blog and tweet about chemophobia. If they took to relentlessly tagging as “chemophobe!” people who are lacking access to the body of knowledge and patterns of reasoning that define chemistry, I’d agree that it was the same kind of punching down as the use of the dihydrogen monoxide joke Clarke describes. To the extent that chemists are actually doing this to assert membership in the Smarter Than You tribe, I think it’s counterproductive and mean to boot, and we should cut it out.

But, knowing the folks I do who blog and tweet about chemophobia, I’m pretty sure their goal is not to maintain clear boundaries between The Smart and The Dumb. When they fire off a #chemophobia tweet, it’s almost like they’re sending up the Batsignal, rallying their chemical community to fight some kind of crime.

So what is it these chemists — the people who have access to the body of knowledge and patterns of reasoning that define chemistry — find problematic about the “chemophobia” of others? What do they hope to accomplish by pointing it out?

Part of where they’re coming from is probably grounded in good old fashioned deficit-model reasoning, but with more emphasis on helping others learn a bit of chemistry because it’s cool. There’s usually a conviction that the basics of the chemistry that expose the coolness are not beyond the grasp of adults of normal intelligence — if only we explain in accessibly enough. Ash Jogalekar suggests more concerted efforts in this direction, proposing a lobby for chemistry (not the chemical industry) that takes account of how people feel about chemistry and what they want to know. However it’s done, the impulse to expose the cool workings of a bit of the world to those who want to understand them should be offered as a kindness. Otherwise, we’re doing it wrong.

Another part of what moves the chemists I know who are concerned with chemophobia is that they don’t want people who are not at home with chemistry to get played. They don’t want them to be vulnerable to quack doctors, nor to merchants of doubt trying to undermine sound science to advance a particular economic or political end, nor to people trying to make a buck with misleading claims, nor to legitimately confused people who think they know much more than they really do.

People with chemical know-how could help address this kind of vulnerability, being partners to help sort out the reliable information from the bogus, the overblown risks from risks that ought to be taken seriously or investigated further.

But short of teaching the folks without access to the body of knowledge and patterns of reasoning that define chemistry everything they know to be their own experts (which is the deficit model again), providing this kind of help requires cultivating trust. It requires taking the people to whom your offering the help seriously, recognizing that gaps in their chemical understanding don’t make them unintelligent or of less value as human beings.

And laughing at the expense of the people who could use your help — using your superior chemical knowledge to punch down — seems unlikely to foster that trust.

Professional communities, barriers to inclusion, and the value of a posse.

Posted on by

Last week, I wrote a post about an incident connected to a professional conference. A male conference-goer wrote a column attempting to offer praise for a panel featuring four female conference-goers but managed to package this praise in a way that reinforced sexist assumptions about the value women colleagues add to a professional community.

The women panelists communicated directly with the male commentator about his problematic framing. The male commentator seemed receptive to this feedback. I blogged about it as an example of why it’s important to respond to disrespect within professional communities, even if it’s not intended as disrespect, and despite the natural inclination to let it go. And my post was praised for offering a discussion of the issue that was calm, sensitive, and measured.

But honestly? I’m unconvinced that my calm, sensitive, measured discussion will do one whit of good to reduce the incidence of such casual sexism in the future, in the community of science journalist or in any other professional community. Perhaps there were some readers who, owing to the gentle tone, were willing to examine the impact of describing colleagues who are women primarily in terms of their looks, but if a less gentle tone would have put them off from considering the potential for harm to members of their professional communities, it’s hard to believe these readers would devote much energy to combatting these harms — whether or not they were being asked nicely to do so.

Sometimes someone has to really get your attention — in a way that shakes you up and makes you deeply uncomfortable — in order for you to pay attention going forward. Maybe feeling bad about the harm to someone else is a necessary first step to developing empathy.

And certainly, laying out the problem while protecting you from what it feels like to be one of the people struggling under the effects of that problem takes some effort. If going to all that trouble doesn’t actually leave enough of an impression to keep the problem from happening some more, what’s the point?

* * * * *

What does it take to create a diverse professional community? It requires more than an absence of explicit rules or standing practices that bar certain kinds of people from membership, more even that admitting lots of different kinds of people into the “pipeline” for that profession. If you’re in the community by virtue of your educational or employment status but you’re not actually part of the discussions that define your professional community, it may help the appearance of diversity, but not the reality of it.

The chilly climate women have been talking about in a variety of male-dominated professional communities is a real thing.

Being a real member of a professional community includes being able to participate fully in venues for getting your work and insights into the community’s discussions. These venues include journals and professional meetings, as well as panels or study sections that evaluate grant proposals. Early in one’s membership in a professional community, venues like graduate seminars and department symposia are also really important.

One problem here is that usually individuals without meaningful access to participation are also without the power in the community required to effectively address particular barriers to their access. Such individuals can point out the barriers, but they are less likely to be listened to than someone else in the community without those barriers.

Everyday sexism is just one such barrier.

This barrier can take a number of particular forms.

For the students on their way into a professional community, it’s a barrier to find out that senior members of the community who you expected would help train you and eventually take you seriously as a colleague are more inclined to sexualize you or full-on sexually harass you. It’s a barrier when you see people in your community minimize that behavior, whether offhandedly or with rather more deliberation.

It’s a barrier when members of your community focus on your looks rather than your intellectual contributions, or act like it’s cute or somehow surprising that someone like you could actually make an intellectual contribution. It’s a further barrier when other members of your community advise you to ignore tangible disrespect because surely it wasn’t intentional — especially when those other members of the community make no visible effort to help address the disrespect.

It’s a barrier when students don’t see people like themselves represented among the recognized knowledge-builders in the professional community as they are being taught the core knowledge expected of members of that community. It’s also a barrier when the more senior members of the professional community are subject to implicit biases in their expert evaluations of who’s cut out to be a full contributing member of the community.

Plenty of well-meaning folks in professional communities that have a hard time fully integrating women (among others) may be puzzled as to why this is so. If they don’t personally experience the barriers, they may not even realize that they’re there. Listening to lived experiences of their female colleagues might reveal some of the barriers — but listening also assumes that the community really takes its female members seriously as part of the community, when this is precisely the problem with which the women in the community are struggling.

* * * * *

Professional meetings can be challenging terrain for women in predominantly male professional communities. Such meetings are essential venues in which to present one’s work and get career credit for doing so. They are also crucially important for networking and building relationships with people who might become collaborators, who will be called on to evaluate one’s work, and who are the peers with whom one hopes to be engaged in productive discussions over the course of one’s career.

There is also a strong social component to these meetings, an imperative to have fun with one’s people — which is to say, in this context, the people with whom one shares a professional community. Part of this, I think, is related to how strongly people identify with their professional community: the connection is not just about what people in that community do but about who they are. They have taken on the values and goals of the professional community as their own. It’s not just a job, it’s a social identity.

For some people, the social component of professional meetings has a decidedly carnal flavor. Unfortunately, rejecting a pass from someone in your professional community, especially someone with more power in that community than you, can screw with your professional relationships within the community — even assuming that the person who made the pass accepts your “no” and moves on. In other cases, folks within the professional community may be perfectly aware of power gradients and willing to use them to get what they want, applying persistent unwanted attention that can essentially deprive the target of full participation in the conference. Given the importance professional conferences have, this is a significant professional harm.

Lest you imagine that this is a merely hypothetical worry, I assure you that it is not. If you ask around you may discover that some of the members of your professional community choose which conference sessions to attend in order to avoid their harassers. That is surely a constraint on how much one can get out of a professional meeting.

Recently a number of conferences and conventions have adopted policies against harassment, policies that are getting some use. Many of these are fan-oriented conventions or tech conferences, rather than the kind of research oriented, academically inclined professional meetings most of us university types attend. I know of at least one scientific professional society (the American Astronomical Society) that has adopted a harassment policy for its meetings and that seems generally to be moving in a good direction from the point of view of building an inclusive community. However, when I checked the websites of three professional societies to which I belong (American Chemical Society, American Philosophical Association, and Philosophy of Science Association), I could find no sign of anti-harassment policies for their conferences. This is disappointing, but not surprising to me.

The absence of anti-harassment policies doesn’t mean that there’s no harassment happening at the meetings of these professional societies, either.

And even if a professional community has anti-harassment policies in place for its meetings, this doesn’t remove the costs — especially on a relatively junior member of the community — associated with asking that the policies be enforced. Will a professional society be willing to caution a member of the program committee for the conference? To eject the most favored grad student of a luminary in the field — or, for that matter, a luminary — who violates the policy? Shining light on over-the-line behavior at conferences is a species of whistleblowing, and is likely to be received about as warmly as other forms.

* * * * *

Despite the challenges, I don’t think the prospects for building diverse and productive professional communities are dim. Progress is being made, even if most weeks the pace of progress is agonizingly slow.

But I think things could get better faster if people who take their professional communities for granted step up and become more active in maintaining them.

In much the same way that it is not science that is self-correcting but rather individual scientists who bother to engage critically with particular contributions to the ongoing scientific conversation and keep the community honest, a healthy professional community doesn’t take care of itself — at least, not without effort on the part of individual members of the community.

Professional communities require everyday maintenance. They require tending to keep their collective actions aligned with the values members of the community say they share.

People who work very hard to be part of a professional community despite systemic barriers are people committed enough to the values of the professional community to fight their way through a lot of crap. These are people who really care about the values you purport to care about as a member of the professional community, else why would they waste their time and effort fighting through the crap?

These are the kind of people you should want as colleagues, at least if you value what you say you value. Their contributions could be huge in accomplishing your community’s shared goals and ensuring your community a vibrant future.

Even more than policies that aim to address systemic barriers to their entry to the professional community, these people need a posse. They need others in the community who are unwilling to sacrifice their values — or the well-being of less powerful people who share those values — to take consistent stands against behaviors that create barriers and that undermine the shared work of the community.

These stands needn’t be huge heroic gestures. It could be as simple as reliably being that guy who asks for better gender balance in planning seminars, or who reacts to casual sexist banter with, “Dude, not cool!” It could take the form of asking about policies that might lessen barriers, and taking on some of the work involved in creating or implementing them.

It could be listening to your women colleagues when they describe what it has been like for them within your professional community and assuming the default position of believing them, rather than looking for possible ways they must have misunderstood their own experiences.

If you care about your professional community, in other words, the barriers to entry in the way of people who want badly to be part of that community because they believe fiercely in its values are your problem, too. Acting like it, and doing your part to address these barriers, is sharing the regular maintenance of the professional community you count on.

_____________
While this post is focused on barriers to full participation in professional communities that flow from gender bias, there are plenty of other types of bias that throw up similar barriers, and that could benefit from similar types of response from members of the professional communities not directly targeted by these biases.

Addressing (unintended) disrespect in your professional community.

Posted on by

I am a believer in the power of the professional conference. Getting people in the same room to share ideas, experiences, and challenges is one of the best ways to build a sense of community, to break down geographical and generational barriers, to energize people and remind them what they love about what they’re doing.

Sometimes, though, interactions flowing from a professional conference have a way of reinforcing barriers. Sometimes a member of the community makes an attempt to express appreciation of colleagues that actually has the effect of treating those colleagues like they’re not really part of the community after all.

Last week, the 8th World Conference of Science Journalists met in Helsinki, Finland. Upon his return from the conference, journalist Nicolás Luco posted a column reflecting on his experience there. (Here’s an English translation of the column by Wladimir Labeikovsky.) Luco’s piece suggests some of the excitement of finding connections with science journalists from other countries, as well as finding common ground with journalists entering the profession in a very different decade with a panoply of different technological tools:

If I hadn’t come, I wouldn’t have had that experience. I have submerged into an atmosphere where what I had seen as the future is already taken for granted. And yet, the fundamentals [e.g., that the story is what matters] remain.

It is, without a doubt, a description of a very positive personal experience.

However, Luco’s column is also a description of his experience of female colleagues at this conference framed primarily in terms of their physical attributes: shining blonde hair, limpid blue eyes, translucent complexions, apparent youth. His description of the panel of journalists using the tools of new media to practice the fundamentals of good journalism describes them as

four Americans: Rose, Lena, Kathleen and Erin (blonde), none older than 25

All of the other conference-goers who are identified by name are identified with surnames as well as given names. We do learn of the two women identified by their full names in the column that they are not blonde. It is left to the reader to imagine the hair color of Philip J. Hilts, the only male attendee mentioned by name.

I understand that Nicolás Luco was aiming to give a vivid visual description to draw his readers into his experience of being in Helsinki for this conference, and that this description was meant to convey a positive, optimistic mood about the future of science journalism.

But I also understand that these stylistic choices carry baggage that make it harder for Rose Eveleth, and Lena Groeger, and Kathleen Raven, and Erin Podolak, the journalists on the panel, to be taken seriously within this international community of science journalists.

Their surnames matter. In a field where they want their work to be recognized, disconnecting their bylines from the valuable insights they shared as part of a conference panel is not helpful.

Moreover, I am told that the journalistic convention is to identify adults by full name, and to identify people by first name alone only when those people are children.

Eveleth, Groeger, Raven, and Podolak are not children. They may seem relatively young to a journalist who came into the profession in the age of linotype (indeed, to the extent that he underestimated their ages, which range from 25 to 30), but they are professionals. Their ages should not be a barrier to treating them as if they are full members of the professional community of science journalists, but focusing unduly on their ages could well present such a barrier.

And, needless to say, their hair color should have no relevance at all in assessing whether they are skilled journalists with valuable insights to share.

As it happens, only days before the 8th World Conference of Science Journalists, Podolak wrote a blog post describing why she needs feminism. In that post, she wrote:

I’m a feminist for myself because yes, I want a fair shake, I want to be recognized for the value of my work and not whether or not my hair looks shiny that day. But, adding my voice to the other feminist voices out there is about more than just me. I’ve got it pretty good. I’m not trying to argue that I don’t. But I can support the women out there who are dealing with overt sexism, who are being attacked. I can try to be an ally. That to me is the real value of feminism, of standing together.

It is profoundly disheartening to take yourself to be accepted by your professional community, valued for the skills and ideas you bring to the table, only to discover that this is not how your presumptive colleagues actually see you. You would think that other journalists should be the ones most likely to appreciate the value of using new technologies to tell compelling stories. What a disappointment to find that their focus gets stuck on the surface. Who can tell whether the work has value if the hair of the journalist is shiny?

You will likely not be surprised that Eveleth, Groeger, Raven, and Podolak were frustrated at Nicolás Luco’s description of their panel, despite understanding that Luco was trying to be flattering. In an email to Luco the four sent in response to the column, they wrote:

Leading your story with a note about your attraction to blondes and then noting Erin’s hair color, is both inappropriate and, frankly, sexist. We were not there for anyone to ogle, and our physical appearance is completely irrelevant to the point of our panel. It is important for you to understand why were are upset about your tone in this piece. Women are constantly appraised for their looks, rather than their thoughts and skills, and in writing your story the way you did you are contributing to that sexism.

And, in a postscript to that email, Kathleen Raven noted:

I was under the impression that you wrote your article using hair color as a narrative tool to tie together your meetings with journalists. I appreciate this creativity, but I am worried that American women can perceive — as we have — the article as not fully respecting us as journalists in our own right.

What Eveleth, Groeger, Raven, and Podolak are up against is a larger society that values women more for their aesthetic appeal than their professional skills. That their own professional community repeats this pattern — presenting them as first young and pretty and only secondarily as good journalists — is a source of frustration. As Eveleth wrote to me:

Last I checked, being pretty has nothing to do with your skills at any kind of journalism. Having long blonde hair is not going to get Erin the story. Erin is going to get the story because she’s good at her job, because she’s got experience and passion, because she’s talented and tough and hard working. The same goes for Kathleen and Lena. 

The idea that it is not just okay, but actually complimentary to focus on a young woman’s (or really any aged woman’s) looks as leading part of her professional identity is wrong. The idea that it’s flattering to call out Erin’s hair and age before her skills is wrong. The idea that a woman’s professional skill set is made better if she is blonde and pretty is wrong. And the idea that someone who writes something like this should just be able to pass it off as “tongue in cheek” or “a cultural difference” is also wrong.

I should pause here to take note of another dimension of professional communities in this story. There is a strong pressure to get along with one’s professional colleagues, to get along rather than raising a fuss. Arguably this pressure is stronger on newer members of a professional community, and on members of that community with characteristics (e.g., of gender, race, disability, etc.) that are not well represented in the more established members of that professional community.

Practically, this pressure manifests itself as an inclination to let things go, to refrain from pointing out the little instances which devalue one’s professional identity or status as a valued member of the community. Most of the time it seems easier to sigh and say to oneself, “Well, he meant well,” or, “What can you expect from someone of that generation/cultural background?” than to point out the ways that the comments hurt. It feels like a tradeoff where you should swallow some individual hurt for the good of the community.

But accepting this tradeoff is accepting that your full membership in the community (and that of others like you) is less important. To the extent that you believe that you make a real contribution to the community, swallowing your individual hurt is dancing on the edge of accepting what is arguably a harm to the professional community as a whole by letting the hurtful behaviors pass unexamined.

Eveleth, Groeger, Raven, and Podolak had more respect than that for their professional community, and for Nicolás Luco as a professional colleague. They did not just sigh and roll their eyes. Rather, they emailed Luco to explain what the problem was.

In his reply to them (which I quote with his permission), Luco makes it clear that he did not intend to do harm to anyone, especially not to Eveleth, Groeger, Raven, and Podolak, with his column. Still, he also makes it clear that he may not fully grasp just what the problem is:

I write as a known voice who can write tongue in cheek and get away with it because I am willing to laugh at myself. 

I strive to make what I write entertaining. And maybe sneak in the more serious arguments.

Sorry about my misjudgment on your ages.  But the point is: you are generations apart.

I did not include your last names because they would interrupt the flow of reading and clog the line with surnames, an obstacle.

Finally, it is so much in U.S. culture to discard the looks vis a vis the brains when the looks, as President Clinton knows so well, can be a good hook into the brains.  And since this is a personal column, in the first person singular, I can tell how I personally react at good looks.  For example, Ms. Anne Glover, was extraordinarily beautiful and charming besides being bright and political, which helps, in front of the probable mean thoughts of envious uglier looking colleagues.

Thank you, I still prize the panel as the best and most important in the Conference.

Is there a way Nicolás Luco could have described his personal experience of the conference, and of this panel within the conference that he found particular valuable, in a way that was entertaining, even tongue-in-cheek, while avoiding the pitfalls of describing his female colleagues in terms that undercut their status in the professional community? I think so.

He might, for example, have talked about his own expectations that journalists who are generations apart would agree upon what makes good journalism good journalism. The way that these expectations were thwarted would surely be a good opportunity to laugh at oneself.

He might even have written about his own surprise that a young women he finds attractive contributed a valuable insight — using this as an opportunity to examine this expectation and whether it’s one he ought to be carrying around with him in his professional interactions. There’s even a line in his column that seems like it might provide a hook for this bit of self-examination:

Erin, the youngest and a cancer specialist, insists that decorations don’t matter: good journalism is good journalism, period. Makes me happy.

(Bold emphasis added.)

Extending the lesson about the content of the story mattering more than the packaging to a further lesson about the professional capabilities of the storyteller mattering more than one’s reaction to her superficial appearance — that could drive home some of the value of a conference like this.

Nicolás Luco wrote the column he wrote. Eveleth, Groeger, Raven, and Podolak took him seriously as a professional colleague who is presumptively concerned to strengthen their shared community. They asked him to consider the effect of his description on members of the professional community who stand where they do, to take responsibility as a writer for even the effects of his words that he had not intended or foreseen.

Engaging with colleagues when they hurt us without meaning to is not easy work, but it’s absolutely essential to the health of a professional community. I am hopeful that this engagement will continue productively.

Strategies to address questionable statistical practices.

Posted on by

If you have not yet read all you want to read about the wrongdoing of social psychologist Diederik Stapel, you may be interested in reading the 2012 Tilburg Report (PDF) on the matter. The full title of the English translation is “Flawed science: the fraudulent research practices of social psychologist Diederik Stapel” (in Dutch, “Falende wetenschap: De fruaduleuze onderzoekspraktijken van social-psycholoog Diederik Stapel”), and it’s 104 pages long, which might make it beach reading for the right kind of person.

If you’re not quite up to the whole report, Error Statistics Philosophy has a nice discussion of some of the highlights. In that post, D. G. Mayo writes:

The authors of the Report say they never anticipated giving a laundry list of “undesirable conduct” by which researchers can flout pretty obvious requirements for the responsible practice of science. It was an accidental byproduct of the investigation of one case (Diederik Stapel, social psychology) that they walked into a culture of “verification bias”. Maybe that’s why I find it so telling. It’s as if they could scarcely believe their ears when people they interviewed “defended the serious and less serious violations of proper scientific method with the words: that is what I have learned in practice; everyone in my research environment does the same, and so does everyone we talk to at international conferences” (Report 48). …

I would place techniques for ‘verification bias’ under the general umbrella of techniques for squelching stringent criticism and repressing severe tests. These gambits make it so easy to find apparent support for one’s pet theory or hypotheses, as to count as no evidence at all (see some from their list ). Any field that regularly proceeds this way I would call a pseudoscience, or non-science, following Popper. “Observations or experiments can be accepted as supporting a theory (or a hypothesis, or a scientific assertion) only if these observations or experiments are severe tests of the theory.”

You’d imagine this would raise the stakes pretty significantly for the researcher who could be teetering on the edge of verification bias: fall off that cliff and what you’re doing is no longer worthy of the name scientific knowledge-building.

Psychology, after all, is one of those fields given a hard time by people in “hard sciences,” which are popularly reckoned to be more objective, more revealing of actual structures and mechanisms in the world — more science-y. Fair or not, this might mean that psychologist have something to prove about their hardheadedness as researchers, about the stringency of their methods. Some peer pressure within the field to live up to such standards would obviously be a good thing — and certainly, it would be a better thing for the scientific respectability of psychology than an “everyone is doing it” excuse for less stringent methods.

Plus, isn’t psychology a field whose practitioners should have a grip on the various cognitive biases to which we humans fall prey? Shouldn’t psychologists understand better than most the wisdom of putting structures in place (whether embodied in methodology or in social interactions) to counteract those cognitive biases?

Remember that part of Stapel’s M.O. was keeping current with the social psychology literature so he could formulate hypotheses that fit very comfortably with researchers’ expectations of how the phenomena they studied behaved. Then, fabricating the expected results for his “investigations” of these hypotheses, Stapel caught peer reviewers being credulous rather than appropriately skeptical.

Short of trying to reproduce the experiments Stapel described themselves, how could peer reviewers avoid being fooled? Mayo has a suggestion:

Rather than report on believability, researchers need to report the properties of the methods they used: What was their capacity to have identified, avoided, admitted verification bias? The role of probability here would not be to quantify the degree of confidence or believability in a hypothesis, given the background theory or most intuitively plausible paradigms, but rather to check how severely probed or well-tested a hypothesis is– whether the assessment is formal, quasi-formal or informal. Was a good job done in scrutinizing flaws…or a terrible one?  Or was there just a bit of data massaging and cherry picking to support the desired conclusion? As a matter of routine, researchers should tell us.

I’m no social psychologist, but this strikes me as a good concrete step that could help peer reviewers make better evaluations — and that should help scientists who don’t want to fool themselves (let alone their scientific peers) to be clearer about what they really know and how well they really know it.

The continuum between outright fraud and “sloppy science”: inside the frauds of Diederik Stapel (part 5).

Posted on by

It’s time for one last look at the excellent article by Yudhijit Bhattacharjee in the New York Times Magazine (published April 26, 2013) on social psychologist and scientific fraudster Diederik Stapel. We’ve already examined strategy Stapel pursued to fabricate persuasive “results”, the particular harms Stapel’s misconduct did to the graduate students he was training, and the apprehensions of the students and colleagues who suspected fraud was afoot about the prospect of blowing the whistle on Stapel. To close, let’s look at some of the uncomfortable lessons the Stapel case has for his scientific community — and perhaps for other scientific communities as well.

Bhattacharjee writes:

At the end of November, the universities unveiled their final report at a joint news conference: Stapel had committed fraud in at least 55 of his papers, as well as in 10 Ph.D. dissertations written by his students. The students were not culpable, even though their work was now tarnished. The field of psychology was indicted, too, with a finding that Stapel’s fraud went undetected for so long because of “a general culture of careless, selective and uncritical handling of research and data.” If Stapel was solely to blame for making stuff up, the report stated, his peers, journal editors and reviewers of the field’s top journals were to blame for letting him get away with it. The committees identified several practices as “sloppy science” — misuse of statistics, ignoring of data that do not conform to a desired hypothesis and the pursuit of a compelling story no matter how scientifically unsupported it may be.

The adjective “sloppy” seems charitable. Several psychologists I spoke to admitted that each of these more common practices was as deliberate as any of Stapel’s wholesale fabrications. Each was a choice made by the scientist every time he or she came to a fork in the road of experimental research — one way pointing to the truth, however dull and unsatisfying, and the other beckoning the researcher toward a rosier and more notable result that could be patently false or only partly true. What may be most troubling about the research culture the committees describe in their report are the plentiful opportunities and incentives for fraud. “The cookie jar was on the table without a lid” is how Stapel put it to me once. Those who suspect a colleague of fraud may be inclined to keep mum because of the potential costs of whistle-blowing.

The key to why Stapel got away with his fabrications for so long lies in his keen understanding of the sociology of his field. “I didn’t do strange stuff, I never said let’s do an experiment to show that the earth is flat,” he said. “I always checked — this may be by a cunning manipulative mind — that the experiment was reasonable, that it followed from the research that had come before, that it was just this extra step that everybody was waiting for.” He always read the research literature extensively to generate his hypotheses. “So that it was believable and could be argued that this was the only logical thing you would find,” he said. “Everybody wants you to be novel and creative, but you also need to be truthful and likely. You need to be able to say that this is completely new and exciting, but it’s very likely given what we know so far.”

Fraud like Stapel’s — brazen and careless in hindsight — might represent a lesser threat to the integrity of science than the massaging of data and selective reporting of experiments. The young professor who backed the two student whistle-blowers told me that tweaking results — like stopping data collection once the results confirm a hypothesis — is a common practice. “I could certainly see that if you do it in more subtle ways, it’s more difficult to detect,” Ap Dijksterhuis, one of the Netherlands’ best known psychologists, told me. He added that the field was making a sustained effort to remedy the problems that have been brought to light by Stapel’s fraud.

(Bold emphasis added.)

If the writers of this report are correct, the field of psychology failed in multiple ways here. First, they were insufficiently skeptical — both of Stapel’s purported findings and of their own preconceptions — to nip Stapel’s fabrications in the bud. And, they were themselves routinely engaging in practices that were bound to mislead.

Maybe these practices don’t rise to the level of outright fabrication. However, neither do they rise to the level of rigorous and intellectually honest scientific methodology.

There could be a number of explanations for these questionable methodological choices.

Possibly some of the psychologists engaging in this “sloppy science” lack a good understanding of statistics or of what counts as a properly rigorous test of one’s hypothesis. Essentially, this is an explanation of faulty methodology on the basis of ignorance. However, it’s likely that this is culpable ignorance — that psychology researchers have a positive duty to learn what they ought to know about statistics and hypothesis testing, and to avail themselves of available resources to ensure that they aren’t ignorant in this particular way.

I don’t know if efforts to improve statistics education are a part of the “sustained effort to remedy the problems that have been brought to light by Stapel’s fraud,” but I think they should be.

Another explanation for the lax methodology decried by the report is alluded to in the quoted passage: perhaps psychology researchers let the strength of their own intuitions about what they were going to see in their research results drive their methodology. Perhaps they unconsciously drifted away from methodological rigor and toward cherry-picking and misuse of statistics and the like because they knew in their hearts what the “right” answer would be. Given this kind of conviction, of course they would reject methods that didn’t yield the “right” answer in favor of those that did.

Here, too, the explanation does not provide an excuse. The scientist’s brief is not to take strong intuitions as true, but to look for evidence — especially evidence that could demonstrate that the intuitions are wrong. A good scientist should be on the alert for instances where she is being fooled by her intuitions. Rigorous methodology is one of the tools at her disposal to avoid being fooled. Organized skepticism from her fellow scientists is another.

From here, the explanations drift into waters where the researchers are even more culpable for their sloppiness. If you understand how to test hypotheses properly, and if you’re alert enough to the seductive power of your intuitions, it seems like the other reason you might engage in “sloppy science” is to make your results look less ambiguous, more certain, more persuasive than they really are, either to your fellow scientists or to others (administrators evaluating your tenure or promotion case? the public?). Knowingly providing a misleading picture of how good your results are is lying. It may be a lie of a smaller magnitude than Diederik Stapel’s full-scale fabrications, but it’s still dishonest.

And of course, there are plenty of reasons scientists (like other human beings) might try to rationalize a little lie as being not that bad. Maybe you really needed more persuasive preliminary data than you got to land the grant without which you won’t be able to support graduate students. Maybe you needed to make your conclusions look stronger to satisfy the notoriously difficult peer reviewers at the journal to which you submitted your manuscript. Maybe you are on the verge of getting credit for a paradigm-shaking insight in your field (if only you can put up the empirical results to support it), or of beating a competing research group to the finish line for an important discovery (if only you can persuade your peers that the results you have establish that discovery).

But maybe all these excuses prioritize scientific scorekeeping to the detriment of scientific knowledge-building.

Science is supposed to be an activity aimed at building a reliable body of knowledge about the world. You can’t reconcile this with lying, whether to yourself or to your fellow scientists. This means that scientists who are committed to the task must refrain from the little lies, and that they must take serious conscious steps to ensure that they don’t lie to themselves. Anything else runs the risk of derailing the whole project.