Monthly Archives: August 2013

Credibility, bias, and the perils of having too much fun.

Posted on by

If you’re a regular reader of this blog (or, you know, attentive at all to the world around you), you will have noticed that scientific knowledge is built by human beings, creatures that, even on the job, resemble other humans more closely than they do Mr. Spock or his Vulcan conspecifics. When an experiment yields really informative results, most human scientists don’t cooly raise an eyebrow and murmur “Fascinating.” Instead, you’re likely to see a reactions somewhere on the continuum between big smiles, shouts of delight, and full-on end zone happy-dance. You can observe human scientists displaying similar emotional responses in other kinds of scientific situations, too — say, for example, when they find the fatal flaw in a competitor’s conclusion or experimental strategy.

Many scientists enjoy doing science. (If this weren’t so, the rest of us would have to feel pretty bad for making them do such thankless work to build knowledge that we’re not willing or able to build ourselves but from which we benefit nonetheless.) At least some scientists are enjoying more than just the careful work of forming hypotheses, making observations, comparing outcomes and predictions, and contributing to a more reliable account of the world and its workings. Sometimes the enjoyment comes from playing a particular kind of role in the scientific conversation.

Some scientists delight in the role of advancer or supporter of the new piece of knowledge that will change how we understand our world in some fundamental way. Other scientists delight in the role of curmudgeon, shooting down overly-bold claims. Some scientists relish being contrarians. Others find comfort in being upholders of consensus.

In light of this, we should probably consider whether having one of these human predilections like enjoying being a contrarian (or a consensus-supporter, for that matter) is a potential source of bias against which scientists should guard.

The basic problem is nothing new: what we observe, and how we interpret what we observe, can be influenced by what we expect to see — and, sometimes, by what we want to see. Obviously, scientists don’t always see what they want to see, else people’s grad school lab experiences would be deliriously happy rather than soul-crushingly frustrating. But sometimes what there is to see is ambiguous, and the person making the observation has to make a call. And frequently, with a finite set of data, there are multiple conclusions — not all of them compatible with each other — that can be drawn.

These are moments when our expectations and our ‘druthers might creep in as the tie-breaker.

At the scale of the larger community of science and the body of knowledge it produces, this may not be such a big deal. (As we’ve noted before, objectivity requires teamwork). Given a sufficiently diverse scientific community, there will be loads of other scientists who are likely to have different expectations and ‘druthers. In trying to take someone else’s result and use it to build more knowledge, the thought is that something like replication of the earlier result happens, and biases that may have colored the earlier result will be identified and corrected. (Especially since scientists are in competition for scarce goods like jobs, grants, and Nobel Prizes, you might start with the assumption that there’s no reason not to identify problems with the existing knowledge base. Of course, actual conditions on the ground for scientists can make things more complicated.)

But even given the rigorous assessment she can expect from the larger scientific community, each scientist would also like, individually, to be as unbiased as possible. One of the advantages of engaging with lots of other scientists, with different biases than your own, is you get better at noticing your own biases and keeping them on a shorter leash — putting you in a better place to make objective knowledge.

So, what if you discover that you take a lot of pleasure in being a naysayer or contrarian? Is coming to such self-awareness the kind of thing that should make you extra careful in coming to contrarian conclusions about the data? If you actually come to the awareness that you dig being a contrarian, does it put you in a better position to take corrective action than you would if you enjoyed being a contrarian but didn’t realize that being contrarian was what was bringing you the enjoyment?

(That’s right, a philosopher of science just made something like an argument that scientists might benefit — as scientists, not just as human beings — from self-reflection. Go figure.)

What kind of corrective action do I have in mind for scientists who discover that they may have a tilt, whether towards contrarianism or consensus-supporting? I’m thinking of a kind of scientific buddy-system, for example matching scientists with contrarian leanings to scientists who are made happier by consensus-supporting. Such a pairing would be useful for each scientist in the pair as far as vetting their evidence and conclusions: Here’s the scientist you have to convince! Here’s the colleague whose objections you need to understand and engage with before this goes any further!

After all, one of the things serious scientists are after is a good grip on how things actually are. An explanation that a scientist with different default assumptions than yours can’t easily dismiss is an explanation worth taking seriously. If, on the other hand, your “buddy” can dismiss your explanation, it would be good to know why so you can address its weaknesses (or even, if it is warranted, change your conclusions).

Such a buddy-system would probably only be workable with scientists who are serious about intellectual honesty and getting knowledge that is objective as possible. Among other things, this means you wouldn’t want to be paired with a scientist for whom having an open mind would be at odds with the conditions of his employment.

_____
An ancestor version of this post was published on my other blog.

Individual misconduct or institutional failing: “The Newsroom” and science.

Posted on by

I’ve been watching The Newsroom*, and in its second season, the storyline is treading on territory where journalism bears some striking similarities to science. Indeed, the most recent episode (first aired Sunday, August 25, 2013) raises questions about trust and accountability — both at the individual and the community levels — for which I think science and journalism may converge.

I’m not going to dig too deeply into the details of the show, but it’s possible that the ones I touch on here reach the level of spoilers. If you prefer to stay spoiler-free, you might want to stop reading here and come back after you’ve caught up on the show.

The central characters in The Newsroom are producing a cable news show, trying hard to get the news right but also working within the constraints set by their corporate masters (e.g., they need to get good ratings). A producer on the show, on loan to the New York-based team from the D.C. bureau, gets a lead for a fairly shocking story. He and some other members of the team try to find evidence to support the claims of this shocking story. As they’re doing this, they purposely keep other members of the production team out of the loop — not to deceive them or cut them out of the glory if, eventually, they’re able to break the story, but to enable these folks to look critically at the story once all the facts are assembled, to try to poke holes in it.** And, it’s worth noting, the folks actually in the loop, looking for information that bears on the reliability of the shocking claims in the story, are shown to be diligent about considering ways they could be wrong, identifying alternate explanations for details that seem to be support for the story, etc.

The production team looks at all the multiple sources of information they have. They look for reasons to doubt the story. They ultimately decide to air the story.

But, it turns out the story is wrong.

Worse is why key pieces of “evidence” supporting the story are unreliable. One of the interviewees is apparently honest but unreliable. One source of leaked information is false, because the person who leaked it has a grudge against a member of the production team. And, it turns out that the producer on loan from the D.C. bureau has doctored a taped interview that is the lynchpin of the story to make it appear that the interviewee said something he didn’t say.

The producer on loan from the D.C. bureau is fired. He proceeds to sue the network for wrongful termination, claiming it was an institutional failure that led to the airing of the now-retracted big story.

The parallels to scientific knowledge-building are clear.

Scientists with a hypothesis try to amass evidence that will make it clear whether the hypothesis is correct or incorrect. Rather than getting lulled into a false sense of security by observations that seem to fit the hypothesis, scientists try to find evidence that would rule out the hypothesis. They recognize that part of their job as knowledge-builders is to exercise organized skepticism — directed at their own scientific claims as well as at the claims of other scientists. And, given how vulnerable we are to our own unconscious biases, scientists rely on teamwork to effectively weed out the “evidence” that doesn’t actually provide strong support for their claims.

Some seemingly solid evidence turns out to be faulty. Measuring devices can become unreliable, or you get stuck with a bad batch of reagent, or your collaborator sends you a sample from the wrong cell line.

And sometimes a scientist who is sure in his heart he knows what the truth is doctors the evidence to “show” that truth.

Fabricating or falsifying evidence is, without question, a crime against scientific knowledge-building. But does the community that is taken in by the fraudster bear a significant share of the blame for believing him?

Generally, I think, the scientific community will say, “No.” A scientist is presumed by other members of his community to be honest unless there’s good reason to think otherwise. Otherwise, each scientist would have to replicate every observation reported by every other scientist ever before granting it any credibility. There aren’t enough grant dollars or hours in the day for that to be a plausible way to build scientific knowledge.

But, the community of science is supposed to ensure that findings reported to the public are thoroughly scrutinized for errors, not presented as more certain than the evidence warrants. The public trusts scientists to do this vetting because members of the public generally don’t know how to do this vetting themselves. Among other things, this means that a scientific fraudster, once caught, doesn’t just burn his own credibility — he can end up burning the credibility of the entire scientific community that was taken in by his lies.

Given how hard it can be to distinguish made-up data from real data, maybe that’s not fair. Still, if the scientific community is asking for the public’s trust, that community needs to be accountable to the public — and to find ways to prevent violations of trust within the community, or at least to deal effectively with those violations of trust when they happen.

In The Newsroom, after the big story unravels, as the video-doctoring producer is fired, the executive producer of the news show says, “People will never trust us again.” It’s not just the video-doctoring producer that viewers won’t trust, but the production team who didn’t catch the problem before presenting the story as reliable. Where the episodes to date leave us, it’s uncertain whether the production team will be able to win back the trust of the public — and what it might take to win back that trust.

I think it’s a reasonable question for the scientific community, too. In the face of incidents where individual scientists break trust, what does it take for the larger community of scientific knowledge-builders to win the trust of the public?

_____
* I’m not sure it’s a great show, but I have a weakness for the cadence of Aaron Sorkin’s dialogue.

** In the show, the folks who try to poke holes in the story presented with all the evidence that seems to support it are called the “red team,” and one of the characters claims its function is analogous to that of red blood cells. This … doesn’t actually make much sense, biologically. I’m putting a pin in that, but you are welcome to critique or suggest improvements to this analogy in the comments.

How far does the tether of your expertise extend?

Posted on by

Talking about science in the public sphere is tricky, even with someone with a lot of training in a science.

On the one hand, there’s a sense that it would be a very good thing if the general level of understanding of science was significantly higher than it is at present — if you could count on the people in your neighborhood to have a basic grasp of where scientific knowledge comes from, as well as of the big pieces of scientific knowledge directly relevant to the project of getting through their world safely and successfully.

But there seem to be a good many people in our neighborhood who don’t have this relationship with science. (Here, depending on your ‘druthers, you can fill in an explanation in terms of inadequately inspiring science teachers and/or curricula, or kids too distracted by TV or adolescence or whatever to engage with those teachers and/or curricula.) This means that, if these folks aren’t going to go it alone and try to evaluate putative scientific claims they encounter themselves, they need to get help from scientific experts.

But who’s an expert?

It’s well and good to say that a journalism major who never quite finished his degree is less of an authority on matters cosmological than a NASA scientist, but what should we say about engineers or medical doctors with “concerns” about evolutionary theory? Is a social scientist who spent time as an officer on a nuclear submarine an expert on nuclear power? Is an actor or talk show host with an autistic child an expert on the aetiology of autism? How important is all that specialization research scientists do? To some extent, doesn’t all science follow the same rules, thus equipping any scientist to weigh in intelligently about it?

Rather than give you a general answer to that question, I thought it best to lay out the competence I personally am comfortable claiming, in my capacity as a trained scientist.

As someone trained in a science, I am qualified:

  1. to say an awful lot about the research projects I have completed (although perhaps a bit less about them when they were still underway).
  2. to say something about the more or less settled knowledge, and about the live debates, in my research area (assuming, of course, that I have kept up with the literature and professional meetings where discussions of research in this area take place).
  3. to say something about the more or less settled (as opposed to “frontier”) knowledge for my field more generally (again, assuming I have kept up with the literature and the meetings).
  4. perhaps, to weigh in on frontier knowledge in research areas other than my own, if I have been very diligent about keeping up with the literature and the meetings and about communicating with colleagues working in these areas.
  5. to evaluate scientific arguments in areas of science other than my own for logical structure and persuasiveness (though I must be careful to acknowledge that there may be premises of these arguments — pieces of theory or factual claims from observations or experiments that I’m not familiar with — that I’m not qualified to evaluate).
  6. to recognize, and be wary of, logical fallacies and other less obvious pseudo-scientific moves (e.g., I should call shenanigans on claims that weaknesses in theory T1 necessarily count as support for alternative theory T2).
  7. to recognize that experts in fields of science other than my own generally know what the heck they’re talking about.
  8. to trust scientists in fields other than my own to rein in scientists in those fields who don’t know what they are talking about.
  9. to face up to the reality that, as much as I may know about the little piece of the universe I’ve been studying, I don’t know everything (which is part of why it takes a really big community to do science).

This list of my qualifications is an expression of my comfort level more than anything else. I would argue that it’s not elitist — good training and hard work can make a scientist out of almost anyone. But, it recognizes that with as much as there is to know, you can’t be an expert on everything. Knowing how far the tether of your expertise extends — and owning up to that when people look to you as an expert — is part of being a responsible scientist.

_______
An ancestor version of this post was published on my other blog.

“There comes a time when you have to run out of patience.”

Posted on by

In this post, I’m sharing an excellent short film called “A Chemical Imbalance,” which includes a number of brief interviews with chemists (most of them women, most at the University of Edinburgh) about the current situation for women in chemistry (and science, technology, engineering, and mathematics, or STEM, more generally) in the UK. Here’s the film:


A Chemical Imbalance
(I’m including my transcription of the film below.)

Some of the things I really appreciate about this film:

  • We get personal impressions, from women of different generations, about what it’s been like for them to be in chemistry in the UK.
  • We get numbers to quantify the gender disparity in academic chemistry in the UK, as well as to identify where in the career pipeline the disparity becomes worse. We also get numbers about how women chemists are paid relative to their male counterparts, and about relative rates of tenure that can’t be blamed on choices about childbearing and/or childrearing. There’s not just the perception of gender disparities in academic chemistry — the numbers demonstrate that the disparities are real.
  • Lurking beneath the surface is a conversation the interviewees might have had (but didn’t in the final cut) about what they count as compromises with respect to parenting and with respect to careers. My sense is that they would not all agree, and that they might not be as accepting of their colleagues’ alternative ways of striking a balance as we might hope.
  • Interviewees in the film also discuss research on unconscious gender bias, which provides a possible causal mechanism for the disparities other than people consciously discriminating against women. If people aren’t consciously discriminating, our intuition is that people aren’t culpable (because they can’t help what their unconscious is up to). However, whether due to conscious choices or unconscious bias, the effects are demonstrably real, which raises the question: what do we do about it?
  • The interviewees seem pretty hesitant about “positive discrimination” in favor of women as a good way to address the gender disparity — one said she wouldn’t want to think she got her career achievements because she’s a woman, rather than because she’s very good at what she does. And yet, they seem to realize that we may have to do something beyond hoping that people’s individual evaluations become less biased. The bias is there (to the extent that, unconsciously, males are being judged as better because they’re men). It’s a systemic problem. How can we put the burden on individuals to somehow magically overcome systemic problems?
  • We see a range of opinions from very smart women who have been describing inequalities and voicing the importance of making things in STEM more equitable about whether they’d describe themselves as feminists. (One of them says, near the end, that if people don’t like the word, we need to find another one so we don’t get sidetracked from actually pursuing equality.)
  • We see a sense of urgency. Despite how much has gotten better, there are plenty of elements that still need to improve. The interviewees give the impression that we ought to be able to find effective ways to address the systemic problems, if only we can find the will to do so within the scientific community.

How important is it to find more effective ways to address gender disparities in STEM? The statistic in the film that hit me hardest is that, at our present rate of progress, it will take another 70 years to achieve gender parity. I don’t have that kind of time, and I don’t think my daughters ought to wait that long, either. To quote Prof. Lesley Yellowlees,

I’ve often heard myself say we have to be patient, but there comes a time when you have to run out of patience, because if we don’t run out of patience and we don’t start demanding more from the system, demanding that culture change to happen faster than it’s happening at present, then I think we not only do ourselves a disservice, but we do the generations both past and the ones to come a huge disservice as well.

It’s been a long time since I’ve seen 13 minutes packed so effectively with so much to think about.

* * * * *
Transcript of “A Chemical Imbalance”:

Dr. Perdita Barran, Reader in Biophysical Chemistry, University of Edinburgh: I’m not sure why it is Edinburgh has such a high number of female faculty, and indeed, female postdoctoral researchers and female research fellows. One of the greatest things about this department is, because they’re are such a high proportion of female faculty — it ranges between 20 and even up to 30 percent at a few times — it becomes less important and we are less prone to the gender bias, because you don’t need to do it. You just think of scientists as scientists, you don’t think of them in terms of their gender.

Prof. Eleanor Campbell FRSC FRS, Professor of Physical Chemistry, Head of School of Chemistry, University of Edinburgh: It’s very difficult to put your finger on it, but I do feel a different atmosphere in a place where you have a significant percentage of women. That’s not to say that women can’t be confrontational and egoistical, of course they can. But on the whole, there is a difference in atmosphere.

Text on screen: 1892 Women are finally allowed to attend The University of Edinburgh as undergraduates.

Text on screen: By 1914, over 1000 women hold degrees.

Prof. Steve Chapman FRSE FRSC, Principal & Vice Chancellor, Heriot-Watt University: There’s still not enough women representation in STEM at all levels, but it gets worse the higher you go up, and when you go to management levels, I think, there is a serious disparity.

Prof. Eleanor Campbell: Yeah, the leaky pipeline is a sort of worrying tendency to lose women at various stages on the career path. [Graph on the screen about “Women in STEM, UK average”.] Here we [discussing the chemistry line on the graph] have roughly 50-50 in terms of male/female numbers at the undergraduate level. It [the proportion of women] drops a little bit at postgraduate level, and then it dives going to postdocs and onward, and that is extremely worrying. We’re losing a lot of very, very talented people.

Text on screen: Women in STEM, UK average
Undergraduate 33%
Professor 9%
(2011 UKRC & HESA)

Dr. Elaine Murray MSP, Shadow Minister for Housing & Transport, Scottish Parliament: I feel that I did — 25 years ago I made the choice between remaining in science and my family. You know, 52% of women who’ve been trained in STEM come out of it. I’m one of them.

Prof. Anita Jones, Professor of Molecular Photophysics, University of Edinburgh: On the whole, women still do take more responsibility for the looking after children and so on. But again, I think there are things that can be put in place, improved child care facilities and so on, that can help with that, and can help to achieve an acceptable compromise between the two.

Dr. Marjorie Harding, Honorary Fellow, University of Edinburgh: The division of responsibilities between husband and wife has changed a lot over the years. When I first had children, it was quite clear that it was my responsibility to cope with the home, everything that was happening there, and the children’s things, and not to expect him to have time available for that sort of thing.

Dr. Carole Morrison, Senior Lecturer in Structural Chemistry, University of Edinburgh: When the children were small, because I was working part time, I felt that I was incredibly fortunate. I was able to witness all of their little milestones. But it’s meant that my career has progressed much slower than it would have done otherwise. But, you know, life is all about compromises. I wasn’t prepared to compromise on raising my children.

Dr. Alison Hulme, Senior Lecturer in Organic Chemistry, University of Edinburgh: I don’t go out of my way to let people know that I only work at 80%, for the very fact that I don’t want them to view me as any less serious about my intentions in research.

Dr. Perdita Barran: I really understood feminism when I had children and also wanted to work. Then it really hits you how hard it is actually to be a female in science.

Text on screen: 1928 Dr. Christina Miller produces the first ever sample of pure phosphorus trioxide.
In the same year British women achieve suffrage.

Text on screen: 1949 Dr. Miller becomes the first female chemist elected to The Royal Society of Edinburgh.

Prof. Steve Chapman: Do I consider myself to be a feminist?

Prof. Anita Jones: Well, that’s an interesting question.

Dr. Perdita Barran: Uh, yeah!

Dr. Marjorie Harding: No.

Dr. Carole Morrison: No, definitely not.

Prof. Eleanor Campbell: No, I’ve never thought of myself as a feminist.

Dr. Alison Hulme: I think that people don’t want to be labeled with the tag of being a feminist because it has certain connotations associated with it that are not necessarily very positive.

Dr. Elaine Murray: I’m of an age when women were considered to be feminists, you know, most of us in the 1970s. There are battles still to be fought, but I think we had a greater consciousness of the need to define ourselves as feminists, and I would still do so. But, there’s been progress, but I think the young women still need to be aware that there’s a lot to be done. All the battles weren’t won.

Text on screen: 1970 The UK Parliament passes The Equal Pay Act.
Over 40 years later, women still earn on average 14.9% less that their male counterparts, and they get promoted less.

Prof. Polly Arnold FRSE FRSC, Crum Brown Chair of Chemistry, University of Edinburgh: The Yale study on subconscious bias was a real shocker. I realized that it was an American study, so the subjects were all American, but I don’t feel that it’s necessarily any different in the UK.

Prof. Steve Chapman: It was a very simple study, but a very telling study. They sent out CVs to people in North American institutions and the only difference in the CV was the name at the top — a male name or a female name. The contents of the CVs were identical. And when the people were asked to comment on the CVs, there was something like a 40% preference for the CV if it had a male name associated with it. Now those people I don’t think were actively trying to discriminate against women, but they were, and they were doing it subconsciously. It scared me, because of course I would go around saying, ‘I’m not prejudiced at all,’ but I read that and I thought, if I saw those CVs, would I react differently?

Dr. Janet Lovett, Royal Society University Research Fellow, University of Edinburgh: You hear the kind of results from the Yale study and unfortunately you’re not that surprised by them. And I think … I think it’s hard to explain why you’re not that surprised by them. There is an endemic sexism to most high-powered careers, I would say.

Prof. Polly Arnold: When I was a junior academic in a previous job, I was given the opportunity to go on a course to help women get promoted. The senior management at the university had looked at the data, and they’d realized that the female academics were winning lots of international prizes, being very successful internationally, but they weren’t getting promoted internally, so what we needed was a course to help us do this. And to this day, I still don’t understand how they didn’t realize that it was them that needed the course.

Dr. Elaine Murray: I think a lot of it isn’t really about legislation or regulation, it’s actually cultural change, which is more difficult to affect. And, you know, the recognition that this is part of an equality agenda, really, that we need to have that conversation which is not just about individuals, its about the experience of women in general.

Text on screen: Women without children are still 23% less likely to achieve tenure than men with children.

Prof. Anita Jones: I’m not really in favor of positive discrimination. I don’t think, as a women, I would have wanted to feel that I got a job, or a fellowship, or a grant, or whatever, because I was a woman rather than because I was very good at what I do.

Prof. Steve Chapman: I think we have to be careful. I was looking at the ratio of women in some of the things that we’re doing in my own institution, and accidentally you can heavily dominate things with males without actually thinking about it. Does that mean we have to have quotas for women? No. But does it mean we have to be pro-active in making sure we’re bringing it to the attention of women that they should be involved, and that they add value? Yes.

Dr. Elaine Murray: I was always an advocator of positive discrimination in politics, in order to address the issue of the underrepresentation of women. Now, a lot of younger women now don’t see that as important, and yet if you present them some of the issues that women face to get on, they do realize things aren’t quite as easy.

Text on screen: 2012 The School of Chemistry receives the Athena Swan Gold Award, recognising a significant progression and achievement in promoting gender equality.

Prof. Steve Chapman: We shouldn’t underestimate the signal that Athena Gold sends out. It sends out the message that this school is committed to the Athena Agenda, which isn’t actually just about women. It’s about creating an environment in which all people can thrive.

Prof. Eleanor Campbell: I think it is extremely important that the men in the department have a similar view when it comes to supporting young academics, graduate students, postdocs, regardless of their gender. I think that’s extremely important. And, I mean, certainly here, our champion for our Athena Swan activities is a male, and I deliberately wanted to have a younger male doing that job, to make it clear that it wasn’t just about women, that it was about really improving conditions for everybody.

Dr. Elaine Murray: I know, for example, in the Scottish government, equalities is somehow lumped in with health, but it’s not. You know, health is such a big portfolio that equalities is going to get pretty much lost in the end, and I think probably there’s a need for equalities issues to take a higher profile at a governmental level. And I think also it’s still about challenging the media, about the sort of stereotypes which surround women more generally, and still in science.

Text on screen: 2012 Prof. Lesley Yellowlees becomes the first female President of The Royal Society of Chemistry.

Prof. Lesley Yellowlees MBE FRSE FRSC, Professor of Inorganic Electrochemistry, Vice Principal & Head of the College of Science & Engineering, University of Edinburgh, President of The Royal Society of Chemistry: I’ve often heard myself say we have to be patient, but there comes a time when you have to run out of patience, because if we don’t run out of patience and we don’t start demanding more from the system, demanding that culture change to happen faster than it’s happening at present, then I think we not only do ourselves a disservice, but we do the generations both past and the ones to come a huge disservice as well.

Text on screen: At our current rate of progress it will take 70 years before we achieve parity between the sexes.

Prof. Polly Arnold: If we’re unwilling to define ourselves as feminists, we need to replace the word with something more palatable. The concept of equality is no less relevant today.