Tag Archives: methodology

Methodology versus beliefs: a comment from Marcus Ross.

Posted on by

Last week, we considered whether good science has more to do with what you do or with what you believe, exploring this issue using the case of Marcus Ross, a Ph.D. geoscientist and young earth creationist. Dr. Ross sent me a response to this post via email. With his permission, I’m sharing that email here:

* * * * *
Hello Janet,
 
Thank you for your thoughtful piece yesterday in Scientific American.  It has been quite a while since the New York Times piece in 2007, so I was surprised to it revisited.  And I found your analysis of the events of my Ph.D. work far more considerate than many of the earlier reactions.  It’s nice not to be referred to as a trained parrot, a textbook case of cognitive dissonance, or a variety of unprintable words!
 
This paragraph from your piece sums things up quite nicely:

“It looks like Ross saw his dissertation as an exercise in presenting the inferences one could draw from the available data using the recognized methods of geoscience. In other words, here’s what we would conclude if all the assumptions about the age of the earth, deposition of fossils, isotope dating methods, etc., were true…”

 
This is a good sketch of what I did, not only for the Ph.D., but for all of my geological education (which was conducted entirely at non-creationist, state schools; and like at URI, at each location I made it known to my advisors that I was a young-Earth creationist).  I always felt that, since I was attempting to earn a degree from an institution which adhered to an ancient Earth and evolutionary explanations of life’s diversity, that I must show myself proficient in these areas. 
 
One clarification which stems from Cornelia Dean’s original article: I never referred to a “paleontological paradigm”.  That term is one she invented from her interview of me, but one I never introduced.  Indeed, the term actually makes very little sense (does anyone speak of a microbiology paradigm?).  In speaking with my students, I refer to the old-Earth and evolutionary paradigms, and I make sure to distinguish the two as well.
 
One issue that you bring up is whether I’ve essentially given up on interaction with the geological community, especially given my position at Liberty University.  Let me assure you that such is not the case.  In both print and in annual meetings, I do what I can to contribute to, and interact with, current geological discussions.  My publication record is not extensive, but it includes papers in a handful of conventional geological journals, including recent geological papers in 2009 and 2010 and co-leading a field trip at the annual meeting of the Geological Society of America (our largest professional association) last year with four other creation geologists.  Even Steven Newton of the NCSE has written, more or less, charitably of my, and my creationist colleagues’, continuing interactions at society meetings over the past few years.
 
Nevertheless, despite my best attempts, and because of some of my old-Earth and evolutionary colleagues’ attitudes towards me, the road of interaction has been bumpy.  I have had chapters of my (decidedly conventional) dissertation rejected from journals and special publications for no other reason than the fact that I am a creationist, sometimes in very explicit terms.  Presentations at society meetings are viewed with deep suspicion that I will make creationist arguments (or even preach!) once given the lectern.  I have, on two occasions, been “outed” as a creationist following my own presentation by scientists who wished to score points with their students and peers, and do damage to my reputation.  But having been open about being a creationist my whole career usually blunts such shoddy attempts at a “gotcha” moment.  The job description for my employment was gleefully mocked at a society presentation while I was in attendance.  And this is from the more legitimate forms of scientific dialogue.  Googling my name gets really ugly, really fast.
 
But such is no major deterrent to me, though it does impede my attempts to publish in conventional literature, for example.  I value the contributions of my colleagues, and have enjoyed many constructive interactions, despite the occasional run-in with less pleasant sorts.  In my classes here at Liberty University I introduce my students to the reasons why geologist think the Earth is ancient, or why various organisms are viewed as strong evidence for evolution.  I do this so that they understand that these arguments are well thought-out, and to teach them to respect the ideas of those with whom they disagree.  And I was grateful for your blog post because, unlike many others, you respect my position enough to treat it with courtesy.  Thank you.
 
Blessings,
Marcus
 
Marcus R. Ross, Ph.D.  

Associate Professor of Geology

Dept. of Biology and Chemistry
Liberty University

Is being a good scientist a matter of what you do or of what you feel in your heart?

Posted on by

If the question posed in the title of the post seems to you to have an obvious answer, sit tight while I offer a situation in which it might be less obvious.

We recently discussed philosopher Karl Popper’s efforts to find the line of demarcation between science and pseudo-science. In that discussion, one of the things you may have noticed is that Popper’s story is as much about a distinctive scientific attitude as it is about the details of scientific methodology. I wrote:

Popper has this picture of the scientific attitude that involves taking risks: making bold claims, then gathering all the evidence you can think of that might knock them down. If they stand up to your attempts to falsify them, the claims are still in play. But, you keep that hard-headed attitude and keep you eyes open for further evidence that could falsify the claims. If you decide not to watch for such evidence — deciding, in effect, that because the claim hasn’t been falsified in however many attempts you’ve made to falsify it, it must be true — you’ve crossed the line to pseudo-science.

And, my sense from scientists is that Popper’s description of their characteristic attitude is what they like best about his account. Hardly any scientist goes into the lab Monday morning with the firm intention of trying (yet again) to falsify the central hypotheses which she and the other scientists in her field have been using successfully (to predict and to explain and to create new phenomena) for years. Hardly any scientist will toss out hypotheses on the basis of a single experimental result that does not match the predictions of the hypotheses. But scientists agree that when they’re following the better angels of their scientific nature, their eyes are open to evidence that might conflict with even their most trusted hypotheses, and they are ready to kiss those hypotheses goodbye if the facts in the world line up against them.

An attitude is something that’s in your heart.

Certainly, an attitude may exert a strong influence on what you do, but if having the right attitude is something that matters to us over and above doing the right thing, we can ask why that is. My best hunch is that an attitude may act as a robust driver of behavior — in other words, having the right attitude may be a reliable mechanism that gets you to do the right thing, at least more than you might in the absence of that attitude.

So, what should we say about a scientist who appears to practice the methodology as he should, but who reveals himself as having something else in his heart?

This question came up back in 2007, when the New York Times reported on the curious case of Marcus R. Ross. Ross had written and defended an “impeccable” dissertation on the abundance and spread of marine reptiles called mosasaurs which (as his dissertation noted) vanished about 65 million years ago, earning a Ph.D. in geosciences from the University of Rhode Island. Then, he accepted a faculty position at Liberty University, where he is an Assistant Director of the Center for Creation Studies.

Ross is a young earth creationist, and as such believes that the earth is no older than 10,000 years. He was a young earth creationist when he wrote the impeccable dissertation in which he noted the disappearance of mosasaurs about 65 millions years ago. Indeed, he was a young earth creationist when he applied to the geosciences Ph.D. program at the University of Rhode Island, and did not conceal this information from the admissions committee.

Some details from the New York Times article:

For him, Dr. Ross said, the methods and theories of paleontology are one “paradigm” for studying the past, and Scripture is another. In the paleontological paradigm, he said, the dates in his dissertation are entirely appropriate. The fact that as a young earth creationist he has a different view just means, he said, “that I am separating the different paradigms.”

He likened his situation to that of a socialist studying economics in a department with a supply-side bent. “People hold all sorts of opinions different from the department in which they graduate,” he said. “What’s that to anybody else?” …

In theory, scientists look to nature for answers to questions about nature, and test those answers with experiment and observation. For Biblical literalists, Scripture is the final authority. As a creationist raised in an evangelical household and a paleontologist who said he was “just captivated” as a child by dinosaurs and fossils, Dr. Ross embodies conflicts between these two approaches. The conflicts arise often these days, particularly as people debate the teaching of evolution. …

In a telephone interview, Dr. Ross said his goal in studying at secular institutions “was to acquire the training that would make me a good paleontologist, regardless of which paradigm I was using.” …

He would not say whether he shared the view of some young earth creationists that flaws in paleontological dating techniques erroneously suggest that the fossils are far older than they really are.

Asked whether it was intellectually honest to write a dissertation so at odds with his religious views, he said: “I was working within a particular paradigm of earth history. I accepted that philosophy of science for the purpose of working with the people” at Rhode Island.

And though his dissertation repeatedly described events as occurring tens of millions of years ago, Dr. Ross added, “I did not imply or deny any endorsement of the dates.”

Ross pursued an education that gave him detailed knowledge of the theories the geoscience community uses, the questions geoscientists take to be interesting ones to pursue, the methods they use to make observations, to analyze data, and to draw inferences. He showed sufficient mastery of the “paleontological paradigm” that he was able to use it to build an additional piece of knowledge (the work contained in his dissertation) that was judged a contribution to his scientific community.

But, if he believed in his heart that the earth was thousands, not millions, of years old as he built this piece of knowledge, was he really a part of that scientific community? Was he essentially lying in his interactions with its members?

It looks like Ross saw his dissertation as an exercise in presenting the inferences one could draw from the available data using the recognized methods of geoscience. In other words, here’s what we would conclude if all the assumptions about the age of the earth, deposition of fossils, isotope dating methods, etc., were true. His caginess about the dates in the interview quoted above, and his professed belief in young earth creationism, suggest that Ross thinks at least some of these scientific assumptions are false.

However, assuming his rejection of the scientific assumptions flows primarily from his commitments as a young earth creationist, the rejection of the claims other geoscientists agree on is based in religious reasons, not scientific reasons. If there were scientific reasons to doubt these assumptions, it seems like examining those could only lead to a stronger body of knowledge in geosciences, and that Ross could have contributed to the field by making such an examination the focus of his doctoral research.

Is it an obligation for a scientist who has concerns about the goodness of an assumption on which people in his field rest their inferences to voice that concern? Is it an obligation for that scientist to gather data to test that hypothesis, or to work out an alternative hypothesis that is better supported by the data? Or is it OK to keep your doubts to your self and just use the inferential machinery everyone else is using?

Maybe people will answer this differently if the scientist in question is planning an ongoing engagement with the other members of this field, or if he is just passing through on the way to somewhere else. More on this in just a moment.

Here’s a shorter version of my question about the scientist’s obligations here: Does intellectual honesty in scientific knowledge-building just cover the way you use the inferential structure and the inputs (i.e., data) from which you draw your inferences? Or does it require disclosure of which assumptions you really accept (not just for the sake of argument, but in your heart of hearts) when drawing your inferences and which you are inclined to think are mistaken?

Does intellectual honesty require that you disclose as well the fact that you don’t actually accept the inferential structure of science as a good way to build knowledge?

Because ultimately, a commitment to young earth creationism seems to be a commitment that the data cannot properly be used to infer any claims that are at odds with scripture.

And here’s where scientists who might be willing to accept Ross’s dissertation as a legitimate chunk of scientific knowledge may have serious concerns with Ross as a credible member of the scientific community. The dissertation may stand (or fall) as a scientific argument that presents a particular array of data, describes accepted inferential strategies (perhaps even defending some such strategies that are themselves new contributions), and uses these strategies to draw conclusions form the data. Even if the person who assembled this argument was wracked with doubts about all the central premises of the argument, the argument itself could still function perfectly well in the ongoing scientific discourse, and other scientists in the community could judge that argument on its strengths and weaknesses — not on what might be in the heart of the person who constructed the argument.

But, if, ultimately, Marcus Ross rejects the “paleontological paradigm” — and the possibility that the data could properly support inferences at odds with scripture — can he function as a member of a community that makes, and evaluates, inferences using this paradigm?

Maybe he could, but his career trajectory makes it look like he has chosen not to be a member of the larger community of geoscientists. Instead, he has positioned himself as a member of a community of “creation scientists”. Whether Ross’s ongoing work on extinct marine reptiles is of any scientific interest to the scientific field that trained him will probably depend on the methodology and inferential structure on display in his manuscripts.

Because methodology and inferential structure are much easier to evaluate in the peer review process than what is in the author’s heart.

* * * * *

If you enjoyed this post, consider contributing a few bucks to a project in my Giving Page in the Science Bloggers for Students 2011 challenge. Supporting science education in public school classrooms will help young people get a better handle on what kind of attitude and methodology makes science science — and on all the cool things science can show us about our world.

Drawing the line between science and pseudo-science.

Posted on by

Recently, we’ve been discussing strategies for distinguishing sound science from attractively packaged snake-oil. It’s worth noting that a fair number of scientists (and of non-scientists who are reasonably science-literate) are of the view that this is not a hard call to make — that astrology, alternative therapies, ESP, and the other usual suspects fall on the wrong side of some bright line that divides what is scientific from what is not — the clear line of demarcation that (scientists seem to assume) Karl Popper pointed out years ago, and that keeps the borders of science secure.


While I think a fair amount of non-science is so far from the presumptive border that we are well within our rights to just point at it and laugh, as a philosopher of science I need to go on the record as saying that right at the boundary, things are not so sharp. But before we get into how real science (and real non-science) might depart from Sir Karl’s image of things, I think it’s important to look more closely at the distinction he’s trying to draw.


A central part of Karl Popper’s project is figuring out how to draw the line between science and pseudo-science. He could have pitched this as figuring out how to draw the line between science and non-science (which seems like less a term of abuse than “pseudo-science”). Why set the project up this way? Partly, I think, he wanted to compare science to non-science-that-looks-a-lot-like-science (in other words, pseudo-science) so that he could work out precisely what is missing from the latter. He doesn’t think we should dismiss pseudo-science as utterly useless, uninteresting, or false. It’s just not science.

Of course, Popper wouldn’t be going to the trouble of trying to spell out what separates science from non-science if he didn’t think there was something special on the science side of the line. He seems committed to the idea that scientific methodology is well-suited — perhaps uniquely so — for building reliable knowledge and for avoiding false beliefs. Indeed, under the assumption that science has this kind of power, one of the problems with pseudo-science is that it gets an unfair credibility boost by so cleverly mimicking the surface appearance of science.


The big difference Popper identifies between science and pseudo-science is a difference in attitude. While a pseudo-science is set up to look for evidence that supports its claims, Popper says, a science is set up to challenge its claims and look for evidence that might prove it false. In other words, pseudo-science seeks confirmations and science seeks falsifications.


There is a corresponding difference that Popper sees in the form of the claims made by sciences and pseudo-sciences: Scientific claims are falsifiable — that is, they are claims where you could set out what observable outcomes would be impossible if the claim were true — while pseudo-scientific claims fit with any imaginable set of observable outcomes. What this means is that you could do a test that shows a scientific claim to be false, but no conceivable test could show a pseudo-scientific claim to be false. Sciences are testable, pseudo-sciences are not.


So, Popper has this picture of the scientific attitude that involves taking risks: making bold claims, then gathering all the evidence you can think of that might knock them down. If they stand up to your attempts to falsify them, the claims are still in play. But, you keep that hard-headed attitude and keep you eyes open for further evidence that could falsify the claims. If you decide not to watch for such evidence — deciding, in effect, that because the claim hasn’t been falsified in however many attempts you’ve made to falsify it, it must be true — you’ve crossed the line to pseudo-science.


This sets up the central asymmetry in Popper’s picture of what we can know. We can find evidence to establish with certainty that a claim is false. However, we can never (owing to the problem of induction) find evidence to establish with certainty that a claim is true. So the scientist realizes that her best hypotheses and theories are always tentative — some piece of future evidence could conceivably show them false — while the pseudo-scientist is sure as sure as can be that her theories have been proven true. (Of course, they haven’t been — problem of induction again.)


So, why does this difference between science and pseudo-science matter? As Popper notes, the difference is not a matter of scientific theories always being true and pseudo-scientific theories always being false. The important difference seems to be in which approach gives better logical justification for knowledge claims. A pseudo-science may make you feel like you’ve got a good picture of how the world works, but you could well be wrong about it. If a scientific picture of the world is wrong, that hard-headed scientific attitude means the chances are good that we’ll find out we’re wrong — one of those tests of our hypotheses will turn up the data that falsifies them — and switch to a different picture.

A few details are important to watch here. The first is the distinction between a claim that is falsifiable and a claim that has been falsified. Popper says that scientific claims are falsifiable and pseudo-scientific claims are not. A claim that has been falsified (demonstrated to be false) is obviously a falsifiable claim (because, by golly, it’s been falsified). Once a claim has been falsified, Popper says the right thing to do is let it go and move on to a different falsifiable claim. However, it’s not that the claim shouldn’t have been a part of science in the first place.
So, the claim that the planets travel in circular orbits wasn’t an inherently unscientific claim. Indeed, because it could be falsified by observations, it is just the kind of claim scientists should work with. But, once the observations show that this claim is false, scientists retire it and replace it with a different falsifiable claim.


This detail is important! Popper isn’t saying that science never makes false claims! What he’s saying is that the scientific attitude is aimed at locating and removing the false claims — something that doesn’t happen in pseudo-sciences.


Another note on “falsifiability” — the fact that many attempts to falsify a claim have failed does not mean that the claim is unfalsifiable. Nor, for that matter, would the fact that the claim is true make it unfalsifiable. A claim is falsifiable if there are certain observations we could make that would tell us the claim is false — certain observable ways the world could not be if the claim were true. So, the claim that Mars moves in an elliptical orbit around the sun could be falsified by observations of Mars moving in an orbit that deviated at all from an elliptical shape.


Another important detail is just what scientists mean by “theory”. A theory is simply a scientific account (or description, or story) about a system or a piece of the world. Typically, a theory will contain a number of hypotheses about what kind of entities are part of the system and how those entities behave. (The hypothesized behaviors are sometimes described as the “laws” governing the system.) The important thing to note is that theories can be rather speculative or extremely well tested — either way, they’re still theories.


Some people talk as though there’s a certain threshold a theory crosses to become a fact, or truth, or something more-certain-than-a-theory. This is a misleading way of talking. Unless Popper is completely wrong that the scientist’s acceptance of a theory is always tentative (and this is one piece of Popper’s account that most scientists whole-heartedly endorse), then even the theory with the best evidential support is still a theory. Indeed, even if a theory happened to be completely true, it would still be a theory! (Why? You could never be absolutely certain that some future observation might not falsify the theory. In other words, on the basis of the evidence, you can’t be 100% sure that the theory is true.)


So, for example, dismissing Darwin’s theory as “just a theory” as if that were a strike against it is misunderstanding what science is up to. Of course there is some uncertainty; there is with all scientific theories. Of course there are certain claims the theory makes that might turn out to be false; but the fact that there is evidence we could conceivably get to demonstrate these claims are false is a scientific virtue, not a sign that the theory is unscientific.


By contrast, “Creation Science” and “Intelligent Design Theory” don’t make falsifiable claims (at least, this is what many people think; Larry Laudan* disputes this but points out different reasons these theories don’t count as scientific). There’s no conceivable evidence we could locate that could demonstrate the claims of these theories are false. Thus, these theories just aren’t scientific. Certainly, their proponents point to all sorts of evidence that fits well with these theories, but they never make any serious efforts to look for evidence that could prove the theories false. Their acceptance of these theories isn’t a matter of having proof that the theories are true, or even a matter of these theories having successfully withstood many serious attempts to falsify them. Rather, it’s a matter of faith.


None of this means Darwin’s theory is necessarily true and “Creation Science” is necessarily false. But it does mean (in the Popperian view that most scientists endorse) that Darwin’s theory is scientific and “Creation Science” is not.


______

*See Laudan, “Science at the Bar — Causes for Concern”, in Robert T. Pennock and Michael Ruse, But Is It Science?

* * * * *

If you enjoyed this post, consider contributing a few bucks to a project in my Giving Page in the Science Bloggers for Students 2011 challenge. Supporting science education in public school classrooms will help young people get a better handle on what kind of attitude and methodology makes science science — and on all the cool things science can show us about our world.

Evaluating scientific claims (or, do we have to take the scientist’s word for it?)

Posted on by

Recently, we’ve noted that a public composed mostly of non-scientists may find itself asked to trust scientists, in large part because members of that public are not usually in a position to make all their own scientific knowledge. This is not a problem unique to non-scientists, though — once scientists reach the end of the tether of their expertise, they end up having to approach the knowledge claims of scientists in other fields with some mixture of trust and skepticism. (It’s reasonable to ask what the right mixture of trust and skepticism would be in particular circumstances, but there’s not a handy formula with which to calculate this.)

Are we in a position where, outside our own narrow area of expertise, we either have to commit to agnosticism or take someone else’s word for things? If we’re not able to directly evaluate the data, does that mean we have no good way to evaluate the credibility of the scientist pointing to the data to make a claim?

This raises an interesting question for science journalism, not so much about what role it should play as what role it could play.

If only a trained scientist could evaluate the credibility of scientific claims (and then perhaps only in the particular scientific field in which one was trained), this might reduce science journalism to a mere matter of publishing press releases, or of reporting on scientists’ social events, sense of style, and the like. Alternatively, if the public looked to science journalists not just to communicate the knowledge claims various scientists are putting forward but also to do some evaluative work on our behalf — sorting out credible claims and credible scientists from the crowd — we might imagine that good science journalism demands extensive scientific training (and that we probably need a separate science reporter for each specialized area of science to be covered).

In an era where media outlets are more likely to cut the science desk than expand it, pinning our hopes on legions of science-Ph.D.-earning reporters on the science beat might be a bad idea.

I don’t think our prospects for evaluating scientific credibility are quite that bad.

Scientific knowledge is built on empirical data, and the details of the data (what sort of data is relevant to the question at hand, what kind of data can we actually collect, what techniques are better or worse for collecting the data, how we distinguish data from noise, etc.) can vary quite a lot in different scientific disciplines, and in different areas of research within those disciplines. However, there are commonalities in the basic patterns of reasoning that scientists in all fields use to compare their theories with their data. Some of these patterns of reasoning may be rather sophisticated, perhaps even non-intuitive. (I’m guessing certain kinds of probabilistic or statistical reasoning might fit this category.) But others will be the patterns of reasoning that get highlighted when “the scientific method” is taught.

In other words, even if I can’t evaluate someone else’s raw data to tell you directly what it means, I can evaluate the way that data is used to support or refute claims. I can recognize logical fallacies and distinguish them from instances of valid reasoning. Moreover, this is the kind of thing that a non-scientist who is good at critical thinking (whether a journalist or a member of the public consuming a news story) could evaluate as well.

One way to judge scientific credibility (or lack thereof) is to scope out the logical structure of the arguments a scientist is putting up for consideration. It is possible to judge whether arguments have the right kind of relationship to the empirical data without wallowing in that data oneself. Credible scientists can lay out:

  • Here’s my hypothesis.
  • Here’s what you’d expect to observe if the hypothesis is true. Here, on the other hand, is what you’d expect to observe if the hypothesis is false.
  • Here’s what we actually observed (and here are the steps we took to control the other variables).
  • Here’s what we can say (and with what degree of certainty) about the hypothesis in the light of these results.
  • Here’s the next study we’d like to do to be even more sure.

And, not only will the logical connections between the data and what is inferred from them look plausible to the science writer who is hip to the scientific method, but they ought to look plausible to other scientists — even to scientists who might prefer different hypotheses, or different experimental approaches. If what makes something good science is its epistemology — the process by which data are used to generate and/or support knowledge claims — then even scientists who may disagree with those knowledge claims should still be able to recognize the patterns of reasoning involved as properly scientific. This suggests a couple more things we might ask credible scientists to display:

  • Here are the results of which we’re aware (published and unpublished) that might undermine our findings.
  • Here’s how we have taken their criticisms (or implied criticisms) seriously in evaluating our own results.

If the patterns of reasoning are properly scientific, why wouldn’t all the scientists agree about the knowledge claims themselves? Perhaps they’re taking different sets of data into account, or they disagree about certain of the assumptions made in framing the question. The important thing to notice here is that scientists can disagree with each other about experimental results and scientific conclusions without thinking that the other guy is a bad scientist. The hope is that, in the fullness of time, more data and dialogue will resolve the disagreements. But good, smart, honest scientists can disagree.

This is not to say that there aren’t folks in lab coats whose thinking is sloppy. Indeed, catching sloppy thinking is the kind of thing you’d hope a good general understanding of science would help someone (like a scientific colleague, or a science journalist) to do. At that point, of course, it’s good to have backup — other scientists who can give you their read on the pattern of reasoning, for example. And, to the extent that a scientist — especially one talking “on the record” about the science (whether to a reporter or to other scientists or to scientifically literate members of the public) — displays sloppy thinking, that would tend to undermine his or her credibility.

There are other kinds of evaluation you can probably make of a scientist’s credibility without being an expert in his or her field. Examining a scientific paper to see if the sources cited make the claims that they are purported to make by the paper citing them is one way to assess credibility. Determining whether a scientist might be biased by an employer or a funding source may be harder. But there, I suspect many of the scientists themselves are aware of these concerns and will go the extra mile to establish their credibility by taking the possibility that they are seeing what they want to see very seriously and testing their hypotheses fairly stringently so they can answer possible objections.

It’s harder still to get a good read on the credibility of scientists who present evidence and interpretations with the right sort of logical structure but who have, in fact, fabricated or falsified that evidence. Being wary of results that seem too good to be true is probably a good strategy here. Also, once a scientist is caught in such misconduct, it’s entirely appropriate not to trust another word that comes from his or her mouth.

One of the things fans of science have tended to like is that it’s a route to knowledge that is, at least potentially, open to any of us. It draws on empirical data we can get at through our senses and on our powers of rational thinking. As it happens, the empirical data have gotten pretty complicated, and there’s usually a good bit of technology between the thing in the world we’re trying to observe and the sense organs we’re using to observe it. However, those powers of rational thinking are still at the center of how the scientific knowledge gets built. Those powers need careful cultivation, but to at least a first approximation they may be enough to help us tell the people doing good science from the cranks.

Scientific credibility: is it who you are, or how you do it?

Posted on by

Part of the appeal of science is that it’s a methodical quest for a reliable picture of how our world works. Creativity and insight is crucial at various junctures in this quest, but careful work and clear reasoning does much of the heavy lifting. Among other things, this means that the grade-schooler’s ambition to be a scientist someday is significantly more attainable than the ambition to be a Grammy-winning recording artist, a pro-athlete, an astronaut, or the President of the United States.

Scientific methodology, rather than being a closely guarded trade secret, is a freely available resource.

Because of this, there is a sense that it doesn’t matter too much who is using that scientific methodology. Rather, what matters is what scientists discover by way of the methodology.
Continue reading→

Objectivity requires teamwork, but teamwork is hard.

Posted on by

In my last post, I set out to explain why the scientific quest to build something approaching objective knowledge requires help from other people. However, teamwork can be a challenge in the best of circumstances. And, certain aspects of scientific practices — especially in terms of how rewards are distributed — can make scientific teamwork even harder.

In this post, I’ll run down just some of the obstacles to scientists playing together effectively to build reliable knowledge about the world.

First, recall that a crucial thing individual scientists hope to get from their teammates in knowledge-building is help in identifying when they are wrong. The sociologist of science Robert Merton noted that a rule of the knowledge-building game, at least as far as Team Science is concerned, is organized skepticism, which I once described like this:

Everyone in the tribe of science can advance knowledge claims, but every such claims that is advanced is scrutinized, tested, tortured to see if it really holds up. The claims that do survive the skeptical scrutiny of the tribe get to take their place in the shard body of scientific knowledge.

In principle, each scientist tries to have their organized skepticism turned up to a healthy level when looking at her own results, as well as the results of others. In practice, there are issues that get in the way of both self-scrutiny and scrutiny of the results of others.

It’s hard to make a scientific career in replicating the results of others.

The first thing to recognize is that as serious as scientists are about the ideal of reproducible results, reproducibility is hard. It takes a while to gain technical mastery of all the moving parts in your experimental system and to figure out which of those wiggly bits make a difference in the results you see.

In itself, this needn’t be an obstacle to scientists working well together. The problem is that scientific rewards are usually reserved for those who generate novel findings — figuring out something that wasn’t known before — rather than for those who replicate results someone else has already put forward. What matters, for the career score-keeping (which drives who gets hired, who gets grant money, who gets promoted, who wins prizes) is whether you are first across the finish line to discover X. Being second (or third, or tenth) across that finish line is a nice reassurance that the first one across had a solid finding, but it doesn’t count in the same way.

Setting up the rewards so that the only winner is the first across the finish line may also provide a disincentive to doing enough experiments yourself to be sure that your results are really robust — the other guy may be sure enough to submit his manuscript on the basis of fewer runs, or might have gotten a head-start on it.

Now surely there are some exceptions, places perhaps where X was such a startlingly unexpected finding that the scientific community won’t really believe it until multiple researchers come forward to report that they have found it. But this is the exception rather than the rule, which means that if the second scientist to have found X cannot add some additional ingredient to our understanding of it that wasn’t part of the first report of X, that second researcher is out of luck.

Scientists are generally pretty smart. Among other things, this means most of them will come up with some strategy for spending their time that takes account of what activities will be rewarded. To the extent that working to replicate someone else’s results looks like a high-investment, low-yield activity, scientists may judge it prudent to spend their time doing something else.

It’s worth noting that scientists will frequently try to reproduce the results of others when those results are the starting point for a brand new piece of research of their own. These efforts can be time consuming and frustrating (see: “reproducibility is hard”). And, in the event that you discover that the other scientist’s results seem not to hold up, communicating to this other scientist is not always viewed as a friendly gesture.

Questions about results can feel like personal attacks.

Scientists work hard to get their studies to work and to draw their best conclusions about what their observations mean — and, as we’ve just noted, they do this while racing against the clock in hopes that some other researcher doesn’t make the discovery (and secure the credit for it) first. Since scientists are human, they can get attached to those results they worked so hard to get.

It shouldn’t be a surprise, then, that they can get touchy when someone else pops into the fray to tell them that there’s a problem with those results.

If the results are wrong, scientists face the possibility that they have wasted a bunch of time and money, blood, sweat, and tears. As well, they may have to issue a correction or even a retraction of their published results, which means that the publication that they’re correcting or retracting will no longer do the same work to advance their career.

In such a situation, getting defensive is understandable. However, getting defensive doesn’t do much to advance the knowledge-building project that science is supposed to be.

None of this is to say that an objection raised to one’s results should be automatically accepted as true. Organized skepticism applies to the critiques as well as to the original results.

That said, though, it strikes me that the best way to the knowledge-building, error-detecting teamwork that the tribe of science could use here might be establishing environments in scientific communities (from research groups to departments to disciplines) where researchers don’t take scrutiny of their results, data, methods, etc., personally — and where the scrutiny is applied to each member’s results, data, methods, etc. (since anyone can make mistakes).

When the players understand the game as aimed at building a reliable body of knowledge about the world that they can share, maybe they can be more welcoming of others pointing out their errors. When the game is understood as each scientist against all the others, pulling back to look critically at problems with one’s own work (especially when they are pointed out by a competitor) doesn’t look like such a great strategy.

(Unwillingness to take critiques of promising results seriously seems to have been a major feature of the Bengü Sezen/Dalibor Sames fraud scandal, and may also have played a role in the downfall of Harvard psychologist Marc Hauser.)

Sharing too much common ground makes it harder to be objective, too.

Part of the reason that scientists try to be alert to ways they could be deceived, even by themselves, is that the opportunities for deception are plentiful. One of the issues you have to face is that your expectations can influence what you see.

We don’t even have to dig into Thomas S. Kuhn’sThe Structure of Scientific Revolutions, embrace his whole story about paradigms, or even peruse the perception experiments he describes to accept that this is a possibility. The potential effect of expectations on observations is one reason that placebo-controlled trials are “double-blind” whenever possible, so neither experimental subject nor researcher is swayed by what they think ought to be occurring. Expectations also play a role in what kind of scientific findings are accepted easily into the shared body of knowledge (because they seem to fit so naturally with what we already know) and which ones are resisted (because they don’t fit so well, and might even require us to identify some of the things we thought we “knew” as wrong). And, expectations can influence scientific knowledge by shaping what kinds of questions researchers ask, what kinds of methods they decide to use to tackle those questions, and what kinds of outcomes they view as within the realm of possible outcomes. (If you’re doing an experiment and you see an outcome outside the range of expectations, often the first thing you check is whether the equipment is malfunctioning.)

Working with other people helps scientists build better knowledge by giving them some information about which observed outcomes are driven by the features of the phenomenon they’re trying to understand and which are driven by subjective features (like expectations). But the other people are must helpful here if their expectations and background assumptions are not identical to our own!

In the case that a scientific community shares all the same background assumptions, expectations, even unconscious biases, these things — and the ways that they can influence how experiments are designed and what findings they produce — may become almost invisible to the scientists in the community.

What this means is that it may be a healthy thing for a community of knowledge-builders to be diverse. How diverse? Ideally, you’d want the community to achieve enough diversity that it’s hard for each individual’s background assumptions to stay in the background, because you’re always in spitting distance of another individual with different background assumptions. Recognizing them as assumptions rather than necessarily true facts about the world can keep potential errors or oversights due to these assumptions on a shorter leash.

Each of these obstacles is linked to an over-arching challenge for scientific teamwork:

Teamwork often isn’t recognized or rewarded in scientific career score-keeping.

Scientists work with each other a lot to divide up aspects of complex research projects, but when it comes time to tally up the score what sometimes seems to matter most is who ends up being first author and who’s lost in the et al. Scientists have detailed discussions about published research in their field, enacting something like post-publication peer review, but if you determine that the other guy’s argument is persuasive or that his published findings actually hold up, it can be hard to capture the contribution you’ve made to Team Science’s joint knowledge-building project on your Curriculum Vitae. Scientists even have detailed pre-publication engagement about results (sometimes as a journal submission is being peer reviewed, sometimes less formally), but helping someone else uncover her mistakes or biases may put her in position to cross the discovery finish line before you do — and again, one doesn’t get much tangible career reward for providing this assist.

Teamwork is essential to making the knowledge scientists produce more objective. Yet the big career rewards in science seem clearly tied to individual achievement. Maybe the assumption is that some combination of competitive impulse (i.e., wanting to knock down other scientists’ putative knowledge claims) and community goodwill is enough to get scientists working together to find the errors and weed them out.

But maybe, if we think objectivity is a quality towards which scientific knowledge should be striving, it would make sense to put some more concrete rewards in place to incentivize the scientific teamwork on which objectivity depends.

The objectivity thing (or, why science is a team sport).

Posted on by

One of the qualities we expect from good science is objectivity. And, we’re pretty sure that the scientific method (whatever that is) has something to do with delivering scientific knowledge that is objective (or more objective than it would be otherwise, at any rate).

In this post, I’m here to tell you that it’s more complicated than that — at least, if you’re operating with the picture of the scientific method you were taught in middle school. What we’ll see is that objectivity requires more than a method; it takes a team.

(I’ll briefly note that my discussion of objectivity, subjectivity, and scientific knowledge building owes much to Helen E. Longino’s book, Science as Social Knowledge. If you want to get into the epistemic complexities of this issue, you may want to read through the comments on this old but related post on my other blog.)

But let’s start at the beginning. What do we mean by objectivity?

It may be useful to start with the contrast to objective: subjective. If I put forward the claim “Friday Night Lights is the best television series ever!” you may agree or disagree. However, you might also point out that this looks like the kind of claim where it seems wrong to assert there’s a definite truth value (true or false). Why? Because it seems unlikely that there’s a fact of the matter “in the world” about what is the best television series ever — that is, a fact outside my head, or your head, or someone else’s head.

“Friday Night Lights is the best television series ever!” is a subjective claim. It isn’t pointing to a fact in the world, but rather to a fact about my experience of the world. There is no reason to think your experience of the world will be the same as mine here; it’s a matter of opinion what the best TV show is.

Of course, if we want to be more precise, we can note that facts about my (subjective) experience of the world are themselves facts in the world (since I’m in the world while I’m having the experience). However, these are not facts in the world that you could verify independently. This means if you want to know how the world seems to me, you’ll have to take my word for it. Moreover, social scientists and opinion pollsters (among others) work very hard to nail down an objective picture of a population’s subjective experience, trying to quantify opinions about TV shows or political candidates or new flavors of potato chips.

Generally speaking, though, we look to science to deliver something other than mere opinions. What we hope science will find for us is a set of facts about the world outside our heads. This brings us to one sense of the word objective: what the world is really like (as opposed to merely how it seems to me).

Another sense of objective heightens the contrast with the subjective: what anyone could discover to be so. We’re looking for facts that other people could discover as well, and trying to make claims whose truth other people could verify independently. That discovery and verification is generally taken to be conducted by way of some sense organ or another, so we probably need to modify this sense of objective to “what anyone with reasonably well-functioning sense organs could discover to be so”.

There’s a connection between these two senses of “objective” that captures some of the appeal of science as a route to knowledge.

One of the big ideas behind science is that careful observation of our world can bring us to knowledge about that world. This may seem really obvious, but it wasn’t always so. Prior to the Renaissance, recognized routes to knowledge were few and far between: what was in sacred texts, or revealed by the deity (to the select few to whom the deity was revealing truths), or what was part of the stock of practical knowledge passed on by guilds (but only to other members of these guilds). If you couldn’t get your hands on the sacred texts (and read them yourself), or have a revelation, or become a part of a guild, you had to depend on others for your knowledge.

The recognition that anyone with a reasonably well-functioning set of sense organs and with the capacity to reason could discover truths about the world — cutting out the knowledge middleman, as it were — was a radical, democratizing move. (You can find a lovely historical discussion of this shift in an essay by Peter Machamer, “The Concept of the Individual and the Idea(l) of Method in Seventeenth-Century Natural Philosophy,” in the book Scientific Controversies: Philosophical and Historical Perspectives.)

But, in pointing your sense organs and your powers of reason at the world in order to know that world, there’s still the problem of separating how things actually are from how things seem to you. You want to be able to tell which parts of your experience are merely your subjective impression of things and which parts of your experience reflect the structure of the world you are experiencing.

Can the scientific method help us with this?

Again, this depends on what you mean by the scientific method. Here’s a fairly typical presentation of “the scientific method”, found on the ScienceBuddies website:

The steps of the scientific method are to:

  • Ask a Question
  • Do Background Research
  • Construct a Hypothesis
  • Test Your Hypothesis by Doing an Experiment
  • Analyze Your Data and Draw a Conclusion
  • Communicate Your Results

Except for the very last bullet point (which suggests a someone to whom you communicate your results), this list of steps makes it look like you could do science — and build a new piece of knowledge — all by yourself. You decide (as you’re formulating your question) which piece of the world you want to understand better, come up with a hunch (your hypothesis), figure out a strategy for getting empirical evidence from the world that bears on that hypothesis (and, one hopes, that would help you discover whether that hypothesis is wrong), implement that strategy (with observations or experiments), and know more than you did before.

But, as useful as this set of steps may be, it’s important to remember that the scientific method isn’t an automatic procedure. The scientific method is not a knowledge-making box where you feed in data and collect reliable conclusions from the output bin.

More to the point, it’s not a procedure you can use all by yourself to make objective knowledge. The procedure is a good first step, but if you’re building objective knowledge you need other people.

Here’s the thing: we find out the difference between objective facts and subjective impressions of the world by actually sharing a world with other people whose subjective impressions about the world differ from our own. (Given the opacity of what’s in our minds, there also needs to be some kind of communication between us and these people with whom we’re sharing the world.) We discover that some things don’t seem the same to all of us: Not everyone likes Friday Night Lights. Not everyone finds knock-knock jokes hilarious. Not everyone hates the flavor of asparagus. Not everyone finds a ’66 Mustang beautiful.

But, if you had the world all to yourself, how would you be able to tell which parts of your experience of the world were objective and which were subjective? How, in other words, would you be able to distinguish the parts of your experience that were more reflective of actual features of the world you were experiencing from the parts of your experience that were more reflective of you as the experiencer?

It’s not clear to me that you could.

If you had the world to yourself, maybe making this distinction just wouldn’t matter. By definition, your experience would be universal. (Still, it might be helpful to be able to figure out whether some bits of your experience were more reliable in identifying real features of the world that mattered for your well being — judging “This fire feels great!” as you were sitting down on the blaze wouldn’t elicit an opposing view, but it might present problems for the continued functioning of your body.)

Our confidence that our experiences are tracking features of the world outside our head depends on our interaction with other people. And let’s be clear that we don’t just need other people to help us identify squishy “value judgments” about what feels good, tastes bad, is the best album, etc. Those senses we use to get knowledge about the world can deceive us, and the sensory information they deliver can be influenced by expectations and by past experiences. However, if we can compare notes with someone else, pointing her sense organs at the same piece of the world at which we’re pointing ours, we have a better chance of working out which parts of that experience are forced by features of the world bumping against human sense organs (i.e., the parts of our experiences of the world where there’s agreement) and which are due to the squishy subjective stuff (i.e., the parts of our experiences of the world where there’s a lot of disagreement).

Comparing notes with more people should get us closer to working out “what anyone could see (or smell, or taste, or hear, or feel)” in a particular domain of the world. Finding the common ground among people whose subjective experiences vary greatly doesn’t guarantee that what we agree about gives us the true facts about how the world really is, but it surely gets us a lot closer than any of us could get all by ourselves.

It’s worth noting that even if the textbook bulleted list version of the scientific method makes it look like you could go it alone, real scientific practice builds in the teamwork that makes the resulting knowledge more objective.

One place you can see this is in the ideal of reproducible experiments. If you’re to be able to claim that a particular experimental set-up produces a particular observable outcome (where you’ll probably also want to provide an explanation for why this is so), you first want to nail down that this set-up produces that outcome more than once. More than this, you’ll want to establish that this set-up produces that outcome no matter who conducts the experiment, and whether she conducts the experiment in this lab or some other lab. Without some kind of check that the results are “robust” (i.e., that they can be reproduced following the same procedure), there’s always the worry that the exciting results you’re seeing might be the result of an equipment malfunction, or a mislabeled chemical reagent — or even of your eyes deceiving you. But if others can follow the same procedures and produce the same results, the odds are better that the results are coming from the piece of the world you think they are.

Peer review, whether of the formal pre-publication sort or the less formal post-publication conversations scientific communities have, is another element of scientific practice that depends on teamwork. Here’s how I described peer review in a post of yore:

It’s worth noting that “peer review” can encompass different things.

Peer review describes the formal process through which manuscripts that have been submitted to journal editors are then sent to reviewers with relevant expertise for their evaluation. These reviewers then reply to the journal editors with their evaluation of the manuscript — whether it should be accepted, resubmitted after revision, or rejected — and their comments on particular aspects of the manuscript (this conclusion would be more solid if it were supported by this kind of analysis of the data, that data looks more equivocal than the authors seem to think it is, this part of the materials and methods is confusingly written, the introduction could be much more concise, etc., etc.). The editor passes on the feedback to the author, the author responds to that feedback (either by making changes in the manuscript or by presenting the editor with a persuasive argument that what a reviewer is asking for is off base or unreasonable), and eventually the parties end up with a version of the paper deemed good enough for publication (or the author gives up, or tries to get a more favorable hearing from another journal).

This flavor of peer review is very much focused on making sure that papers published in scientific journals meet a certain standard of quality or acceptability to the other scientists who will be reading those papers. There’s a lot of room for disagreement about what sort of quality is produced here, about how conservative reviewers can be when faced with new ideas or approaches, about how often reviewer judgments can be overturned by the judgment of editors (and whether that is on balance a good thing or a bad thing). As we’ve discussed before, the quality control here does not typically include reviewers actually trying to replicate the experiments described in the manuscripts they are reviewing.

Still, there’s something about peer review that a great many scientists think is important, at least when they want to be able to consult the literature in their discipline. If you want to see how your results fit with the results that others are reporting in similar lines of research, or if you’re looking for promising instrumental or theoretical approaches to a tenacious scientific puzzle, it’s good to have some reason to trust what’s reported in the literature. Otherwise, you have to do all the verification yourself.

And this is where a sort of peer review becomes important to the essence of science…

The scientist, looking at the world and trying to figure out some bit of it, is engaged in theorizing and observing, in developing hunches and then testing those hunches. The scientist wants to end up with a clearer understanding of how that bit of the world is behaving, and of what could explain that behavior.

And ultimately, the scientist relies on others to get that clearer understanding.
To really trust our observations, they need to be observations that others could make as well. To really buy our own explanations for what we observe, we need to be ready to put those explanations out for the inspection of others who might find some flaw in them, some untested assumption that doesn’t hold up to close scrutiny.

Science may be characterized by an attitude toward the world, an attitude that gets us asking particular kinds of questions, but the systematic approach to answering these questions requires the participation of other people working with the same basic assumptions about how we can engage with the world to understand it better. Those other people are peers, and their participation is a kind of review.

In both the ideal of reproducibility and the practice of peer review, we can see that the scientist’s commitment to producing knowledge that is as objective as possible is closely tied to an awareness that we can be wrong and a desire not to be deceived — even by ourselves.

Science is a team sport because we need other people in order to build something approaching objective knowledge.

However, teamwork is hard. In a follow-up post, I’ll take up some of the challenges scientists face in playing as a team, and how this might bear on the knowledge building scientists are trying to accomplish.

Stay tuned!