In the last 24 hours there has been an interesting conversation on the Twitters (with contributions from @drugmonkeyblog, @CackleofRad, @mbeisen, @Namnezia, @dr_leigh, @doc_becca, @GertyZ, @superkash, @chemjobber, @DoctorZen, and a bunch of other folks) on the value of standardized tests (like the GRE) in evaluating candidates for a lab position.

The central question at issue seems to be whether GRE scores are meaningful or meaningless in identifying some quality in the candidate that is essential for (or maybe reliably predictive of) success in the environment of an academic lab. And, it’s worth noting that the conversation has not been framed in terms of using GRE scores as the only piece of evidence one has about applicants. Rather, it’s been about the reliability of GRE scores as a predictor compared to college transcripts, letters of recommendation, personal essays, and the like.

I have thoughts about this issue, thoughts which are informed by:

• my teaching experiences
• my own experiences with the SAT and the GRE (I aced them)
• my own experiences doing research in four different lab settings (three of them while I was an undergraduate)
• my experiences teaching test preparation courses (for SAT I, SAT II, and MCAT)
• my experiences as the graduate student representative on a graduate admissions committee (albeit not for a science department)
• my experiences on hiring committees (where GRE scores weren’t an issue but things like letters of recommendation, grades, and personal statements were)
• broader ongoing conversations with colleagues about the challenges of finding reliable proxies with which to assess the success of our educational efforts.

What I have observed from these:

1. There are extremely smart, capable people with severe test-anxiety. I’m talking puking-at-the-very-thought-of-sitting-fot-the-test anxiety. The people I’ve known with this manifest it most strongly when faced with standardized tests; generally they’ve found ways to deal with the other kinds of exams that are part of their schooling. I doubt that GRE scores would be reliable indicators of the fitness of such people for a position in an academic lab, unless that position involved taking standardized tests on a regular basis.
2. My own success on standardized tests is mostly a measure of how well I understood the structure of those standardized tests. This is a lesson that was reinforced by my experience teaching others how to do better on standardized tests. I did not make my test prep students smarter about much of anything except strategies for taking the standardized tests. (In a few instances, my work with them may have helped them identify conceptual issues or problem solving skills that they needed to sharpen before test day, but again, I take it the “help” they got was primarily a matter of knowing what material and skills the test was going to assess.) Is understanding the structure of the GRE, or developing a good strategy for taking it, a crucial component of success in an academic lab? Probably not. Is it a reliable proxy for something that is? Maybe, but it would be nice to see an explanation of what that is rather than just putting our faith in the test to tell us about something that matters.
3. Plenty of people with awesome test scores are hopeless in the lab. Plenty of people with non-awesome test scores are really successful in the lab. What’s the level of correlation? I don’t know, and you probably don’t either. Maybe someone should do an empirical study so we know.
4. One place that standardized tests seem to be of use (or so I’ve heard repeatedly over the years from lots of admissions committee folks) is in “calibrating” grades, especially of schools with which one might have less familiarity. What does an A at Podunk U. mean compared to an A at Well-Known Tech? Presumably the GRE scores of the candidates give us some information (so, if they’re really low from the Podunk U. student, maybe Podunk U.’s As aren’t requiring the same level of mastery as Well-Known Tech’s As). But, there’s always the possibility that Well-Known Tech has a better developed organization from the point of view of getting its students into grad school, and that part of this might include in-house test prep. Also, what if the lone Podunk U. student who is applying to your program has test-anxiety?
5. GRE scores are often thought of as an objective counterbalance to letters of recommendation because, as the common wisdom has it, letter writers lie. Or maybe they just put the best possible spin on the candidate’s talents. Or maybe they’re actually just overestimating the candidate’s potential. Or maybe they don’t write good enough letters for the students who are not like them in certain relevant respects (including scientific style, socioeconomic background, gender, race, sexuality, etc.). Surely, in many cases there is something like a positive bias in letters of recommendation (and some faculty will advise students to ask someone else for a letter if they themselves are unable to write a glowing recommendation). And, there are instances in which a letter writer will undervalue the talents and potential of students (although one hopes that the other letter writers in such cases will compensate). Still, the letters at least present a space in which actual concrete examples of the student’s awesomeness (or shortcomings) can be discussed. Some of these examples may touch on situations or challenges directly relevant to what the applicants may have to face in the academic lab in which they are seeking a position. Plus, at least in fields that are not totally enormous, there is (or could be) a professional cost to lying to a colleague in the profession, even in a letter of recommendation for a student.
6. If I had to rely on just one proxy, it would be the applicant’s personal statement. Again, it strikes me that this is an instrument that creates a space where an applicant can describe past experiences and current interests, challenges overcome and lessons learned from them that might be applied to future challenges. A personal statement can give you a glimpse into what the applicant cares about and why. It can also give you a sense of whether the applicant can think and communicate clearly. However, this is probably another area where someone should do some empirical work to see what kind of correlation there actually is between the quality of the personal statement and the success of the applicant in the position for which the personal statement was part of the application package.
7. Every single proxy we might look at to select among applicants can fail. It’s not clear to me that it could be otherwise, especially given that we’re using the proxies to try to predict future success, which you can’t do with perfect accuracy unless you have a machine for seeing into the future (and even then …).
8. It strikes me that active thinking-on-your-feet interview questions might provide more relevant information. It used to be that you couldn’t really use these for things like grad school admission because you couldn’t afford to fly all your applicants out to campus. (By the time you saw prospective grad students, they were admits trying to choose between the programs that had accepted them.) But maybe now with tools like Skype those looking to make sensible choices among applicants should do some video interviewing?
9. Then again, if video interview questions for lab positions become a thing, someone will probably set up a video interview preparation company.

Yeah, I’d say to take GRE scores with a grain of salt. But, I think that’s the right attitude to take to all the bits of evidence an applicant presents. Honestly, my attitude toward test scores probably has a lot to do with my knowledge about how easy it can be to do well on them (at least compared to the other pieces of one’s application package). It probably also has to do with at least a few gatekeepers who treated GRE scores as definitely more reliable simply because they were quantitative, rather than qualitative.

If you have an applicant-screening item that has never led you astray, please share it in the comments.