In the last 24 hours there has been an interesting conversation on the Twitters (with contributions from @drugmonkeyblog, @CackleofRad, @mbeisen, @Namnezia, @dr_leigh, @doc_becca, @GertyZ, @superkash, @chemjobber, @DoctorZen, and a bunch of other folks) on the value of standardized tests (like the GRE) in evaluating candidates for a lab position.
The central question at issue seems to be whether GRE scores are meaningful or meaningless in identifying some quality in the candidate that is essential for (or maybe reliably predictive of) success in the environment of an academic lab. And, it’s worth noting that the conversation has not been framed in terms of using GRE scores as the only piece of evidence one has about applicants. Rather, it’s been about the reliability of GRE scores as a predictor compared to college transcripts, letters of recommendation, personal essays, and the like.
I have thoughts about this issue, thoughts which are informed by:
- my teaching experiences
- my own experiences with the SAT and the GRE (I aced them)
- my own experiences doing research in four different lab settings (three of them while I was an undergraduate)
- my experiences teaching test preparation courses (for SAT I, SAT II, and MCAT)
- my experiences as the graduate student representative on a graduate admissions committee (albeit not for a science department)
- my experiences on hiring committees (where GRE scores weren’t an issue but things like letters of recommendation, grades, and personal statements were)
- broader ongoing conversations with colleagues about the challenges of finding reliable proxies with which to assess the success of our educational efforts.
What I have observed from these:
- There are extremely smart, capable people with severe test-anxiety. I’m talking puking-at-the-very-thought-of-sitting-fot-the-test anxiety. The people I’ve known with this manifest it most strongly when faced with standardized tests; generally they’ve found ways to deal with the other kinds of exams that are part of their schooling. I doubt that GRE scores would be reliable indicators of the fitness of such people for a position in an academic lab, unless that position involved taking standardized tests on a regular basis.
- My own success on standardized tests is mostly a measure of how well I understood the structure of those standardized tests. This is a lesson that was reinforced by my experience teaching others how to do better on standardized tests. I did not make my test prep students smarter about much of anything except strategies for taking the standardized tests. (In a few instances, my work with them may have helped them identify conceptual issues or problem solving skills that they needed to sharpen before test day, but again, I take it the “help” they got was primarily a matter of knowing what material and skills the test was going to assess.) Is understanding the structure of the GRE, or developing a good strategy for taking it, a crucial component of success in an academic lab? Probably not. Is it a reliable proxy for something that is? Maybe, but it would be nice to see an explanation of what that is rather than just putting our faith in the test to tell us about something that matters.
- Plenty of people with awesome test scores are hopeless in the lab. Plenty of people with non-awesome test scores are really successful in the lab. What’s the level of correlation? I don’t know, and you probably don’t either. Maybe someone should do an empirical study so we know.
- One place that standardized tests seem to be of use (or so I’ve heard repeatedly over the years from lots of admissions committee folks) is in “calibrating” grades, especially of schools with which one might have less familiarity. What does an A at Podunk U. mean compared to an A at Well-Known Tech? Presumably the GRE scores of the candidates give us some information (so, if they’re really low from the Podunk U. student, maybe Podunk U.’s As aren’t requiring the same level of mastery as Well-Known Tech’s As). But, there’s always the possibility that Well-Known Tech has a better developed organization from the point of view of getting its students into grad school, and that part of this might include in-house test prep. Also, what if the lone Podunk U. student who is applying to your program has test-anxiety?
- GRE scores are often thought of as an objective counterbalance to letters of recommendation because, as the common wisdom has it, letter writers lie. Or maybe they just put the best possible spin on the candidate’s talents. Or maybe they’re actually just overestimating the candidate’s potential. Or maybe they don’t write good enough letters for the students who are not like them in certain relevant respects (including scientific style, socioeconomic background, gender, race, sexuality, etc.). Surely, in many cases there is something like a positive bias in letters of recommendation (and some faculty will advise students to ask someone else for a letter if they themselves are unable to write a glowing recommendation). And, there are instances in which a letter writer will undervalue the talents and potential of students (although one hopes that the other letter writers in such cases will compensate). Still, the letters at least present a space in which actual concrete examples of the student’s awesomeness (or shortcomings) can be discussed. Some of these examples may touch on situations or challenges directly relevant to what the applicants may have to face in the academic lab in which they are seeking a position. Plus, at least in fields that are not totally enormous, there is (or could be) a professional cost to lying to a colleague in the profession, even in a letter of recommendation for a student.
- If I had to rely on just one proxy, it would be the applicant’s personal statement. Again, it strikes me that this is an instrument that creates a space where an applicant can describe past experiences and current interests, challenges overcome and lessons learned from them that might be applied to future challenges. A personal statement can give you a glimpse into what the applicant cares about and why. It can also give you a sense of whether the applicant can think and communicate clearly. However, this is probably another area where someone should do some empirical work to see what kind of correlation there actually is between the quality of the personal statement and the success of the applicant in the position for which the personal statement was part of the application package.
- Every single proxy we might look at to select among applicants can fail. It’s not clear to me that it could be otherwise, especially given that we’re using the proxies to try to predict future success, which you can’t do with perfect accuracy unless you have a machine for seeing into the future (and even then …).
- It strikes me that active thinking-on-your-feet interview questions might provide more relevant information. It used to be that you couldn’t really use these for things like grad school admission because you couldn’t afford to fly all your applicants out to campus. (By the time you saw prospective grad students, they were admits trying to choose between the programs that had accepted them.) But maybe now with tools like Skype those looking to make sensible choices among applicants should do some video interviewing?
- Then again, if video interview questions for lab positions become a thing, someone will probably set up a video interview preparation company.
Yeah, I’d say to take GRE scores with a grain of salt. But, I think that’s the right attitude to take to all the bits of evidence an applicant presents. Honestly, my attitude toward test scores probably has a lot to do with my knowledge about how easy it can be to do well on them (at least compared to the other pieces of one’s application package). It probably also has to do with at least a few gatekeepers who treated GRE scores as definitely more reliable simply because they were quantitative, rather than qualitative.
If you have an applicant-screening item that has never led you astray, please share it in the comments.
I largely agree, but evidence ought to be weighted how reliably it predicts success.
So is there any evidence showing a correlation between GRE scores, academic success, or research productivity?
“My own success on standardized tests is mostly a measure of how well I understood the structure of those standardized tests.”
That’s exactly why standardized tests can in many (not all) cases be good predictors of success in science. The people who do well, in most cases, have grasped what’s *really* being tested — the real, underlying meta-question, rather than what the exam merely appears to be about.
Dario: There’s this:
Kuncel NR, Hezlett SA. 2007. Standardized tests predict graduate students’ success. Science 315(5815): 1080-1081. http://www.sciencemag.org/cgi/content/summary/315/5815/1080
I recall a round of criticisms in the letters sections later. There is much more research, I am sure, and I expect much will say the test is not highly predictive.
I received my BS while on scholastic probation, and was accepted as a probational student in an MS program on the basis of my really impressive GRE scores. As a professor, I encouraged one of my undergraduate students, who had every success quality except a good academic record, to apply to our MS program. We required GRE scores from applicants to our MS program. I lurked around the office and intercepted his, which never made it into his folder, which was not noticed. He did a fine job on his thesis, had an appropriate job before he finished his MS, became president of his professional society,etc.
Your #2 is particularly full of crap. First, I’ve been reading your stuff for a long time and had a chance to have several personal conversations with you. Those reinforce your claim to good scores, i.e., I would have predicted you are smart as all get out and would have done well on standardized tests in the past. So you are expressing false modesty that you somehow got by on a sort of technical insight. In terms of your teaching test prep, meh. Are you suggesting you can take anyone at all and make them a perfect scorer? If so, I buy your argument. If you cannot do much other than improve scores by essentially a fixed amount from their baseline…I’m not so impressed. Related to this, I want to see you partial out the effects of 1) taking the test a couple of times and 2) A mild sedative/anxiolytic against your test prep. Then I want to know more about those that select test-prep. Are they enriched for those who perform “suspiciously” low? are they outliers from the get-go? if so, you talking about dealing with the variance is not so convincing about the central tendency.
I was a decent test taker in most of my educational history, but I wasn’t happy with my GRE scores the first time through. So I went back and crammed for a few weeks and upped both the verbal and the math about a hundred points (without a test prep program). There wasn’t *time* for any real intelligence/qualities of mind/problem-solving prowess changes, only test taking skill development (i.e. practice). And it’s not like my technical insight or test taking skills were non-existent to start with.
So I’m pretty sure the difference between a sort of OK score and an “asset on a graduate school application” score can be pretty trivial.
DM @5, I don’t want to claim the scores aren’t correlated to *something* (or, more likely, some constellation of things). It’s just that my hunch is that what exactly they’re measuring is a pretty ham-handed proxy for the kind of intelligence, qualities of mind, problem-solving prowess, or what have you that the people looking at the scores *want* to know about.
Also, the attribution of false modesty rankles, especially as I’m happy to grant:
I can probably ace any standardized test you want to give me (especially if you give me some practice items and an explanation of the scoring rules), and
I’m pretty smart.
The thing is, being damned good at standardized tests is a relatively narrow skill — certainly not one that helped me much in figuring out how to troubleshoot experimental protocols, or how to develop original research questions, or how to analyze data, or how to write a dissertation. I had to develop these other “smarts”, many of them after I aced the tests that got me in the door.
Especially since I know a non-negligible set of people who are smart as all get out in each of my chosen disciplines but who did not do particularly well on the standardized tests, it seems to me like a gap in something like technical insight into the tests may have been at play for them. For whatever reason, if test scores were all you had to sort out prospects, they might not have made the cut — even though their intelligence, qualities of mind, problem-solving prowess, or what have you seemed frequently to be sharper than mine when it came to the real-world challenges of doing research.
Honestly, though, I think someone ought to do a study here. Otherwise, we’re just putting our intuitions up against each other (and usually we like our own intuitions best because, well, they’re ours).
I’m a science grad student, and I’ve been on the admissions committee for my program for two years now. Given that my attitude toward standardized testing is the same as Janet’s (I can ace pretty much anything, but I think those skills have very little bearing on the skills I use as a grad student), I was a little bit surprised to find out that the professors who have sat on the committee for a long time believe that GRE scores can be good predictors of anything. I was even more surprised at what the professors think the GREs predict:
(a) low scores are a bad sign because they imply that the student didn’t care enough about grad school applications to study for the test;
(b) exceptionally high verbal scores are a good sign because they imply that the student is well-read, which is correlated with having sought out a wide range of ideas, which is correlated with creativity and flexible thinking.
Students typically don’t get any special consideration for exceptionally high math scores, even though we’re in a very quantitative field. The professors don’t have actual data to back up these hunches, of course, but they do play a small part in our decision-making.
On the other hand, several of our applicants from East Asian schools have verbal scores above 90th percentile yet have fluency issues in their personal statements. Gotta look at the whole package.
I suspect there’s a false-positive / false-negative issue here. Standardized tests do measure something – it’s hard to get a high score without being smart and/or well-educated – but there are plenty of reasons why someone may underperform.
The tests also don’t tell you anything about the higher-level skills of the applicant. I’m the Mr. Rogers of standardized tests (never met one I didn’t like)…and yes, I’ve got a high IQ and a mastery of 6th-grade reading and ‘rithmetic. It doesn’t guarantee that I have the temperament for research or the ability to function in a workplace.
If I were reading these things (I’m a grad student, so not yet), I would also be inclined to look at the personal statement. It does say something about how the student thinks, and whether they can present those thoughts clearly and professionally. The quality might vary a lot with how well the student was advised – I’m quite sure my own first attempt was terrible – but then again, I fixed it with the help of the internet, since I’d been out of school for years and had no one to ask. It’s possible to teach yourself what works if you care enough to do so.
As said above, I received my BS while on scholastic probation. I had no intention of going to graduate school, but when I heard of the GRE, I decided to take it because I am good at standardized tests; so it couldn’t hurt. This is in 1957, and I doubt people had figured out one could study for the GRE. So, anytime you encounter something you are good at, get it on your record.