It has been eleven years since I was last on the market for an academic job, and about six years (if I’m remembering correctly) since I was last on a search committee working to fill a tenure-track position in my department. Among other things, this means that I can consider the recent discussion of “conference interviews” at The Philosophy Smoker with something approaching “distance”.
However, as I’m well aware, distance is not the same as objectivity, and anyway objectivity is not the kind of thing you can achieve solo, so I’m going to do a little thinking out loud on the screen in the hopes that you all may chime in.
The nub of the issue is how search committees in philosophy (and in at least some other academic disciplines) use preliminary interviews (typically 30 to 60 minutes in length) to winnow their “best” applicants for a position (as judged on the basis of writing samples, publication records, letters of recommendation, transcripts, teaching evaluations, and other written materials) down to the finalists, the number of which must be small enough that you can reasonably afford to bring them out for campus interviews.
The winnowing down is crucial. From more than a hundred applications, a search committee can usually reach some substantial agreement on maybe twenty candidates whose application materials suggest the right combination of skills (in teaching and research, and maybe also skills that will be helpful in “service” to the department, the institution, and the academic discipline) and “fit” with the needs of the department (as far as teaching, advising students, and also creating a vibrant community in which colleagues have the potential for fruitful collaborations close at hand).
But even if we could afford to fly out 15 or 20 candidates for campus interviews (which typically run a day or two, which means we’d also be paying for food and lodging for the candidates), it would literally break our semester to interview so many. These interviews, after all, include seminars in which the candidates make a research presentation, teaching demonstrations (hosted in one of our existing classes, with actual students in attendance as well as search committee members observing), meetings with individual faculty members, meetings with deans, and a long interview with the whole search committee. This is hard enough to squeeze into your semester with only five candidates.
So, the standard procedure has been to conduct preliminary interviews of shorter duration with the 20 or so candidates who make the first cut at the Eastern Division Meeting of the American Philosophical Association. For departments like mine, these interviews happen at a table in a ballroom designated for this purpose. Departments that have a bit more money will rent a suite at the conference hotel and conduct the interviews there, with a bit less background noise.
Job candidates pretty much hate this set up. The conference falls during winter holidays (December 26-30 or so), which means travel is more expensive than it might be some other time of year. Search committees sometimes don’t decide who they want to interview at the convention until quite late in the game, which means candidates may not hear that a department would like to interview them until maybe a week before the conference starts (boosting the price of those plane tickets even more, or making you gamble by buying a plane ticket in advance of having any interviews scheduled). Even at conference rates, the hotel rooms are expensive. Occasionally, winter storms create problems for candidates and search committee members try to get to, or to flee from, the conference. Flu season piles on.
Search committee members are not wild about the logistics of traveling to the convention for the interviews, either. However, they feel like the conference interviews provide vital information in working out which of the top 20 or so candidates are the most likely to “fit” what the department wants and needs.
But this impression is precisely what is in question.
It has been pointed out (e.g., by Gilbert Harman, referencing research in social psychology) that interviews of the sort philosophy search committees use to winnow down the field add noise to the decision process rather than introducing reliable information beyond what is available in other application materials. This is not to say that search committees don’t believe that their 30 or 60 minutes talking with candidates tells them something useful. But this belief, however strong, is unwarranted. The search committee might as well push itself to identify the top five candidates on the basis of the application materials alone, or, if that’s not possible, randomly pick five of the top twenty for campus interviews.*
Of course, search committees seem not to be in a great hurry to abandon conference interviews, at least in philosophy. My (brief) experience on the scientific job market didn’t include conference job interviews per se, but I did have preliminary interviews of very much the same nature and duration with some private sector companies and national labs — which is to say, I don’t think it’s just philosophers who are making hiring decisions that are at least partially grounded on a type of information we have reason to believe could be misleading.
The question, of course, is what to do about all this.
Search committees could abandon these preliminary interviews altogether. That would surely put more pressure on the written components of the applications, some of which might themselves be misleading in interesting ways. I’m guessing search committees would resist this, since they believe (although mistakenly, if the research is right) that they really are learning something important from them. It’s not obvious to me that job candidates would unanimously endorse this either (since some see the interview as a chance to make their case more vividly — but again, maybe what they’re making is pseudo-evidence for their case).
Search committees could work to structure preliminary interviews so that they provide more reliable information (as the research suggests properly structured interviews actually do).** This would require search committee members to learn how properly to conduct such interviews (and how properly to record them for later examination and evaluation). Moreover, it would require that search committee members do something like acknowledging that their instincts about how to conduct free-flowing, open-ended preliminary interviews that are also informative are probably just wrong. This is a task with a difficulty level that’s probably right around what it takes to get science faculty to acknowledge that having learned a lot about their field might not be sufficient to be able to teach it effectively, and that science education research might be a useful source of empirically grounded pedagogical insight. In other words, I think it would be really hard.
Search committees could keep conducting preliminary interviews as they always have. Inertia can be powerful, as can the feeling that you really are learning something from the interviews. However, it seems like a search committee would have to take into account the claim that, empirically, interviews are misleading when drawing conclusions on the basis of preliminary interviews. (Of course this is a normative claim — the search committees ought to take this worry into account — rather than a claim that mere exposure to a research finding would be enough to remove the search committee’s collective powers of self-delusion.)
Or … search committees could do something else?
What else could they do here? How do those of you in scientific fields handle the role of interviewing in hiring? Specifically, do you take concrete measures to ensure that interviews don’t introduce noise into hiring decisions? Or do you feel that the hiring decisions you need to make admit of sufficiently objective information that this just isn’t a problem for you?
If you prefer to comment pseudonymously for this discussion, feel free, but one pseudonym to a customer please.
_____
* For all I know, campus interviews may introduce some of the same kinds of noise to the decision-making process as conference interviews do. However, many include teaching demonstrations with a sample from the actual student population the candidate would be asked to teach if hired, a formal presentation of the candidate’s research (including responding to questions about it), and ample opportunity for members of the hiring department to get a sense of whether the candidate is someone with whom one could interact productively or instead someone who might drive one up a wall.
** It is worth noting that some search committees, even in philosophy departments, actually do conduct structured interviews.
Hm. I would like to see more detail of the research on interviewing before jumping to any conclusions. For instance, are they talking about free-flowing interviews with no set questions, or interviews in which the same questions are asked for every candidate?
Hi Janet — Can you give me some cites to research on what kinds of structured interviews can be relied upon to generate what sorts of reliable information? (I worry that highly structured interviews might be great for finding out whether someone is comfortable coding in Dart, but not so much finding out if they are a broadly good ‘fit’ for the department. I worry that the latter is *not* something that can be determined by *any* short interview structure, and hence not something we should be trying to determine via such interviews. But it seems to be what many people want to get out of those interviews. And if that’s the case, all social psych research can tell us is “you are barking up the wrong tree.”)
BTW: I am (as you might suspect) in broad agreement with your claims, including the difficulty in moving from the acknowledgement that 20-30 minute interviews at best provide no useful information, and at worst just permit us to give our worst biases freer reign, to finding an alternative. My arguments that, of our top candidates, we should just pick 3-5 to fly out, and acknowledge that those picks would be somewhat arbitrary, have not met with any real support…
In my field, if pre-interviews are done they are occasionally done at conferences as you describe but more often done via phone. Usually there is a prescribed set of questions asked to all of the candidates. At least in that mode you have a way of recording the interview or having someone taking good notes to help with the “data acquisition” problem. Still not sure about signal to noise ratio though. We’ll be doing phone interviews in January for a search I’m on, maybe I’ll have more thoughts on it then.
The research, as it’s been represented to me, indicates that the only kind of interviews that don’t introduce noise into the decision process are structured ones with a fixed set of questions asked of every candidate. A couple of examples of literature claimed to support structured interviews (which I’ve yanked from the comment thread of the above-linked Philosophy Smoker post) are here:
Wiesner and Cronshaw 1988 http://boardoptions.com/jobinterviewpredictivevalidity.pdf
Campion Pursell and Brown 1988 http://www.krannert.purdue.edu/faculty/campionm/Structured_Interviewing_Raising.pdf
Jonathan, I think our worries about interview utility are in the same neighborhood — and maybe it comes down to there not being many reliable predictive proxies for the constellation of things we’d really like to have in a new colleague. If that’s so, ultimately we’re going to have to accept some amount of trial and error. However, given how much time and energy it takes to conduct a search and hire someone (even if you dropped the interviewing), that’s a bitter pill to have to swallow.
I’m a software consultant not associated with Academia. In reading the article couple thoughts came to mind:
1. What is the 21st Century solution? Greener is better.
2. Has anyone done postmortem studies of the the clunkers hired, where the interview process failed?
1.What is the 21st Century solution?
It would seem that a video conferencing interviews , and viewing a live lectures, and Facetime-like conversations would avoid a lot of time, expense, and travel.
2. A collection of case studies of “disappointing” hires to review by the search committee along with personal experiences of lackluster hires would also go a long way toward improving the odds.
The search committee would seem to be weakest link. Conducting mock interview processes and discussing the goals of each step of the search process would seem like the best opportunity for improving the outcome. Moving to a 21st century approach would require this type of education process of the committee as well.
My experience with hiring process by committee is typically a 20-60-20 process. 20% of the time the hire exceeds expectations, 60% of the time they meet them, and 20% of the time they are disappointing.
My experiences my not be typical but in 40 years in industry little to no education/training was given toward improving the interviewers/committees skills.
A nice 21st century green solution would be to take advantage of virtual presence via technology and save the time and expense of the interviewers and interviewees. But this would require some re-education.
I’ve never understood the emphasis of the humanities on conference interviews, as opposed to Skype/phone interviews. The science departments I’ve interviewed with all did their preliminary screening by phone/skype, with a largely preset list of questions. It’s easier on the committee and the candidate, and for a half-hour or so interview, I don’t think you get much less information than you would in person. I’ve never been on the interviewing side, but this also seems more consistent with non-academic practices, where phone interviews are used to whittle the long list down to a short list for in-person interviews.
Maybe it’s because I’m a librarian and not in academia, but to me the very idea of forcing a candidate to fly across the country, get a hotel room, pay for conference expenses, and spend hundreds of dollars (while presumably unemployed!) in the process just for a thirty-minute interview isn’t just wasteful for the hiring committee, it’s outright immoral and unethical. This goes doubly when part of the reason for this extra step is explicitly to shift the costs to them. At least with the main conference there’s the possibility the candidates can do multiple interviews, which seems much less likely with this second stage. It sounds like if you need to do these interviews, then it’s time to invest in a webcam or a telephone.
Yes, a big (and legitimate) concern about the conference interview protocol is the financial hurt it puts on the applicants. My understanding is that most people who interview at the conference manage to line up multiple interviews (with different institutions that are all in attendance), and often placement committee folks from their graduate departments are there to help “spin” on their behalf at the social functions … it’s supposed to be a “learning experience,” but it’s far from humane. It would certainly be cheaper for everyone to do the interviews by Skype or Facetime or what have you.
But the question remains: Would such remote interviewing still introduce more noise than information into the decision-making (albeit for cheaper)?
Interesting. I did a few phone interviews. At least one I felt that the search committee and I ended up with a better understanding of our potential fit after the conversation. I’m would think that phone/skype would work well for the narrowing to the finalists stage.
Well, in the science departments i’ve been in, where there are no screening interviews and we just get five or six candidates from apps alone, I have to say that there are usually
about two in each round where you can tell from ten minutes into their seminar that they’re going to suck. So I think there might be utility in doing some briefer screening interviews, whether in person or by Skype.
Yeah….we’re interviewing now, and the latest one….I’m pretty sure the hiring committee wants to punch his letter writers in the face. Completely not ready to be interviewing. Some sort of preliminary skype interview would have made this pretty clear.
I am in the phone/Skype crowd. It does seem borderline immoral and wasteful for both candidates and search committees to do conference interviews.
Also, I chaired a search last spring where one of the candidates looked very good on paper, and was placed into the top three (based on a common rubric). We could not do conference interviews, but brought in the top three candidates. This candidate in person was terrible–we all knew during the first hour of a full day interview that s/he would not get an offer, and the rest of the day was more or less a waste of time. But with a phone interview, we could have screened him/her out immediately, this giving us more information than noise.
It also seems like a pre-screening need not be either/or. You could announce you were doing pre-screening, to get rid of candidates like the ones Peter R describes, and then let candidates know that they have the choice of meeting in person at the APA (or equivalent) or doing a phone/Skype interview.
This way you also defend against another big problem, that there might be a very appealing overseas candidate for whom it would be cost or time prohibitive to travel.
As long as you put in some defense against bias towards people one has met in person, this seems pretty workable.
So … I’m getting the sense that you lot think that some kind of screening interview can give a search committee actually reliable information that you wouldn’t get from the letters of recommendation, writing samples and/or published work, teaching evaluations, CV, transcripts, and such.
The interviews-are-just-noise camp says, sure you think you’re getting information, but you’re not actually (and that’s a positive argument to drop all but structured interviews, so you won’t be misled by the pseudo-information).
Is this simply a rejection of the conclusions of the interviews-are-just-noise camp? Or a hunch that the way they’re casting how reliable assessments of candidates are (with or without interviews) in a really narrow way that misses some of what search committees are actually looking for? Or a hunch that interviews are at least reliable in some rough way in the instances where the written application materials completely misrepresent the competency of the candidate (and you can totally figure that out when the candidate is in front of you)?
It’s not that I don’t share a lot of the intuitions expressed here. Rather, I’m trying to work out why we seem to be resistant to empirical research that’s supposed to tell us that we don’t actually get the information it feels like we’re getting in screening interviews (whether they are face to face or not).
First of all, thank you, Janet, for the links to the empirical research – I wish I’d seen that before I chaired a search committee last year. One that ended up bringing one of the most diverse sets of candidates to our campus that year, I might add. I’ll have to read up on the research in hindsight to see if we did anything right (or wrong). That said, let me share my experience from this (ultimately successful) search, and from serving on another search (that failed) a few years earlier.
As others have noted above, the norm in Biology (my discipline) is to do a round of phone interviews for winnowing down the finalists. Our department uses a standard questionnaire in 30-min structured interviews, with the conversations being recorded, and each committee member taking notes on copies of the questionnaire. The recordings are useful in cases where a committee member may have a scheduling conflict – but they are also available to other faculty members.
In general, I feel that this phone-interview setup does provide rather useful information for the screening process, and our structured questions help reduce the noise. One important element being that it keeps us from straying into areas that could be troubling from an EEO viewpoint. The committees of which I was a part found it pretty easy to pick the finalists based on the phone interview. During last year’s search, even our initial ranking of candidates was only confirmed by the campus interviews. In other words, the campus interviews gave us considerably more information, but little that changed the committee’s ranking of candidates based just on the phone interviews.
What about candidates who were winnowed out? Based on answers to the standard questions on the phone interview, we thought they were either poorly prepared, or more importantly, were likely to be a poor match to our department because they had misconceptions about the level of research facilities or teaching loads at an institution such as ours (which is similar to yours). That last part (unrealistic expectations) was really obvious based on answers to key questions, so I feel pretty good about the interview instrument we are using.
Our department has had several failed searches in the past, and not all committees have done things exactly the same way – so I can’t speak for them, obviously. If we leave out the more obvious reasons for searches failing (undesirability of our location; non-competitive salary; relatively small startup packages), and for several hires subsequently leaving, one big factor for failure is the one I mentioned above: lack of fit between the candidate (or person hired) and our campus in terms of academic culture / resources / expectations. Part of this I suspect may be due to a bit of overreach on our part, in trying to attract top candidates who were more geared towards RO1 type of institutions.
In at least one instance, though, I am aware that we dodged a bullet when negotiations failed after an offer had been made – because that candidate later turned out to have strong creationist sympathies!! Thereafter, I argued (successfully) that we include a question about a candidate’s comfort level when faced with a creationist student (of which we have many) in our phone interview. This question now turns out to offer us additional insight into the person’s pedagogic approach and philosophy about teaching critical thinking skills, so its been a good addition to the screening toolkit.
I’ll be happy to share our questionnaire with you if you are interested – email me.
Thanks again for raising such an important ponderable!
Hi Janet —
Thanks for the links! The note above re: the useful of structured interviews for avoiding EEO pitfalls is telling. At my current school, we use semi-structured interviews now (in part for that reason), and doing so, I think, introduces *less* noise into the process, and gives our (mostly unconscious) biases *less* room to be engaged. That’s good.
What’s not good is the belief that we are still learning something important about the candidate, either as a potential colleague or as a philosopher, from our 30 minute structured interview. Do we *think* we get useful information? Of course. Do we even agree on what that “information” means? Usually. Do I, on careful reflection, really think that I have good reason to believe we are right? No. Which is why I object to the process.
There may be people who are such disasters in person, but have somehow so impressed their committee members / recommenders, that a 30 minute interview does reveal something important. But they are few and afar between. The worry that this means you’ll have to spend two days with that rare disaster on the fly-out doesn’t cut it for me. I think it is just as likely that some of the people who, by chance (an off day, a bad headache, a missed connection…), had a terrible 30 minute interview, would in fact shine over two days, and would turn out to be fine colleagues.
Insofar as we want to learn about e.g. what the person might like to teach, or what ideas they have for new programs, or where they see their research going and how that might fit in with various research programs already ongoing in our department, we can ask about that as part of the application process, and read their considered, written responses. (Is being able to think up a new class on the spot, under stressful conditions, at a conference, a good talent to select for? I’m not sure, but I really don’t think so.) If a structured interview provides useful information, for philosophy, I think, the same questions in written form would provide equally or more valuable info. (You could send the questionnaire to your top 20 candidates, in place of the first round interview.) Insofar as we want to get a sense of how they might get on in the department, a few days on the ground isn’t great, but it is a lot better than a 30 minute interview (which is probably useless)! And I’m not sure what else we think we want to, or should want to, know…
Just my $02.
As a candidate for biology positions, I always appreciated phone interviews, even when they were highly stressful and didn’t go that well. I learned a lot about each school and in particular each department and how it worked based on the phone interview. I also appreciated the opportunity to ask questions myself–the internet means a huge quantity of information is available on every school, but most webpages are so generic that a phone interview is really the candidate’s first opportunity to tell what the school and department are actually like, what they actually value, how they actually work. I was invited to a few on-campus interviews without phone interviews, and those were much more difficult because I really felt I was going in with a lot of holes in my knowledge about the job and frequently felt blindsided by large details that I knew nothing about.
For the job I took, which I still have now, the phone interview was wonderful. I got a real feeling for how this place works and could see myself interacting, working, and even being friends with the people on the committee. It also taught me so much more about the school and changed the job from one I applied to because I applied to everything to one that I really wanted.
Pingback: 2012 in review: 12 months of Adventures in Ethics and Science. | Adventures in Ethics and Science
This is very late, 31 January, but I hope it’s of use.
My last full-time teaching was at King’s College London where I was a full prof in the War Studies Department (after a career in the US in academia and government).
All hiring at tax-supported UK universities is done with an interview committee using only structured interviews. Each member asks exactly the same questions of every single candidate in the same order, and experienced chairs know how to structure the time so it takes 60 or 90 minutes.
Every attempt is made to have the top 3-4 candidates interview sequentially on the same day, and also to give a research lecture; they don’t teach a class. The brilliant part of the process is that not later than the next morning’s mail every single candidate learns his fate. The person who gets the offer frequently knows the same day. I think the process works well, and is fair to all. I had the pleasure of snagging two brilliant lecturers who subsequently got hired away for very fine long-term jobs.
Of course, the department chair had spoken with me several times on the phone before I was interviewed. This was before Skype, so no video. I’m sure he also pre-interviewed several others; this was a titled chair.
peter zimmerman