Andrew Chen

Weird, contrarian business ideas
One of the best books I’ve read in the recent past is Hard Facts, Dangerous Half-Truths, and Total Nonsense by Stanford Professor Bob Sutton. (He now also has a great book called The No Asshole Rule, which you may have heard of also) In the Hard Facts book, he talks about a variety of different common business topics, and compares the academic research on each of the topics versus what paid management consultants often preach.

In particular, one of those question is – there’s a ton of anecdotes around the idea of The War for Talent, popularized with such phrases like “A-Players hire A-Players, B-Players hire C- and D-Players” etc. Embedded within many of these notions is, of course, the really big assumption that you can actually interview for talent, and that interview processes actually work. In the Hard Facts book, Professor Sutton actually points at a bunch of research that says that in fact, there’s tons of evidence that the hiring process doesn’t work well. And if you look at the marks that people get coming out of a hiring process versus the on-the-job marks they get in their first year in a job, they are actually not correlated at all.

I personally find the idea that interviews being poor predictor of job performance both unsurprising, but also troubling! Interviews predicting job performance seems like one of the core building blocks of American business.

This has been a particularly interesting topic for me to think about because of the differential that exists between technical and non-technical interviews also. All of the non-technical interviews I’ve ever involved in have been terrible, and I’m still not 100% satisfied by my thoughts on how to improve them.

Anyway, I wanted to embed a video interview of Professor Sutton discussing his book below, which you can watch at your convenience. Unfortunately he doesn’t mention job interviews in it, but he talks about a bunch of other interesting stuff.

(scroll down past the video to continue reading the blog post)

Short-term activity used to predict long-term activity
In fact, the core of job interviews really is about using some short-term activity (like dating, interviews, etc.) to try and predict some longer-term success (marriages, job performance). These little prediction scenarios pop up all over the place, and they’re inherently subjective.

Here are some other places where this takes place:

• Investors evaluating a pitch, in order to invest in a company
• Looking at headshots to cast someone in a play
• Reading the script of a movie to greenlight the film
• Being a great premed student versus becoming a doctor
• etc.

Some of these evaluation processes are closely aligned with the actual long-term activity, but sometimes they are not. For job interviews, it would seem that it may not be the same skills to get your resume noticed by a recruiter, then filtered up to a hiring manager, then passing an interview – that is hugely different than actually doing the job in a team setting. This discussion also reminds me of a common discussion I had with pre-meds back in college, where the primary shor-term selection criteria seemed like acing the easiest classes possible, pulling all-nighters, and memorizing obscure biology/chemistry textbooks. However, in the long run, being a good doctor was as much about dealing with people – be it other doctors, nurses, and patients – as it is about having good grades.

Is dating a good way to predict marriages?
OK, before we jump into job interviews, let’s talk about something more fun: Dating

The traditional notions of dating are a funny thing, because of how contrived it is in many ways. In particular, you might consider dating to have characteristics like:

• Pre-defined activities designed to be fun and happy
• Strong cultural, familial, and peer pressure that specifies standards and traditions
• Low sample size relative to long-term marriage commitment (dating for 6-12 months and then intending to be married for 60+ years)
• Relatively strong separation of stuff like finances, scheduling logistics, etc.
• And of course, no sense of what raising children is like

Contrast this to actually being married, as imagined by an unmarried guy like myself:

• Long-stretches of normal, domestic life that can be exciting, but is often not
• A very long-term outlook on the relationship, spanning 60+ years
• Lots of intermingling of legal issues and logistics
• And of course, the entire process of raising kids, having a house together, is almost an enterprise in itself regardless of the romanic situation

The fact that the before and after is so different, in many ways, means that you better hope that the stuff you learn about the other person while dating gives you a strong view of how the long-term relationship would work.

Job interviews as predicting long-term work relationships
Of course, job interviews are very much like dating as well. Inside of the job interview process, there are a bunch of inherent assumptions about what kinds of candidates are good candidates.

For example, you often have processes that prioritize top schools, or that prioritize “culture fit” and other intangibles. Interview processes often test very specific skills against a “snapshot” of a candidates skills at any given moment of time. Is it fair to ask engineers questions about SQL or specific language trivia when it’s something they might learn or pick up in hours or days, not months? It’s not clear.

The worse issue is around the inherent bias that comes into play. People like to hire people like them, and every startup full of Stanford-educated 20-something guys ends up hiring more Stanford young dudes. Sutton refers to this in his book as homosocial reproduction. How important is it, really, what school you went to? Is it more important your grades? Do you really need to know obscure things about a programming language, or go lower-level, when your day-to-day job is unlikely to utilize that knowledge? I think a lot of these biases come from the people who design the interview, and don’t objectively evaluate success or failure in professional settings.

Do interviews miss the “intangibles?”
Most damning, of course, is if interviews simply don’t test the majority of a job applicant’s fit to a role. Let’s have a thought experiment where for any job candidate, that skillset accounts for 20% of their performance, and other things like motivation, communication, and possibly weird, obscure skillsets actually contribute 80%? Then you can test the 20% all you want, but in any sort of 1:1, contrived conference room interview setting, you can’t scratch the 80%. In fact, you might find candidates that fail most of the 20% but are such amazing fits elsewhere that it’s in fact an awesome match!

Over time, I’ve come to believe that interviews likely test a small amount of a job candidates skills, and you have to more directly test them in realistic work scenarios to get at the other stuff.

How would you more directly test job performance?
In many ways, thinking about job interviews as inherently bad predictors has a strong tie to the Built to Fail blog post I did a while back – except rather than assuming that your code is bad, and needing unit testing to support it, instead you assume job interviews are bad, and you need a larger framework to support that.

So taking the ideas from that post, I would recommend the following:

• Accept that traditional job interviews suck, and you can’t learn much about a person in 30 minutes to an hour
• You should interview MORE people, and potentially lots of weird people that don’t seem to be good matches right upfront
• You must streamline your interview process to handle more people, in larger batches simultaneously – thus 8 hours of 1:1 interviews probably doesn’t scale
• And you should test your job candidates in realistic work scenarios – assigning real tasks to groups of candidates (potentially mixed in with employees), working together in tandem
• Also perhaps instead of focusing your hiring on specific individuals, instead you make offers conditional to entire teams that seem to work well together, and keep them together in their actual job

In many ways, I think this is closest to the Boiler Room or Bootcamp view of the world, which was pointed out to me by a long-term mentor, Bill Gossman. You bring in more people, test them in real-world settings, and hire whoever comes out on the other side, regardless of background or performance. To me, this has the benefit of a truly meritocratic society, where people are hired because of their real performance, rather than what the designers of the interview decided were subjectively important or unimportant.

Want more?
If you liked this post, please subscribe or follow me on Twitter. You can also find more essays here.