Making Hiring Decisions: Why "I'll Know It When I See It" Isn't a Strategy

In my last post on leadership decision-making, I took aim at the durable myth of the gut call, in which the executive absorbs the debate, consults some inner oracle, and delivers a verdict with the serene confidence of someone who has never been audited. That myth finds its most comfortable habitat not in the boardroom but in the interview room, where leaders who would never approve a capital expenditure without a business case will cheerfully make a six-figure hiring decision on the strength of a forty-five-minute conversation and a feeling that the candidate "just gets it."

We've all heard the phrases, and many of us (myself included, somewhere back in the early chapters of my career) have said them: strong presence, great energy, not sure she's a culture fit, I'll know it when I see it. Each one sounds like an assessment, and each one is, on closer inspection, a description of the interviewer's reaction rather than the candidate's capability.

A Lieutenant, a Clipboard, and a Near-Mutiny

The problem is hardly new, and neither is the fix.

In the mid-1950s, a 21-year-old lieutenant in the Israeli army's psychology unit was handed the task of repairing the interview process used to assign recruits to combat units. The existing method relied on interviewers forming an overall impression of each young man, and those impressions had proven to be of almost no predictive value. The lieutenant - Daniel Kahneman, who would go on to win the Nobel Prize in economics - replaced the free-form conversation with factual questions tied to six specific traits, each scored separately and in a fixed order, before any overall judgment was permitted.

The interviewers, as he later recounted in Thinking, Fast and Slow, came close to revolt, complaining that he was turning them into robots. So Kahneman offered a concession: once the six ratings were complete, they could close their eyes, imagine the recruit as a soldier, and assign an intuitive score. The structured ratings predicted performance substantially better than the old approach, and the intuitive score, arriving after the disciplined collection of evidence, proved useful as well. Intuition wasn't banished from the process so much as asked to wait its turn until it had something reliable to work with.

Seventy years later, though, most organizations still let intuition go first.

Decide What You're Deciding (Before You Meet Anyone)

The most consequential hiring decision is made before a single résumé is reviewed, and it's the decision about what, precisely, you're hiring for.

Establishing criteria at the outset - the three or four capabilities that will distinguish success in the role a year from now, what each looks like in practice, and which are non-negotiable versus which can be developed after someone walks in the door - is tedious work. Which is why it’s routinely skipped in favor of dusting off the old job description and giving it a cursory glance in order to get the requisition opened by EOD. And the cost of skipping this work shows up later.

When criteria aren't set in advance, they get assembled after the fact and have an uncanny habit of reshaping themselves around whichever candidate the hiring manager happened to like. It's the organizational equivalent of shooting the arrow first and then painting the bullseye around wherever it landed; the marksmanship looks impeccable, and no one learns anything about whether the archer can hit a target. The candidate with deep operational experience becomes the obvious choice because "we need someone who can execute," while a week earlier, with a different frontrunner, the same team was certain the role demanded strategic vision above all else.

Setting criteria beforehand won't eliminate that tendency, since nothing will, but it does give everyone something to be measured against, including the hiring manager's own enthusiasm.

Structure Works, With an Asterisk

The research case for structured interviews (the same core questions for every candidate, asked in the same order, rated against predefined criteria) is about as strong as anything in industrial-organizational psychology. When Paul Sackett, the University of Minnesota psychologist, and his colleagues re-examined decades of selection research in 2022, they found that structured interviews may be the strongest predictor of job performance - not cognitive ability, which had held that title in the field for more than half a century.

The same analysis, though, carries another detail that deserves attention. The mean validity for structured interviews was .42, but the 80% credibility interval around that figure ranged from .18 to .66 - which means a structured interview done well is among the most reliable tools we have, while one done poorly may not be much better than the conversational chat it was meant to replace. Structure, in other words, is a necessary condition rather than a sufficient one.  It's like handing every member of an orchestra the same sheet music, which does very little to prevent musical chaos if half of them read the notation differently than the other half.

And that brings us to the part of the process organizations most consistently neglect.

What, Exactly, Does "Meets" Mean?

Picture a scorecard with a competency labeled stakeholder management and a five-point scale beneath it, with a 3 designated as "meets." One interviewer awards a 3 because the candidate described navigating a contentious budget negotiation with a skeptical CFO; another awards a 3 because the candidate seemed personable and spoke warmly of former colleagues; a third, who privately believes no one deserves a 5 on anything, awards a 3 to a candidate who was plainly exceptional. The result is three identical scores masking three different judgments, followed by a debrief in which everyone believes they agree. Anyone who survived college remembers the professor who handed out an A for showing up and the one who hadn't awarded an A since the Carter administration. A hiring panel without shared definitions is a faculty composed of both.

The remedy is less glamorous than any piece of technology. Getting this set calls for behaviorally anchored descriptions of what a 2, a 3, and a 4 look like for each competency, written in the language of the job itself. This is then followed by calibration sessions in which interviewers score the same sample answer and discover, sometimes to their considerable surprise, how far apart they are.

Beneath all of this is a training gap that most organizations would prefer not to examine: hiring managers are rarely taught to interview and make selection decisions. Rather, they're promoted into the responsibility, usually on the strength of being skilled at something else entirely, and then expected to absorb the craft by osmosis. Returning to the distinction from my previous post, asking a good question is a skill, but running a sound selection process - probing past the rehearsed answer, recording evidence rather than impressions, and resisting the pull toward the candidate who reminds you of your younger self - is a competency. And competencies require deliberate development.

Where Judgment Still Lives

None of this removes judgment from hiring, and it would be a poor outcome if it did.

The final decision still belongs to people, and people will continue to weigh things a scorecard can't fully capture: how a candidate might complement a team's existing strengths, whether the timing is right, or what the organization will need eighteen months from now that it can't quite articulate today.

Building the scaffolding that supports those decisions - the intake conversation, the criteria, the anchored scorecards, the interviewer training - is front-loaded work that organizations tend to under-invest in until a regrettable hire forces the issue.

What structure offers is not certainty but a fairer contest between evidence and instinct, one in which the gut still gets a vote (Kahneman's interviewers were allowed to close their eyes, after all), though it no longer gets to vote first and count the ballots too.

-----

author: Robin Schooling