Google Respins Its Hiring Process For World Class Employees 305
An anonymous reader writes "Maybe you've been intrigued about working at Google (video), but unfortunately you slept through some of those economics classes way back in college. And you wouldn't know how to begin figuring out how many fish there are in the Great Lakes. Relax; Google has decided that GPAs and test scores are pretty much useless for evaluating candidates, except (as a weak indicator) for fresh college graduates. And they've apparently retired brain teasers as an interview screening device (though that's up for debate). SVP Laszlo Beck admitted to the New York Times that an internal evaluation of the effectiveness of its interview process produced sobering results: 'We looked at tens of thousands of interviews, and everyone who had done the interviews and what they scored the candidate, and how that person ultimately performed in their job. We found zero relationship. It's a complete random mess.' This sounds similar to criticism of Google's hiring process occasionally levied by outsiders. Beck says Google also isn't convinced of the efficacy of big data in judging the merits of employees either for individual contributor or leadership roles, although they haven't given up on it either."
This has led TechCrunch to declare that the technical interview will soon be dead.
Have you ever built something that worked ... (Score:5, Interesting)
Equally important, and admittedly a little strange to some, it to ask about their personal programming projects. Nothing work related, nothing school related, just things that they sat down and programmed motivated by their own personal needs or curiosity. If a person can not offer "something" a warning bell is going off. I don't care how small, trivial, silly, etc the personal project is. I mostly want to see that personal projects exist. To me they are an indicator that the interviewee is someone who has a genuine interest in programming, that they are not merely someone who got a degree because a parent or guidance counselor told them it was a good career path.
Not THAT surprising. (Score:5, Interesting)
But not because Google went about it wrong and screwed up its hiring process.
I've been now through a few hiring processes, have sat on Interviews, decision committees. And while I like to think that my Interviews and candidate ratings were spot-on (I correctly predicted one failure and one early resignation), I'm pretty sure that's just skewed by the small sample size. What I do know is that I went through all kinds of approaches, both as an interviewer and an interviewee. I've done brainteasers, role-playing, decision explanations, code walkthroughs, resume deep-dives, online candidate research, just shooting the breeze, and more. And I haven't found a single thing that strongly correlates with acing the interview or hiring a good worker. Resumes can lie (sometimes subtly), and you'll never find out without hiring a private investigator. Role-playing can confuse people, especially if they're trying to figure out what you're looking for. Brain teasers can be memorized, shooting the breeze can lead to unreasonable judgments (positive or negative), interviewers and interviewees can have a bad day, the other person doesn't like your first name, and a million other things.
Especially when you start talking 10s of thousands of interviews, you're actually looking at so much data, so many influencing variables that I doubt you can find one common variable that stands out from the rest. What I'm concerned about (and that comes partially from being married to someone in HR) is that there is still a drive to find the one process that will automate the hiring process. As far as I can tell, it doesn't exist. Well, let me walk that back a tiny bit: there's one thing that will work better than anything else: have the interview done by the best people you have, have them take it seriously, and spend some time on it. But it takes time, is fuzzy, and is entirely reliant on managers knowing who their best people are.
I'm glad to see that Google doesn't think Big Data is the answer to everything. I just hope that this percolates through to the rest of the HR universe. There's much too much of a drive to automate hiring, like performance reviews and firing has been.
Re:Only 1 sensible answer to interview brainteaser (Score:3, Interesting)
The days when a back-of-the envelope calculation is enough are long gone (and probably never existed int he real world anyway).
Very much disagree on the lack of back-of-the-envelope calculations. von Braun and co. solved some of the hardest problems of Satern V development with paper napkins. I use quick calculations and engineering judgement all the time, and hire folks who are good at them too. In fact, we often spend far too much effort doing excessive studies when a few minutes of napkin math would give you the 80% answer. However, being able to figure out brain teasers and being able to quickly perform sound engineering judgements in a real work environment are two very different things.
Re:Only 1 sensible answer to interview brainteaser (Score:5, Interesting)
I view interviews as two way. I'm evaluating the company. For example if the "senior engineer" giving me the above test doesn't know who Knuth is I probably don't want to work there. He did, but he pointed out my unconventional answer to the manager of the team. A person with a business background not a technical background. This manager asked what "Knuth" was and I explained. He then got a big smile, he loved my answer. A few days later I got a job offer. I worked there for four years, he was a suit, but he was a good one. He shielded us from as much BS as he could and he trusted and generally accepted our technical recommendation even when he personally had doubts.
No new information here (Score:5, Interesting)
I suppose it's news that the internal study found no correlation between interview scores and job performance, but everyone at Google recognizes that getting hired is a crapshoot. Not totally random, of course; there are plenty of candidates who simply aren't going to get hired, ever, because they don't have what it takes. But (I'm speaking of engineers here, dunno about other areas), everyone knows that candidates who are of the caliber Google seeks may or may not pass the interview process, and whether or not they do is pretty much a toss of the dice. I've heard rumors of a an internal study that took successful Google engineers and put them through the interview and hiring process, obscuring their employee status... and about half of them were "re-hired".
Also, as McDowell's blog post says, Google has always instructed interviewers not to use "brainteaser" questions. It probably does still happen once in a while -- indeed one of my interviewers asked me a "bonus" question, after I'd already demolished his design/coding problem, which arguably falls into that category (I failed to answer it) -- but they're doing it wrong and the hiring committee will let them know it.
Anyway, so if Google's process has such random results, why do they continue to use it? Simple: because nobody has found a better way. And the study results mentioned are a little misleading if you don't understand them in context: The study was of tens of thousands of interviews and their correlation with the performance of people who were hired. And nearly all of the people who are hired by Google go on to have successful careers at Google. What the study shows is that the degree of success is not correlated with the strength of the hiring recommendations.
On the other hand, as someone who came to Google with 20+ years of industry experience already behind him as a basis for comparison, I'll tell you one thing about the Google hiring process: It hires good people. It also fails to hire a lot of good people, but there are vanishingly few plodders or obstructionists around. In the 2.5 years I've worked for Google I have worked with well over 100 engineers (my work tends to touch lots of teams), and I've met one, maybe two, who weren't bright, highly competent and very effective, and even those one or two would be good-performers most places. That is very different from my prior experience, and I worked with a lot of high-profile companies.
As another data point, at every one of my prior employers I was something of a star, commonly called a "genius" and similar in performance reviews. At Google... I'm merely competent, perhaps a bit below average. Many of my colleagues are much smarter than me, and the superstars at Google are absolutely brilliant. One woman in particular who I've worked with quite a bit is always at least four steps ahead of me. She constantly says things that I think are stupid... until I have time to catch up with her thought process. She also talks faster than anyone I've ever met, in an attempt to try to keep up with her brain, I think. Talking to her is exhausting, but exhilarating. I've taken to structuring my conversations with her so they are always interrupted after no more than five minutes because that's about all I can take before I need to go process for a while. My consolation is that I notice many other people interact with her in the same way. Overall, my experience of Google employees that they're all smart, energetic and talented, with a strong leavening of the truly brilliant, and that perception extends even outside of engineering. Hell, our building facilities manager is really sharp.
What I experience of my colleagues is exactly what Google aims to achieve: since there's no known way to make accurate hiring decisions, the interview process aims primarily to filter out candidates who aren't fairly outstanding. In the process, it excludes a lot of really talented people, but it's very effective at excluding basically all of the poor to mediocre candidates.
I'm just glad the dice went my way when I interviewed.
Re:In conclusion (Score:5, Interesting)
Because (while no HR department or team manager will ever admit it in a public forum) we as a civilisation have precisely zero idea how to hire decent staff.
Oh, we'd love to pretend we do. We come up with all sorts of wonderful ideas like technical interviews (what the hell is a technical interview and how should it be structured anyway? I've never yet been given any training on that, yet I've had to devise them on a few occasions - I usually went for questions that demonstrate the candidate is trying to think through the problem in a methodical way rather than just guessing or reciting answers they've memorised), brainteasers, psychological evaluations - yet I'm quite sure we'd get just as good results on average just pulling names out of a hat.
Re:In conclusion (Score:4, Interesting)
In my experience its not the questions or the answers (unless complete wrong). I look at their demeanor. Are they noticeably flustered or do they take a breathe and start working it out. I'm looking to see if they can speak to subject matter they list on the resume. How do they speak about it... concisely or scattered. This tells me their real experience level and I can then decide if they are a good fit for my needs. Then I just ask them what they are passionate about, what makes them stay up at night thinking or experimenting. This gives me a feel for how they will grow in their skills. Is it aligned with the job or headed in a different direction.
This doesn't always work but I've been right more than wrong with an 80% success rate. I had one guy who got divorced weeks after I contracted him and just lost all ability to focus. Unfortunate circumstances but life happens and you've got to roll with it. Had to let him go. Wasn't pulling his weight.
I've brought on two so far who've been promoted to managers themselves and several others who are leads on other teams now.
Re:Puzzles are pointless (Score:4, Interesting)
Casual chats are okay, but they miss a lot, and favor those who are good at chatting over those who aren't.
For years my approach was to give a couple of simple programming problems to weed out those who'd waste my time, followed by a chat like you describe. It was okay. But the interview training given to me by Google showed me a much better way. It's not about "puzzles", those are pointless and Google has never used them. What works much better is to give people problems to solve and watch how they go about it. You want problems that are fairly realistic, but sufficiently self-contained they can be solved and coded in 30 minutes, and sufficiently open-ended that when you get a really good candidate who just blasts through it there's plenty of room to explore variations. You should also not be afraid to give hints if the candidate is clearly getting hung up on some bit. Obviously if you end up having to walk the person through the whole solution they're not a good hire, but even sharp people sometimes need something pointed out when they're under time pressure and being watched.
Above all, you want to identify the people who really engage with the problem, who forget about the interview and dive into it, and who show good problem-solving ability and agility.
This approach provides the interviewer with a lot more insight than casual chats, including helping you to find those people who are really capable but aren't good conversationalists.
Re:My interview experience with Google... (Score:5, Interesting)
I'm not going to tell you that you should have regretted it, but I think you made a bad decision. It's likely she asked you about the RFC in order to see if this was something you already knew about. If you had indicated knowledge of it, she'd have moved to something else. Since you didn't already know it, it was exactly what she needed, an opportunity to watch you try to work your way through a problem. And the fact that your solution was more wrong than right apparently didn't dissuade her from thinking your approach indicated good ability.
This is assuming that she was asking you to come up with a solution, not just to regurgitate facts. If it was the latter, well, she was a poor interviewer, sorry. Google tries to train people not to do that, but training can fail sometimes.
Re:Have you ever built something that worked ... (Score:3, Interesting)
Actually, speaking as a doctor in the UK (general surgery), on paper a lot of doctors look very similar from a training/logbook point of view. Prestigious jobs are very competitive and traditionally the most important way of discriminating between them is their publishing of academic papers, attendance at relevant conferences, hospital audits, completion of extra courses - usually done (at least in the UK) in their spare time. This shows interest in their field and is analogous to computer programmers having hobby projects. I'm a doctor who makes computer programs in his spare time which makes me a little odd.
Re:In conclusion (Score:5, Interesting)
In my experience its not the questions or the answers (unless complete wrong). I look at their demeanor.
I've done exactly this and while I think it's probably a better way to hire good staff, I've been told that it's a bad idea from an HR perspective.
Apparently they like a nice simple list of questions with model answers, and a hiring decision based purely on how close the answers given are to the model. This is nothing to do with ensuring you get good staff; it's so the people you reject can't claim they've somehow been discriminated against.
Oddly, those same HR people are remarkably bad at answering the simple question "Okay. So how exactly do I write your list of questions and answers in order to ensure that your method is as good as mine for filtering out bad hires?"