Harder programming questions do a worse job of predicting outcomes - <AUTHOR> https://triplebyte.com/blog/interview-questions-are-too-hard-and-too-short <comment><USER2> Google recruiters call me a lot. I think I'd do a good if not stellar job working there. I've passed multiple FAANG interviews and been very successful as a senior developer.

In my email I have an "interview prep packet" from them that essentially tells me to brush up on algorithms and read Cracking the Coding Interview to prepare for their interview process.

I'm fairly happy in my job. If they offered more money or a really interesting project I'd consider working for them. But I'm pretty lazy about redo-ing college algorithms class during my free time at home to go work there, so I probably won't.

There's an opportunity cost with interviews like this where an M.S. and long career of getting shit done counts for very little and memorization of undergrad level topics that you can look up in two minutes in Knuth if you have a problem that requires it can make or break an interview.

I've made a career fixing a ton of horribly shitty, inefficient code that's been produced exclusively by people who pass these interviews.

<comment><USER3> You wouldn't be a good fit for Google. With their algorithmic interviews that require college-grad level of studying, they filter for people that are ready to follow orders without complaining.

That's who they want to hire at the end of the day: some coders that don't get too critical about their job and do what they are asked to do, even if it is repetitive, stupid and doesn't really make sense (such as re-studying algorithms implementation details for two weeks before an interview when it can be looked up super-easily online).

<comment><USER4>
      some coders that don't get too critical about their job

Couldn't be farther from the truth.

As someone who works there I've noticed this general trend of animosity towards Google, it seems to be fueled by the fact that people feel a sort of inferiority complex when they don't clear an interview or they feel they wouldn't be able to crack it if they ever gave it a shot, this leads to an overcompensation of attitude in the other direction, a sort of "sour grapes" narrative where there is an effort to downplay the prospects of working at Google or ridicule the people working there like you did just now.

<comment><USER5> >it seems to be fueled by the fact that people feel a sort of inferiority complex when they don't clear an interview or they feel they wouldn't be able to crack it if they ever gave it a shot

Some of the most talented people I've ever met have failed Google interviews. As in far, far more talented than the majority of folks I know who do work at google. No amount of bullshitting is going to convince me that Google's False Negative interviewing system is the correct one. It simply bypasses too many talented candidates even beyond the reasons generally mentioned above.

One guy in particular I know is now a CEO of a company which, ironically enough, employs several former googlers. Yes, I know Google doesn't give a shit they missed on this individual. But to mis-characterize the anti-Google- interview narrative as "sour grapes" sounds a bit like you're drinking their champagne.

<comment><USER6> I've met my share of Googlers who are lazy, don't know how to solve problems (as opposed to write code itself), or still need help with basic tasks after several years on the job.

You just can't test for some qualities like focused, persistent problem solving and the desire to find ways to speed up your work.

</comment></comment></comment></comment></comment><comment><USER7> Aside from all the things mentioned in the article, this also seems like a fairly predictable application of Goodhart's Law: "Any observed statistical regularity will tend to collapse once pressure is placed upon it for control purposes."

Once upon a time, skill at doing these sorts of problems might have correlated (imperfectly) with general aptitude as a programmer or software engineer. But the very act of trying to leverage that correlation for hiring purposes probably also made it go away. Now you've got a whole lot of people practicing hard on these sorts of problems, spending huge chunks of their free time grinding away on Project Euler and Advent of Code and HackerRank. That muddies the quality of this stuff as a proxy for what it was originally trying to detect: natural aptitude. I'm guessing having time to level grind like that also correlates inversely with other traits that are desirable in a programmer.

<comment><USER8> What's frustrating to me is that we haven't found a way to teach programming that doesn't rely on natural aptitude.

It's really a travesty that we can't teach it the same way that we teach maths or natural languages.

The end result is we're left trying to divine whether someone is the programming equivalent of being illiterate. As with illiteracy people find ways to fake it.

<comment><USER9> As someone who has studied both math and computer science (and is now a professional programmer), I have no idea what you are talking about. We are far better at teaching people to program than we are at teaching them to do math. Its just that most people self select out of math [0], so, due to selection bias, it appears that we are great at teaching those that remain.

As an aside, as someone who also studied linguistics [1] (and a couple of foreign languages at a beginner level), I am very confident in saying that our approach to teaching natural languages relies almost entirely on natural aptitude. It is just that, absent a serious mental disorder, all humans have a very large natural aptitude for natural languages

[0] Probably because we are absolutly terrible at teaching it.

[1] I assume not what you mean by teaching natural languages, but the topic of language acuisition (including in adults) does come up; plus it gives some perspective on how teaching language would look if we didn't rely on natural aptitude.

<comment><USER10> I took math as part of my engineering degree. And that was 30 years ago. The older I get the more I think the way it was taught was terrible. Same reason I think teachers are bitching about the US's fetish for academic testing. Testing pressures teachers to teach students to mechanically solve problems. But with shallow understanding.

<comment> crooked-v I think of monads, which are real easy to explain in a programming context as soon as someone understands lists and map/reduce, but are total gibberish in a math context even if you've gotten through calculus.

</comment></comment></comment></comment></comment><comment><USER11> The less talked about absurdity is that successfully passing programming interviews is a skill itself. It's especially absurd because the time I spend developing that skill is less time spent developing skills and knowledge more directly relevant to my job. Yet, programming interview skill is more relevant to progressing my career.

edit: now if you'll excuse me, I need to do some dynamic programming problems.

<comment><USER12> This is kind off strange for a non US resident to grasp. I've been employed as a programmer three times and noone tested my coding abilites what so ever on the interviews. No samples, nothing. Never heard of any collegue doing that either.

<comment><USER13> >This is kind off strange for a non US resident to grasp.

I've worked for two non-technical companies as a software developer and one highly technical company, and interviewed at a few Silicon Valley companies.

The difference between the interview processes is staggering; my current job's interview was two hours of conversation, no code tests, just a general assessment of "do you know what you're doing" by the hiring manager and a couple other members of the team. The highly technical company had a code assessment then the in-person interviews had zero coding.

The SV companies must have a good reason for this, but golly the amount of coding in those interviews is nuts. I'm a process over code speed kind of coder, and I've failed every SV-level test because of it; my code comes from talking to non-technical users like medical researchers and study operations managers and tossing something together in Python or a cloud service that makes their lives easier. Needless to say, I don't go over algorithm fundamentals on a regular basis, and I generally fall out after the first or second interview.

It's especially odd that interviews are so intensely focused on those couple hours since I personally don't see any dev or any resource for that matter contributing in any meaningful way in so fast a time, or even within 90 days. I'm not sure how this problem could be solved with the limited time companies can dedicate to interviews, though; maybe rely more on portfolios?

<comment><USER14> > "The SV companies must have a good reason for this"

In fact, no, nobody has a good reason for it.

<comment><USER15> Go start a company. Hire people without doing any sort of coding interviews. Report back in a year.

The reality is that these are extremely desirable positions with a staggering number of applicants who _do not know how to code_. Not as in "I can't solve a dynamic programming problem without studying up on it", but as in literally don't know what a for loop is.

The process is far from perfect and the frustration is understandable, but it works well enough as a filter from these companies' point of view.

<comment><USER13> I imagine technical screening is necessary, but in-person coding assessments are nonsensical. It's not like programming is a spectator sport, so why are we tested on it live? I suppose the process is self-selecting, because I personally have given up responding to SV recruiters or interviewing with those companies.

</comment></comment></comment></comment></comment></comment><comment><USER16> Why do developers complain so much about hard interview questions that can be supposedly be gamed by studying to the test? Every high paying industry heavily engages in gatekeeping, because the number of people who want to make 400k/year is far larger than the number of 400k/year jobs available. The traditional forms of gatekeeping involve requiring that people have the right personal/familial connections, or have an elite school pedigree (finance, consulting, law), or that they spend several years and half a million dollars in post-college schooling (medicine).

The tech industry's preferred form of gatekeeping is asking people to do algorithmic puzzles, which is far tamer and less exclusionary than what other industries do. If you believe that the only thing standing between you and 400k/year is a few dozen hours of practicing leetcode, why are you whining about it instead of taking advantage of the situation to wildly enrich yourself with a fairly modest amount of effort?

<comment><USER2> I already make plenty of money in my current job and I'm fairly happy.

If I wanted to switch companies I'd be interviewed as if I were a recent college graduate.

Anyone experienced and good at this isn't going to have time for that because they're not going to jump through hoops to increase a 275k salary to a 300k one.

So what the interview selects for are people desperate enough to study hard enough to fool the interviewer.

And this is one reason the industry leans heavily toward young privileged male candidates and loves to reinvent the wheel every six months.

<comment><USER16> If you're experienced and good you can do way better than 300k at some companies. Also if you think spending a few hours practicing leetcode makes you desperate, wait till you hear the hoops people have to jump through to become a doctor or investment banker.

<comment><USER17> There are more interesting things to do. Coding your side project, contributing to the opensource project, learning a new programming language or even reading random GitHub project code and issue trackers. All of these activities help to improve your programming skills more than leetcode. If I am making max for my local area and I am not willing to relocate, why would I waste time for interview preparation?

<comment><USER18> I don't know what you are replying to the parent poster at all. If you don't want to interview for a job in a new location, or for more money... then the advice to study isn't meant for you. Like, sure some people find studying fun for it's own sake, but the purpose of studying _is for the interview which you just said you don 't even want to do_.

<comment><USER17> What I am saying that there are more interesting activities that would actually improve your skills. Leetscode and drilling interview questions are waste of time even if you are looking for a job. There are places that hire based on take-home interview, onsite pair programming or based on your portfolio/recommendation.

</comment></comment></comment></comment></comment></comment><comment><USER19> > However, whether or not a candidate answers a question correctly is not the > only source of signal during an interview. You can also evaluate their > process by, for example, observing how long it takes them to finish, how > clean their code is, and how much they struggle while finding a solution. > Our analysis shows that this second source of signal (process) is almost as > predictive as the first (correctness).

I seem to do OK when an interviewer:

\- asks me a question that sounds like a real problem, not a contrived one (although on occasion I'll have fun with a contrived puzzle if the interviewer has a sense of humor & makes the process light hearted)

\- doesn't push me down a path that requires me to implement a "simpler" solution I'd never consider (e.g. asking me a question that clearly wants an O(N) solution & then pushes me to try the O(2^n solution first)

\- talks like a person with a problem, and not as someone who clearly knows what they want & simply won't say it

\- doesn't try to "see how I think", because I code as much in my head as I do on a screen, meaning most of the code I throw into a text editor is the latest thought in a stream of random ideas until I get to one that works

\- doesn't constantly interrupt me

\- states their actual expectations, such as "I don't expect you to finish, what I'm really looking for is X"

<comment><USER20> As a frequent interviewer I will say that you are not alone in that: any interviewer that isn't following the above guidelines is unlikely to get a good signal from anyone.

To your final point about expectations: One question that we try to ask ourselves about any particular interview question is "how may bits of information are we getting out of this?" Trivia questions are usually less than 1 - you'll find out if they know some trivia, but you don't honestly really even care. The best questions are those that get you several bits over the course of the interview period, rather than a single yes/no answer at the end. Questions where there is no final end point or where we explicitly do not expect candidates to finish, are often the most effective; we can explore the pathways that are working, dive into side channels that seem interesting, and get data along the way rather than just a checkmark on some individually- useless problem.

<comment><USER19> I like the idea of extracting as many useful bits of information as possible!

</comment></comment></comment><comment><USER21> A while ago we had a job applicant who had travelled very far and long to reach us. (literally from the other side of the world.)

So, as a courtesy we figured, why not spend a few hours extra with this applicant in the programming test. We set up a laptop with a clean Ubuntu install, devised a programming test that was quite involved. Not algorithmic hard, just more complex than what can normally be done within a 20-minute whiteboard interview. We expected it to take at least 2-3 hours. Google/Stack overflow/etc access was allowed and encouraged. "Just act as like you would normally do when solving a problem."

We spent like 2x4 hours devising this problem, based on our codebase (cutting out something somewhat easily digestible and making it able to run standalone).

It took like one hour to get productive. Explaining the problem, setting up editors, compilers, etc.

We took turns, but most of the time someone in the interview team (of two) sat next to the guy. We did give him some alone time.

This is probably nothing new in terms of interviewing techniques, but to us it was such a revelation. We learned so much more about the applicant. Perhaps it worked well with this guy because he happened to be a bit more outgoing than our typical successful applicant. We'd never felt so confident about giving someone an offer before.

I'm really looking forward towards testing out this approach with local candidates to see if we can replicate this "data gathering success".

<comment><USER22> This can be problematic, you waste a lot of time setting up the PC / explaining the problem.

Last time I was involved with this interview style it always seemed to take an hour to get setup which meant a long interview of 3-4 hours particularly if a candidate went down the wrong path.

In the end we optimised for SOLID principles with a blackbox dll that had a function that slept for 2 seconds and a calling class that had mixed responsibilities (logging and calling the dll). We started folks off with a test or two and hoped they'd inject a mock to get rid of the delay and split logging off into a separate class.

I'm not saying it was a great test but you could do something within an hour or so then maybe spend half an hour talking through what techniques they'd use for a more complicated scenarios.

If you can afford the time then more realistic testing is great and I do think you should try.

</comment></comment><comment><USER23> There's a problem here. The only thing that Triplebyte can claim based on their data is that easier programming questions are more predictive of performance _among candidates who received an offer_. Since candidates who get offers are (in theory) different from candidates who don't get offers, we can't necessarily generalize from one population to the other.

There's also a question about how to mix question difficulty. Should you ask nothing but easy questions, or is it good to throw in a harder question or two to see how the candidate reacts to something they can't answer? I can see a good interviewer getting a lot of signal out of that, but in the hands of a bad interviewer it would not work well.

<comment><USER24> Yep. Lot's of "data analysis" without even defining what would be a "false positive" (someone who did well in an interview, got an offer but turned out to be a bad hire? how is TB exposed to that critical data exactly?)

<comment><USER25> It wasn't clear at all in the article what it even meant when it referred to "final outcome".

</comment></comment></comment><comment><USER26> I think that asking a candidate to perform a code review can be an effective method of evaluating quite a few desirable qualities. Can they understand someone else's code? Can they engage in constructive critical discussion? Are they able to effectively refactor something to make it better? Can they spot mistakes and do they have an opinion about how to avoid such mistakes?

<comment><USER27> Yeah that's great but what if they can't reverse a binary tree?

<comment><USER28> You spend a lot of work days reversing binary trees?

<comment><USER29> I think he was being facetious. :-)

</comment></comment></comment></comment><comment><USER30> I see one aspect of this trend of asking programming questions that require a lot of memorization: We have had for the past ~10-15 years people in the workforce (and thus acting as interviewers) who went through a public education system where heavy emphasis was placed on passing tests that required a lot of memorization.

I'd be interested to know how many of these interviewers actually think they're able to identify a solid candidate this way? Not to mention, are they even factoring in how many people don't test well but are otherwise superb software engineers?

Ultimately it seems like there is a soft element to interviewing that is being tossed out now, which is: do I think we can work with this guy/gal? Are they someone that can become part of our team on a personal level? Can they get good work done? Fizz Buzz can't tell you that. What can tell you that is experience. It's a hard-to-put-your-finger-on-it X-factor that I think companies think they can ignore.

<comment><USER31> So much this. I've been programming for 30+ years. My brain only has so much cache space and it dumps frequently. Asking me questions that clearly are testing my ability to hold large amounts of data in my meat computer isn't testing my ability to design and write computer programs.

Sure, I'd love to have a "mind palace" like Sherlock. Alas, I do not. I often admit this as early in the interview process as possible to avoid wasting time.

<comment><USER7> If anything, it's testing your ability to _mis_ design computer programs.

I hate working on software that was written to be read by an audience with perfect recall, because, as a human, I just don't have that. Give me code that assumes I have the memory of a goldfish, and can't keep track of anything that isn't right in front of my face.

I'm pretty sure that's what half of Dijkstra's papers were trying to say, weren't they?

<comment><USER31> Agreed and Knuth's Literate Programming. Our industry is caught in the event horizon of the black hole named, "cargo cults".

<comment><USER7> The black hole that gets me is a love of complicated things. It seems like, given a choice between two different things that are equally capable of solving the problem, 9 out of 10 hairless apes will pick the one that has more switches and knobs.

I haven't tried literate programming, but one of the things that entices me about it is that my instinct says that it's a vaccine against unnecessary complexity. If you can't express it comprehensibly in both English and code, there's likely an easier way to do it.

</comment></comment></comment></comment></comment><comment><USER32> I usually ask what's your strongest language; then ask questions about that programming language. f.e. if it is python:

> how would you explain the with statement to a junior developer ? then > increasingly difficult questions that go into the language runtime/concepts.

one other favourite question of mine is:

> Imagine, you got a standard website the serves data from a database. When a > customer types in the url into the browser bar what needs to happen until > the customer see the website.

> Go as deep as you can in the answering the question.

When you got an answer, you'll see frontend engineers explain more about the browser, while backend engineers talk more about the backend.

There was one very senior engineer, that actually talked about the ethernet layer, he talked for more than 15min. Most medior engineers are done in 5mins. ;)

<comment><USER33> There is so much that can happen between the browser making a request and receiving a response, that someone could completely gloss over the OSI model and still talk for a very long time.

I've toyed around with starting my answer with what happens when the physical return key on the keyboard is pressed. Could spend quite a bit of time on what happens before the browser even knows the return key was pressed :D

<comment><USER34> See [https://github.com/alex/what-happens-when](https://github.com/alex/what- happens-when)

<comment><USER32> hehe. however, if you look at this. do you think this is a server engineer or a client engineer ? :D

</comment></comment></comment></comment><comment><USER35> What is important to employers?

1\. A candidate who can solve puzzles but is not willing to do the dirty work with team, solving production issues, doing debugging, bug fixing with usual stuff. 2\. Another Candidate, who is willing to learn, is ready to work with team and do the dirty work.

I am on a hiring committee as a Tech Lead, and I always try to weed out 1.

Works great, we hire as interns and then assess them. Someone from Google who we hired full time, _was_ detrimental to team's morale, grunting, complaining about code, complaining about food and what not.

Another experienced smartass was self centered on his skills and didn't want to teach junior engineers anything or willing to admit he needs to update his skills. The moment he realized his skills have no values, started attending pointless conferences. Now his LinkedIn profile has "aware of block chain technology", "attended machine learning seminars".

I said not to interviewing at Google and FB because I don't have cycles to spend months on leetcode. Did I erred? Perhaps. But I am sure neither can provide me same work quality I execute in my current mid size company. I regret nothing :).

<comment><USER36> I didn't interview with Facebook or Google either, both because I didn't want to grind leetcode, and because those companies are pure evil.

<comment><USER35> For me evil is not what I consider. Its pure business, if other party don't need for my skillset, I aint gonna put my time to change my skillset to suit theirs.

</comment></comment></comment><comment><USER37> The best interviews I've been a part of have been a short cultural style interview followed by a (roughly) 2 hour paired-programming session on a relatively simply to-do application.

1\. The problem is pretty well understood (but does offer room for interpretation).

2\. Provides time to cover all key aspects (Frontend, Backend, Database, Networking, Debugging, Testing, Caching, etc) in at least some capacity. In particular, it shows you what areas the developers focus on.

3\. Provides a more relaxed/realist environment. It's also more accommodating to developers switching stacks - familiar with good programming patterns but not the specifics of stack (e.g. "Here's how I'd do [some specific task] in [other stack]. How do I do it here").

4\. It's clearly a throw away task so there's no concern about "interview labor". It can also be pre-prepped so you don't have to worry about jumping too far in.

5\. You can cut short with bad candidates and expand the problem for more complex candidates.

</comment><comment><USER38> >how clean their code is,

I got bit in the ass by this one as triplebyte itself. They asked me to make a tic tac toe game, and gave me iirc 30 minutes (less?) to do it. Except, it wasn't "build a tic tac to" game, first it was "draw a board to the console," "take user input from the console," etc a bunch of instructions in a convoluted path that perhaps another engineer would do when knowing from the outset that the goal was to build a tic tac toe game in 30 minutes, but not me.

So we'd get to a portion where I'd be writing a quick test on user input, or extrapolating something to a function, and the interviewer would say "don't worry about that, just worry about {getting the grid to print to console or whatever}."

Later on I got my feedback and they said they were disappointed with my user input tests and repeated, extractable code in the tic tac toe portion.

Triplebyte is trying to do good things in the interview space but I think they're still learning. All in all my interview with them was about as positive an experience as a harried and bad interview could be, from my perspective.

<comment><USER39> You failed your Triplebyte interview because you neglected an extremely important aspect of the job: communication. You made assumptions about the ask which turned out to be grossly out of tune with those of the interviewer. In the real world, engineers are often left holding the bag when other participants of the process leave out important details. It's our job to ask questions and establish the boundaries of each problem before diving into a solution.

<comment><USER38> As verbatim as I can remember our interaction:

Me: "I'm going to just put some user input checks here, no guarantees they'll actually input X or O, yea?"

Interviewer: "Oh, don't worry about that, assume for now that you'll get X on X's turn, O on O's turn."

Later:

Me: "Normally I'd extrapolate this to a function, so let me just - "

Interviewer: "For now, just focus on getting the program to response to the next user input."

Me: "Ok... well the fastest way to do that right now is just copy paste this code down here."

Interviewer: "That's fine."

But, it turns out, it was not fine.

I felt I was bamboozled.

<comment><USER40> Triplebyte sounds like a bad girlfriend.

When she says "don't worry about it" or "that's fine", you can't take her word for it.

You need to be psychic and understand her needs without her saying anything.

<comment><USER41> Please don't do this here.

</comment></comment></comment></comment></comment><comment><USER42> At my consultancy we recently streamlined our interview process:

1\. Phone screen which takes 15 or 20 minutes. 2\. The candidate fills out an essay, including showing us some code they're proud of. 3\. If the essay ticks the boxes we conduct a 1 hour on site interview. We use the same a set of questions for every candidate, so the investment is easy to manage, and our team has a shared set of expectations on what is good or bad. 4\. If the interview goes well, we give them a take home assignment. Takes between 2 and 6 hours, depending on how experienced the candidate is. Problem is in C and/or Python (or both) 5\. We wrap up with a 2 to 3 hour onsite interview. We walk through the assignment and have a deeper conversation about culture and fit.

The results have been positive for us: we've made some great hires and weeded out some candidates who weren't a good fit.

We've also been able to scale it down to the process we use for interns.

The 1 hour interview has some typical programming interview questions, but we wrap them into a real-world example. The goal isn't to prove they know how to program, but more about allowing them to show us how they think/work out a problem.

<comment><USER43> If you want to do it for junior developers, fine, but as a senior dev, I'm not doing any take home assignment. It's an immediate pass from me. There is very little possibility that you can glean anything from my code other than picking apart it for code review bullshit. I'm not going to waste 2 to 6 hours on this - done it before, never again.

<comment><USER44> And this is why I built a site to allow people to search for jobs based on interview type. Although I haven't done any proper marketing yet, still a work in progress ([https://softwarejobs.xyz](https://softwarejobs.xyz))

Everybody is different.

</comment></comment></comment><comment><USER45> My group likes starting with easy questions and ramping up the difficulty, not to eliminate people with wrong answers, but for two other reasons: first, to see whether people understand the questions and whether they try to make up answers, or ask questions, or say "I don't know". Second, to see what the limits & boundaries of their experience is. We know that people don't know everything, and we measure more for potential than for knowledge, but it's still useful to understand someone's experience and exposure level.

More important than question difficulty to me is attitude, and I'd love to see whether attitude is measurable and how it compares to later performance, but curiosity and optimism and communication really do go further than right or wrong on math and engineer questions for me. That point might even be tired already, I know people say it all the time, but I'm going to keep saying it because we still have blog posts on question difficulty, when easy vs hard engineering questions are pretty low on my list of what matters when I'm hiring.

<comment><USER46> At the end of the day, the question itself is a tool, not the goal. When interviewing, I look at how the person approaches a problem and works through it. I don't really care if they get to the end or not.

People who complain about memorization and difficulty are kinda missing the point. Just like learning math at school isn't really about knowing how to do trig, but being able to think logically and do problem solving.

</comment></comment><comment><USER47> A few jobs back I was tasked with hiring new developers to bolster a thin front-end team. The job was very CSS/JavaScript heavy, so I asked questions that were pertinent to what the candidate would be doing if hired. Of the five candidates, only one answered all the questions perfectly, and he turned out to be the biggest bust for us.

The other candidates, after answering some of the harder questions incorrectly, seemed very upset with themselves. They knew they were cracking a bit under pressure, but actually showed that they knew the answers when we chatted further. I hired 3/4 of those people because of how well I felt they'd do given the opportunity. All three became leads within a year and a half.

I think personality has a lot to do with outcomes. If you are someone who shows they are hungry to learn and knows how to improve their skills, I will never dismiss you for screwing up a few coding questions.

<comment><USER48> Personality has a huge amount to do with outcomes. Especially in teams, where the best teams have a mix of complementary talents.

If you're hiring for Generic Developer Skills you're going to get generic developers - and much less development than a more flexible approach would give you.

</comment></comment><comment><USER49> One interview of mine asked something i didn't know yet I said there's no doubt I figured it out via Google. The interview pretty much ended there and I'm glad it did! Any place or interviewer that says you shouldn't use Google of OverFlow to get your work done is no place I want to work for.

<comment><USER50> What, you couldn't derive a solution in a vacuum from first principles? I bet you think referring to API documentation isn't cheating, too!

<comment><USER49> Hmmm hard decipher your comment.

But, that interviewer is the type of developer I don't want to work with. A know it all who only acts like he knows it all and uses Google secretly. Who puts others down to make themselves look good. AKA an insecure P word I want nothing to do with.

<comment><USER50> > Hmmm hard decipher your comment.

I'm being facetious and agree with your sentiment.

</comment></comment></comment></comment><comment><USER51> I got my yearly review recently and I got very good feedback. At the same time I have been doing Leetcode at home for fun, starting with easy problems, and I get my ass handed to me.

I find it hard to reconcile these two experiences. How can I thrive at a top tech company while failing to solve an 'easy' coding challenge. It makes me concerned about what would be of me if I had to look for a new job now.

<comment><USER52> Thats exactly what your top tech company wants. For you not to look for a another job. For you not to try to get a higher salary.

</comment></comment><comment><USER53> This is an interesting post given that up until recently TripleByte's facebook/Twitter ads were asking very nonrealworld programming questions: [https://twitter.com/minimaxir/status/1054596563585052673](https://twitter.com/minimaxir/status/1054596563585052673)

Now, the ads ask simpler things like floating point precision and function variable scoping ([https://www.facebook.com/triplebyte/ads/?ref=page_internal](https://www.facebook.com/triplebyte/ads/?ref=page_internal) ); legit problems, but not sure if they are an indicator of how good a developer they'd be in the real world working on a CRUD app.

<comment><USER54> How is that nonrealworld? Dealing with things like endianness is quite normal for a low level programmer.

<comment><USER36> You nailed it. For someone who needs to determine the endianness of a machine on the fly, the question is trivial to solve; for everyone else, it's trivial in the sense of "unimportant."

<comment><USER54> Their pre-screening quiz uses a bunch of questions on various technologies to ensure that some subset of the questions tests you on technologies you're familiar with. Which is good, since not everyone is a web engineer.

</comment></comment></comment></comment><comment><USER55> The bigger problem is the inconsistency amongst interviewers when judging candidates. All these articles from TripleByte and Gayle (who've built business on the flaws of interviewing) focus on the questions instead. Doesn't matter how hard the question is if the interviewer knows what they're looking for, are experienced enough, show no nepotism and have good communication skills.

My worst interview ever was with Facebook when a non-native, new college grad gave me a Leetcode hard problem in half-broken english and went back to his work without even looking up or walking with me through the problem.

</comment><comment><USER56> If they want something that mimics the common, everyday..

Give the candidate a project with 300,000 loc, tell them to make the most local change possible that fixes the reported bug. Update the tests to reflect the new logic.

Bonus: discuss architectural changes that would have resolved the bug and/or improved performance.

<comment><USER57> They do something like this with a couple thousand lines of code and failing tests.

</comment></comment><comment><USER58> ,,Hard questions do filter out bad engineers, but they also filter out good engineers (that is, they have a high false-negative rate). Easy questions, in contrast, produce fewer false-negatives but more false-positives''

The philosophy at Google is that it's better to filter out 3 good engineers than to let in a bad one. The consequence of this is that it's really hard to get kicked out of Google.

The other part (whether it's more important to work on long easier questions to see how the candidate works on a large code base) is orthogonal reasoning, and that part may be true, depending on what type of engineers somebody is looking for.

<comment><USER59> > The philosophy at Google is that it's better to filter out 3 good engineers > than to let in a bad one.

Just fire the bad ones. You're going to get bad ones anyway. At larger companies you might never notice whether someone is good or bad.

From the way they structure their interviews, it seems like they'll still get plenty of bad ones - it's just they'll get bad ones that are great at algorithms, with unknown skill at everything else (like the actual work done).

<comment><USER60> >Just fire the bad ones

Who and what decides what a bad software engineer is (note this is very different than deciding who is a good)? Is it a manager? Managers are single people so shouldn't be the sole deciding point. So you need to gather feedback, aggregate it and decide what constitutes bad. Then have that feedback reviewed by independent people to ensure that no one is being unfair. That takes time and bureaucracy.

</comment></comment></comment><comment><USER61> Another solution to this problem is contract to hire. I realize this is kicking the can down the road to the contracting firm but hear me out: that's the business the contracting firm is in. They can get really good at their hiring practice since that's their core business. That's not our core business. We've been doing this for the past two years and it's worked out great. Now you can see how well people do the actual job and if you're not satisfied, which happens from time to time, just get someone else.

<comment><USER36> Why would I leave my W2 job for your contract to hire position? All I can see here is a company that doesn't want to fire people who aren't working out, possibly to save on benefits and unemployment insurance premiums. And why do you have so many people not working out that you have to do this? None of this inspires any confidence.

<comment><USER62> Yeah, in my experience asking the basic question of why I would even entertain moving from my W2 job to a contract position where I assume 100% of the risk is met with dead air on the other end, followed by a hasty assurance that we don't really have to go the contract route.

It gives me the feeling that I will be working with people who either don't know their own value and so are willing to agree to this kind of deal, or are so poor performing they don't have the leverage to insist on a regular employment contract.

</comment></comment></comment><comment><USER63> The thing that bothers me about these questions is that there isn't a shortage of practical questions to ask that will test if the candidate can truely contribute to the real problems you're trying to solve.

"Right now our roboticists use a hacked together QT based GUI to manage customer robot fleet data. It takes 1-10 minutes to load on a slow network and is hard to add more features to. I know you have far less information than you'd want but walk through your thought process for how you'd replace this system over the next 12 months. We can make assumptions."

And then the next 15 mins can be an organic conversation about the problem space. You can direct the conversation into corners most relevant to their potential role: "you mentioned using web tech because we discussed how all usage is across the internet. Can you talk about the merits of Http vs. websocket?" "How would you ensure that we don't accidentally take every single customer offline if we centralised our data store?" "What kinds of UI technology would lend itself to robot mapping? Can we just use Google Maps?"

If you really need to dig deeper into technical prowess, find something relevant in your conversation and dig deep into it. "We talked about saving changes to floor layout. Can you whiteboard/laptop how you might implement undo/redo for floor elements?"

</comment><comment><USER64> Instead of doing this, read, then follow [https://sockpuppet.org/blog/2015/03/06/the-hiring- post/](https://sockpuppet.org/blog/2015/03/06/the-hiring-post/)

</comment><comment><USER65> This whole rock star/ninja interview process is ridiculous. They need to paint a house but they want to hire Michelangelo.

</comment><comment><USER66> Having recently gone through some interviewing (for machine learning research), a very cynical but overlooked aspect (in this thread) is the following:

After receiving an offer from a big tech company, the interviewing process has already completely turned me off from the idea of working there.

Now despite this being a dream position for many and me having no alternative but to take it currently, the smug interviewers have already gotten me in the corporate mind-set: No matter, the reputation and salary, treat it like any other job, do not bother being loyal - they will not be.

So the terrible interview process has at least the advantage of reminding future employees what they are signing up for.

I wonder what kind of psychological filtering is at play. Do employees feel loyalty after an interview process that is best described as hazing? Are they projecting the humiliation they experienced when interviewing future employees? That's always been my impression.

<comment><USER67> Many interview processes generally paint the company in a grimmer picture than it actually is. So I would do additional research than turning down an offer because of a bad interviewing experience. Of course, you are completely in your right to do so, but it might not be the most optimal behavior.

Usually, interviewers are just bad at interviewing and aren't really aware of what proper behavior is. It actually isn't easy, nor is it a desirable activity nor do you really get points for it in your performance review.

<comment><USER66> Oh I won't turn down the offer, as said, I have no choice and it would be an irrational career move to do so.

The ultra-negative interviewing experience just completely changed the mindset with which I will go into that job. This then got me thinking about the hidden cost of a negative interviewing process.

<comment><USER67> Sure, but in general all of the big company interviewing processes will convey the similar amounts of dread these days. They are so worried about false positives and have so little concern about false negatives that they push many candidates away simply by not focusing enough on the "why should I work here" part of the interview.

</comment></comment></comment></comment><comment><USER68> Specific examples of what classifies as a "hard" or "easy" interview question would be very helpful to have reference points and assess one's own interview process.

<comment><USER69> Easy - reverse a string, determine if a string is a palindrome, reverse the digits of an integer, determine if one string is an anagram of another.

Hard - implement a subset of regex match in optimal time+space, find the operations required to turn 1 word into another word given a list of transitory words, find the median of 2 sorted arrays in optimal time, find the next permuted value.

<comment><USER70> That's more like small/large than easy/hard.

<comment><USER69> I'd argue finding the next permutation is actually very difficult if you don't know the standard algorithm.

</comment></comment></comment></comment><comment><USER71> You realize that this approach is flawed the moment there are blogposts / books ( e.g. Cracking the coding interview ) on how to crack it. The how to crack 'x' becomes a field altogether ( coaching / youtube videos / blogs / books etc ).

Also, platforms like hackerrank are adding fuel to fire. I read the CEO write somewhere that he wished the below "were taught in schools :

1) Communicating complex ideas with clarity 2) Systems thinking 3) Grunt work tasks 4) Boundaryless thinking 5) Self-awareness / EQ"

Please note almost all of these are not evaluated on their platform (they profit from coding tests) or during interviews and almost all are soft / intangible skills (skills which are not immediately obvious about the candidate during a typical programming interview). [ Side note : some could say that coding tests are the problem they have chosen to solve - in which case, why are they worried about these skills ? Are the companies seeing coders crack the tests on their platform, while not performing well on the above skills post hiring ? We could only speculate. ]

All good work is done by teams, and to be effective in a team requires a lot of intangibles which aren't even assessed in a typical interview.

A better approach could be from this article: [https://leerob.io/blog/technical-recruiting-is- broken/](https://leerob.io/blog/technical-recruiting-is-broken/)

Or : A couple of weeks of work with a task being assigned and the mentor or interviewer looking at how the candidate is approaching the problem and whether he is able to solve the problem within the time constraints (an easy task shouldn't take long, and a hard problem shouldn't be short circuited to give a sub-optimal solution.) and other such observable traits can be evaluated.

My two cents!

EDIT : Poor wording above (i.e., couple of weeks). A task should be assigned and evaluated post a time (which is ideal for task completion as per the interviewer). No constant interaction with the candidate and spending loads of time with that candidate - that isn't scalable when the demand-supply equation is imbalanced already.

EDIT 2 : The idea above is not about spending weeks for recruitment. The idea was about being practical about the kind of questions / tasks that are given during interview (example : code a feature or fix an issue we have, as another user has suggested well in the comments). Took me a while to realize we have missed the point I tried to convey for the logistics of how it should be done.

<comment><USER72> Who has time to spend a few weeks off from their full time job just to interview?

<comment><USER73> I agree with the sentiment, but I noticed recently that TaxJar (I considered applying there at one point) mandates a trial period along these lines. From [https://life.taxjar.com/distributed-team-hiring- process/](https://life.taxjar.com/distributed-team-hiring-process/), "... We hope a candidate is able to spend somewhere between 80 and 160 hours working with TaxJar during their trial. ..."

Presumably there are some people who are willing to do a trial, either concurrent with their existing employment, or on the chance that they will be hired after the trial.

<comment><USER71> This is good to know, typically it should be in concurrent with the existing employment. If we plan to switch companies, we commit time apart from existing office work for interview prep anyway. This interview prep time is what we would end up committing to work on their tasks etc. I think working on a project or task from the prospective employer would be much fun than preparing / memorizing interview questions etc. What do you think ?

<comment><USER36> Most companies won't allow such a thing, especially if the other company is a competitor. In any case, doing so would likely require me to take 1-4 weeks off from my regular job. Again, why would I want to blow all my vacation time working, just for the possibility of another job?

<comment><USER71> Okay my experience was, i didn't have to take leaves from my regular work. I had to simply dedicate a couple of hours post office time, to work on the task (the couple of weeks was given owing to time constraints due to my regular work). Post office hours, i would have to anyway commit to interview prep if i had decided to switch anyway is what i meant to say.

In any case, this is not the only method or approach that is the better alternative :) Maybe it will work for some and not for others. But evaluating alternatives is a good exercise. Currently we are stuck with this method because we do not have a better solution to the hiring problem.

</comment></comment></comment></comment></comment></comment><comment><USER74> I really agree. At companies I've worked at for the last 20 years (at least) the approach has been to ask pretty simple questions; people who struggle on those we don't want; people who don't struggle, even when they make silly mistakes (due to feeling interview pressure) are good.

Questions about the standard library of the programming language in question are good. Questions about the dusty corners of said library: bad.

And don't ask about floating point: most likely the candidate won't really know more than the usual things; anybody who really _does_ understand them will probably give answers over the interviewer's head :-).

</comment><comment><USER75> I like some coding in interviews. I try to make the tasks fairly real-world though. I am a UI engineer so my tasks typically fall into one of two buckets:

\- A take home (2-3 hours) task for retrieving tabular data from an API and displaying it. Here I'm looking for general framework chops, readability, some design sense.

\- An in-person (~45m) not-quite-pair programming task, with a real computer, tools, editor etc., for doing a typical UI operation, e.g. truncating text. Starts simple and gets more complex as time allows: make a function to truncate text to x chars; now add an ellipsis only if truncation occurred; now make sure not to truncate in the middle of a word, etc.

</comment><comment><USER76> I've never been giving given a practical programming test. Nearly all are useless algorithms. Create a function that takes a integer greater than 0. Build an array equal in size to the argument. Fill the array with numbers that when summed together equal zero.

It was worded far worse than that. What exactly is that telling you about the engineer?

On the flip side I'm asked to code full fledged applications but not to spend too much time on them... okay...

Another time I was asked to code a luhn algorithm. Oh and do it while a room of people watches you on giant screen cause that's what your day to day job will look like... I failed miserably and still got the job. What?!?!?

<comment><USER20> FWIW, I regularly pass interview candidates who "fail" at certain questions. The point of those questions isn't to complete it perfectly (and sometimes we get less information from those, honestly), it's to see how you go about getting there, _how_ you fail, and how you deal with that failure.

For instance, one of the problems I frequently ask has a structure that really encourages people to try inventing heuristics to solve the problem, even though ultimately all of those heuristics fail. Seeing how people react to "but what if your input looks like this?" questions is often very enlightening- can they rethink their approach? Do they just keep glomming on more special cases? Can they deal with someone pointing out that sort of flaw?

</comment></comment><comment><USER77> Most programming interviews are a waste of time and energy for everyone involved and the results are, for the most part, a complete facade. Asking puzzle questions gives interviewers the feeling that they're really testing hard for top talent, when in reality they're just demonstrating their lack of interviewing skills.

I recently interviewed managers at two different technology companies, both nationally known, for a report I am working on. Most managers admitted their technical interviews were flawed, but didn't know any other way to assess skills. They also admitted that a significant number of people they recruit refuse to even take the technical challenge and end up working elsewhere.

In interviewing a couple dozen engineers, I found most just don't want to waste their evenings and weekends on a technical puzzle for a job, especially when there are a lot of companies out there who don't bother with them, so they end up searching for companies that don't waste their time with technical challenges.

Another funny thing I discovered during my research is that just under half of the employees at both companies I've interviewed so far were not able to successfully complete their own technical challenges.

Another problem with technical challenges is that often times the interviewer knows less about the topic than the interviewee. I recently went through the interview process with a local technology company who uses Elixir and Go (both of which I know). During the onsite interview, the interviewer kept saying things like, "Don't forget to..." or "You forgot..." I kept explaining that I didn't need to do as he was suggesting. In the end, my code worked, my tests worked, and I passed the interview. In spite of this, I was rejected because the interviewer, "Wasn't feeling it."

I still have a lot of research to do, but I haven't found anything, so far, that suggests that technical interviews predictably result in top-talent getting hired. It seems to be the same crapshoot interviewing people without using technical challenges is, because in the end, most people decide within the first couple of minutes if they like someone and hire based on that, regardless of the rest of the interview process.

</comment><comment><USER78> I just finished reading Bill Kilday's Never Lost Again. It was amazing that four engineers produced the ground-breaking Google Maps in 16 months. Algorithm/Math questions used to be really effective at finding engineerings like those four engineers. There was a reason that Microsoft resorted to algorithm puzzles in its hay days as well.

It's just unfortunate that there's so much prepping materials online nowadays that the programming puzzles have become ineffective. It gets worse as many interviewers were not good enough to ask follow-up questions. For instance, addition with big integers is a pretty easy interview question, right? But if a candidate can go as deep as this article: [https://bearssl.org/bigint.html](https://bearssl.org/bigint.html), I can be pretty confident that the candidate is really really good.

That said, I personally don't find it necessary to join the rat race. Instead, I'd suggest engineers just take time to thoroughly study just one book on algorithm designs. In fact, an introductory book, such as Kleinberg's Algorithm Design or Udi Manber's Introduction to Algorithms, will be good enough. It may not get you into Google, but it will likely get you into another damn good company. The best part of this approach is that passing interview is really just the byproduct of you trying to become a better engineer.

</comment><comment><USER79> Another clarification: this data only applies to general programming exercises and not domain-specific knowledge. For example, if your role requires bio- informatics knowledge (or ML or AI etc.), then by all means ask about bio- informatics—even if it means asking harder questions. (We do, however, recommend keeping the content of general programming exercises as vanilla as possible, so that you don't filter out someone because they lack mastery of a subject that you don't actually intend to measure.)

</comment><comment><USER80> On the interview its build a tic tac toe but when you get to the real job they give you a week to fix a line of code. This industry is fucked up.

</comment><comment><USER81> If you are asking interview questions that has one beautiful precise answer, you are doing it wrong. Good interview question should start with something very simple that even very beginner can think and answer, then gradually add complexity and constraints little by little.

Example:

1\. Write function that multiplies two integers.

2\. What if these numbers were real numbers but computer can only operate on integers? How do we use same number of bytes as ints to hold a real number?

3\. What if I wanted infinite precision? What would be run time of your algorithm and storage complexity? (don't insist that candidate must hit the known optimal).

4\. Can I have complex numbers as well?

5\. Imagine complex numbers not only has "i" but also "j" and "k". How do we handle this?

It is astonishing how many candidates won't be able to move past #2.

The key is to look at how candidate approaches handling complexity, create representations and use it to craft clean solutions. Whether they eventually arrive at known optimal/great answers is unimportant.

<comment><USER82> I couldn't get past #2. Does that mean I'm a bad programmer or just missing the experience for the particular field?

My point is that questions should be tailored to the work the programmer is expected to perform.

<comment><USER81> Above example is not a test of your programming skills. May be you are great NodeJS ninja who doesn't need to care about any of this. I think there is two types of hiring that often happens (1) for a specific project, specific task (2) long term member of the mission that company has.

For #2, I prefer to hire people who are generalist problem solvers. They need very strong coding skills but more than that they need to be able to operate in new domains easily and adapt. If I was running a startup, I would need them to work on MySQL database one day and react-native stuff other day and perhaps also pick up some of deep learning practitioner skills two months down the line. Above example question doesn't expect candidate to be familiar with floating point representation but it is interesting to see what they _might_ have thought if they were the ones doing it. Assuming everyone works with numbers all the time, people are hopefully familiar with basic issues of precision, rounding etc.

</comment></comment></comment><comment><USER70> So they saw a correlation among people they hired and made a conclusion about people who applied, with no reject inference?

</comment><comment><USER18> >Harder questions ultimately filter out too many qualified candidates to be optimal.

This was the key sentence if anyone missed it. "Optimal" for whom, exactly? Not for FAANG certainly. They don't need to worry about filtering too many candidates out, because they have a nearly infinite pool of applicants, and infinite money to conduct a search.

They can ask as difficult questions as they want, because they can pass hundreds and thousands of qualified candidates, and still have plenty more where that came from.

Edit: If you consider "optimal" to be the expected cost of a hire compared to the expected profit, it is fully plausible that if your margins are big enough, asking hard questions is the most effective way to ensure low false- positives. But as everyone knows, comments on articles about interviews are never about the economics, it's only about human ego of feeling rejected.

</comment><comment><USER83> Is it a surprise to anybody? My favourite example of interviews going bad is a tech company in SF asking to find cross currency arbitrage loops with all known currencies over the phone in the first round hiring data engineers for the ETL team. I am glad I did not pass that round.

</comment><comment><USER84> I've seen this borne out in practice after administering dozens of phone screens and in-person interviews over the years. I started off with questions that were a little too hard and couldn't get a good read of the candidate unless they happened to have already done the problem on some interview prep site.

Switching to more practical, simpler problems allowed me to really observe how they work and solve a coding problem. As the article said, I was also able to add requirements or features to the problem which let me see how the candidate adapts to changing requirements, or refactors their own solution to handle a new edge case. Simpler is generally better if you are timeboxed to 45 minutes.

</comment><comment><USER85> I changed careers from another technical non programming career to web development. I got a handful of interviews but I never felt like I was showing them what I could do... learn.

Until one finally gave me a fairly straightforward homework style project. It was probably more apt for a noob and I threw myself into it and submitted my work and explained what it over the phone. I was in the office the next day and we talked about it and I had an offer.

The homework style interviews are understandably controversial, but at least I got to show my work, me doing my work, my thought process, outside of a few moments at a keyboard.

</comment><comment><USER86> I always thought that the primary quality measured by this popular interviewing style is "being-like-your-interviewer" which leads naturally to the avalanche effect due to which the practice becomes even more popular within the company and simply common sense after a while. The interesting part, to me, is how exactly similar the methodologies should be for companies different by size and domain. Even if you assume that some flavour of technical interview worked for Microfaceboogle, it might as well kill a copycat company (or may be completely irrelevant to its success or failure).

</comment><comment><USER87> I wonder if hard questions, especially hard algorithm questions, has high false positive rate as well. Being good at solving those questions simply requires you to do tons of practice on leetcode, and those who are willing to spend tons of time are probably those who find it difficult to get jobs (or new grads). I don't think the skills for solving hard algorithm questions correlate well with the actual performance at work. Especially if someone is a new grad, they could be good at thise questions but don't have a clue about how real software systems are built.

</comment><comment><USER88> Hold on, who is qualifying the outcome?

At one of my jobs I nailed it as the top candidate, by far, of the 79 people they interviewed in person. The interviewers were looking for competence, freedom from frameworks, experience and so forth. I have in this line of work more than 20 years and do it as a hobby so I nailed the interview.

At the job though I worked with a bunch of fresh juniors who only know how to write code the one way they learned in school. According to them I am shitty developer because I didn't write code in the one way they understand.

Who qualifies the outcome?

</comment><comment><USER89> This article makes a lot of good points. After going through the "implement a red-black tree on this whiteboard" experience as a more junior dev, I always promised myself I would never use this kind of stupid questions to hire.

Now, 13 years later, I mostly rely on "homework" type exercises. I think they address most of the issues. They are more "real world", no time pressure, etc. However, even those now are being heavily criticized. What's left to be used?

<comment><USER90> There is a problem with homework, candidate spends couple of hours and you spend couple of minutes to assess candidates. It's not fair. Google doesn't do that, Netflix doesn't do that. Why anyone would believe random startup and spend their weekend unless you're desperate looking for any job?

<comment><USER89> It is both true and false. Yes, the candidate may spend 1 hour (I don't give difficult homework exercises) while it takes me 15 to 20 minutes to review the submission, but I have to do it for maybe 5 candidates.

I totally understand the main criticism for homework, it takes time, the company may never call you back after you poured 2 hours into their stupid exercise. But it is an attempt at fixing all other alternatives: * the onsite whiteboarding is BS * using open source contribution is totally unfair to candidates who don't participate * the "contract for 1 month and then maybe we'll hire you" is also total BS in my book. Who would leave a FT job with benefits for a contract that may end?

I'm not sure I can think of any alternative that has 0 drawbacks.

<comment><USER90> > the candidate may spend 1 hour

> after you poured 2 hours

and at the end it turns out to be 5-10 hours. Companies say "you should be able to finish it in 1-2 hours", but it's almost never true.

I think alternative is to have 1 hour coding session on a good problem. Most problems are complicated, but there could be something else. I was once asked by Uber to implement timer in JavaScript that will update DOM, also create APIs to stop/start/pause. This kind of a challenge doesn't involve any algorithm, but it shows your ability to code.

It should be enough to bring you onsite.

But I open source should work. If you contributed to Linux Kernel and it was accepted - onsite no questions asked. If you have github repo of 1000+ stars - onsite no questions asked. If you have already passed Google tech screen and was invited onsite - invite onsite no questions asked.

</comment></comment></comment></comment><comment><USER91> Given the 5th & 6th paragraphs about "correctness signal" and "process signal" it seems like the obvious solution would be to do both, i.e. ask questions that are easy alongside those that are hard. The easy ones are "process questions" and the hard ones are "correctness questions." Or maybe you have a range of difficulty from easy to hard, and each question has a number attached to it from 0 to 1 that reflects its intended use.

</comment><comment><USER92> One issue I've noticed is that the elite companies are trend setters, and their interview methods are being copied by non elite players.

Perhaps Facebook needs and wants engineers that can bust out A* on the spot, but I doubt Nordstrom or Starbucks needs that level of talent.

This has changed the field where now to even get an average job at an average company you need to study at the new normal, whose interview ideas were designed to look for the top 1% of the field.

</comment><comment><USER93> I totally agree with this. You shouldn't ask a question where some amount of luck of figuring out the problem is necessary. Instead ask something practical and relevant to your domain. Also, leave room for exploration if they sail through it and recovery for someone that goes down a bad path.

[https://github.com/spullara/interviewcode](https://github.com/spullara/interviewcode)

</comment><comment><USER94> I'd agree but on the condition interviewer actually know ehat they are doing.

As I gained more experience (300 interviews and counting, baby) I realized that I pick up more on candidates skills that are not directly related to their performance on particular question. At this point the question itself is just a conversation starter and so it's better if it's simpler and more broad because it leaves lots more avenues for the conversation to go.

</comment><comment><USER95> I ask what they've built, and how it was done.

This shows you way more than what a technical question can answer.

There are edge cases to this. Expertise positions specifically.

</comment><comment><USER96> I'm like 0/15 on interviews in the last 2 months. more than half of them were with foreigners who i could barley understand.

</comment><comment><USER97> Another use of these kind of interviews is age discrimination to filter out of older candidates who don't have time or motivation to prepare for this BS. This is just another fad.. In late 90s there was fad for MSFT style puzzle interviews..then it went out of fashion.. this will also go out of trend.. just wait for couple of more years.

</comment><comment><USER98> "Easier interview questions are also less stressful, which is an important advantage. Stress causes candidates to under-perform. But, on the other hand, when candidates are more comfortable, they perform their true best, which actually makes interviews more predictive."

I cannot believe that _any_ interview situation is comfortable.

</comment><comment><USER99> Interviews are like taking the SAT or ACT. For better or worse they're highly standardized and you just need to prepare. There are a bunch of different prep service like InterviewCake.com, Pramp.com, and PracticeCodingInterview.com that are good if you feel like you need more than just leetcode grinding.

</comment><comment><USER100> Where I earn my money is thinking of a solution to a hard problem while not actually at work. It probably only comes up a few times in a year.

Questions you can answer immediately simply don't test if you can do this. I have no idea how you would test to see if you can do this.

</comment><comment><USER54> Isn't this just saying that easier questions are more correlated with overall interview performance? That might be a good thing (a sign of consistency), but it could just as easily be a bad thing (i.e. it doesn't provide unique signal).

</comment><comment><USER101> I took a technical interview today over the phone, and you know what, they were pretty fair with their questions. I don't know what kind of competition I'm up against for the position, but very fair questions.

</comment><comment><USER102> I've worked for several companies who made up tests that nobody on staff could pass, including the authors. But the test did imply that the existing staff was superhuman, since no applicant could pass.

</comment><comment><USER103> Interesting post! I appreciate this being shared.

Would Triplebyte mind sharing the data? I'd love if the numbers could speak for themselves, rather than having to rely on an interpretation of the numbers.

</comment><comment><USER104> It seems like this article would be stronger with some examples. What are some examples of good and overly-difficult questions, according to TripleByte?

</comment><comment><USER105> I'd love for this article to provide examples of what they consider "hard" and "easy".

</comment><comment><USER106> Heads up I think there's a typo here: "questions that carrying".

Interesting piece!

</comment><comment><USER90> Let it stay here [https://twitter.com/dhh/status/834146806594433025?lang=en](https://twitter.com/dhh/status/834146806594433025?lang=en)

"Hello, my name is David. I would fail to write bubble sort on a whiteboard"

</comment><comment><USER107> That's because professional outcomes for programmers aren't related to aptitude. Nobody ever got promoted for writing good code. (But the reverse is very common.)

</comment><comment><USER108> What alternative would you suggest?

</comment><comment><USER109> Because they're designed that way. It all comes down to:

* Rejecting far more candidates than you need to -- so you can feel like you're hiring "the top 1 percent"

* Giving yourself the feeling that you have an objective hiring process (when really you don't)

* Making your own team members feel like they're super brilliant and special when really they're not

That's what the modern hiring process is designed to do. And in fact it works quite well, to serve this purpose.

<comment><USER110> Don't forget that they introduce market friction to reduce developer turnover and reduce salaries.

I suspect one of the reasons Google is so open about their process and the need to study is so that everyone follows suit. Thereby forcing people to take days off and do homework for even the most mediocre of positions, causing the switching costs of interviewing anywhere to become higher.

<comment><USER111> Interesting idea. I got an email from Facebook with invitations for the interview. Replied: "I can do it but i'm Letting you know, I will not spend a minute preparing for it. So, we can do it today".

I never heard from this person again, which brings a question - is probability of passing it without putting personal time is so low it makes no point in even trying?

<comment><USER112> More likely the phrasing of your response provided signals that they shouldn't proceed.

</comment></comment></comment></comment><comment><USER113> I don't give questions like this primarily because:

\- Workers are going to be around for 15 months or less and they have domain expertise on 1 stack already and I don't need to screen for how they would hypothetically function across all stacks

\- Worker's process and resource finding skills are more indicative of the time they will spend on a task

\- Worker's process includes collaborate use of version control and code reviews, if they pass the screening but can't really integrate on these things then thats what will get them booted from the team

It isn't always more expensive to have a not great developer. Look in your organization and see if what I experience is true for you, and you'll save everyone a lot of time.

</comment><comment><USER114> An example and actual numbers would be a good addition to this article.

</comment><comment><USER115> Why do we talk about this every week on HN?

I'm pretty sure it's either once a week or once every two weeks.</comment>


The Curious Case of Convexity Confusion - <AUTHOR> https://googleprojectzero.blogspot.com/2019/02/the-curious-case-of-convexity-confusion.html <comment><USER2> this is extremely cool and well explained. the fact that the exploit comes literally in the form of shapes that can be drawn and visualized is very neat.</comment>


Teaching Ruby to High School Girls - <AUTHOR> http://teachingkids.railsbridge.org/2009/08/15/teaching-ruby-to-high-school-girls.html

<comment><USER2> Here's a crazy idea. Let's teach Ruby in intro programming classes instead of C, so as not to frighten away 90% of the population with pointer dereferencing and segfaults. You should teach C eventually to make them appreciate high level languages, but just not right away.

<comment><USER3> We teach Shakespeare and Calculus in high school, though those are certainly not the easiest choices in their respective fields. For an introductory programming class, I agree that Python is a great choice. However for seniors in a computer science class, C should be a fairly important part of the curriculum. There are some fundamentals of computer science that are best grasped through experimentation with C (pointers, memory, etc).

Note that several years ago the AP Computer Science exam got switched from C++ to Java. I personally support this change because most high schoolers are not mature enough to fully grasp the complexities of C++. Java is a bit gentler (Python would be better). However I don't see why C can't be the Shakespeare of Computer Science in high school. Everyone hates studying it, but it's probably better for the student in the long run.

On a slightly related note, I don't hear of too many efforts to make Calculus more accessible to girls or Shakespeare more accessible to men. This whole women in programming thing is an interesting phenomena. Don't get me wrong, efforts like this are great, and I've even participated in a few myself. But let's be clear about the source of the 'problem'. Deeply embedded society pressures and bias in early education are two major factors working against women in programming. By the time they've gotten to high school we've already lost quite a few for good. Let's also be careful not to attribute every inequality in sight to socialization. That's a nice PC answer, but unlikely.

<comment><USER4> "We teach Shakespeare and Calculus in high school, though those are certainly not the easiest choices in their respective fields."

I'm not sure that's the best argument. In my sophomore year in high school, we covered Romeo and Juliet. At that point, it is fair to say that I had reading skills _well_ above average. I could read original texts from the 19th, 18th, and even to a limited extent the 17th centuries and understand them just fine.

Shakespeare I could make the plot out, but little more. The footnotes assured us that all kinds of funny jokes and puns were being made, but we pretty much had to take that on faith. Seeing the plays helps a bit more because you get the additional channel of seeing real people, but even then we're taking the humor and puns on faith. I suspect that if someone from Shakespeare's time could see a modern production they would consider it atrocious (once they got past the superior delivery technology); places my footnotes assured me were hilarious were usually delivered straight in the videos I saw, as if the actors didn't realize they were making a pun and it was supposed to be funny. Most likely because, well, they _didn't_ realize it.

I question whether we teach Shakespeare as anything other than "the traditional curriculum", and with a desperate-but-futile hope that this will somehow inculcate a desire to be exposed to more "good culture" in the students... fresh on the heels of teaching the students that "good culture" is written in an incomprehensible vaguely-English language which is completely inaccessible without footnotes which assure us that it's good, really, in plain contradiction to the evidence of their eyes. The shock of the literati at the failure of this plan is amusing, but still sort of odd.

I see a lot of parallels to this with teaching high school students C. We're going to teach them a nearly-40-year-old language (might as well be from the 1500s) written to run on computers the size of rooms with less power than the cell phones the students have, with copious verbiage telling the students _no, really, this is computing!_ despite the fact their eyes will tell them their programs resemble none of the programs they actually use? Gosh, how could that _ever_ fail?!

Save it for college.

Just to highlight one of my points, I think one of the worst things you can do is contradict what the students are seeing. They see an archaic, musty language that can barely be understood, that resembles nothing they see in current use, but you assure them that this is the height of Western culture and they'll be better for understanding it. All you do when you do that is blow your precious credibility. I'm sure one or two people will pop up here and insist that Shakespeare changed their lives in school. Maybe one of them is even actually telling the truth, and not just defending the status quo because it's the status quo. But nevertheless, statistically, it's a terrible approach; it can't convert anyone who isn't already very interested and it turns away the _vast majority_ of any who may be interested. I'm not saying we should be feeding our students educational candy, but let's teach things in High School that, when we claim they are awesome, might actually be able to _stand up_ to that point, within the student's _actual_ reference frame and not the "desired" one, which only exists in the educator's head. Dickens, or Melville, Swift perhaps, or perhaps Python or Processing. Jumping to C or Shakespeare isn't done to help the students, it's done to gratify the teachers, and it fails in entirely predictable ways.

A world full of great literature and great computer languages, and we insist on homing in on the very things most likely to turn off the students forever. Then we wonder how schools manage to turn out so many students who hate "learning". Gosh, how could that be?

 _Edit_ : One final note. I often get a lot of shock when I question Shakespeare's value. But I think "question dogma" is part of the Hacker's credos. Is Shakespeare holy because you've worked with exposing people to it and seen it change their lives (not just your own, either) and because you yourself love it, or is it holy to you because people have been telling you all your life how wonderful it is to the point that saying "Shakespeare is great!" has become a social signaling mechanism to indicate how cultured you are? I freely acknowledge that there are people who truly do love it... I just don't think they are that numerous.

<comment><USER5> You are in effect making the argument that "tough learning" has no place in the curriculum. It's absolutely crucial that students are confronted with learning curves that are discontinuous compared to what they have previously faced. The other road leads to common denominator mediocrity. The real problem is that you cannot set an individual curve for students because of the need to teach them en masse.

The most significant experiences in my life have been about overcoming learning curves that seemed insurmountable. Two experiences stand out for me, one with mathematics, the other with literature. Even one experience like that can last you a lifetime. It can lead to a career and a lifelong passion.

My epiphany in mathematics came at age 16, the one in literature much later at age 22. The reason both came so relatively late was that my arrival was by circuitous paths of self-study. Unfortunately I never had a truly inspiring teacher.

I'll say a little (or a lot) about the epiphany I had with literature since you seem to have a particular problem with canonical literature. Until my teens, I read more than most kids but not very much and virtually nothing but pulp fantasy and science fiction. When I became interested in programming and later mathematics I read a lot of technical books in those areas. But anything vaguely literary with a challenge accompanied by no immediate reward was a complete turn-off.

Poetry in particular was incomprehensible to me. I could not understand any of the poetry we covered in school, and I could not understand how anyone could understand it, much less appreciate it. In short, my attitude was the same as yours: All these people who pretend to love it must be pretentious pricks all hoodwinked by the Emperor's New Clothes.

I eventually stopped reading almost any fantasy and science fiction and stuck to nonfiction, mostly technical material. But when I was age 22 I picked up a copy of Pullman's The Golden Compass and read it through in one sitting. I thought (and still think) it was the best work of fantasy I had read.

I looked the author up online and found that elements of the books had been inspired by Milton's Paradise Lost. I had heard of this work but never read it. I swifly found a free copy on Gutenberg and discovered to my dismay that it was a narrative poem. A fucking poem! But this writer guy says it's the best poem in the English language, so there must be something to it. I tried to read the first page but was defeated by Milton's tortuous syntax and mythological allusions again and again.

I gave up and told myself that poetry was simply not for me. But at least I had now convinced myself that this kind of old and difficult poetry was not an elaborate inside joke or some sort of hoax.

A few months later, after rereading The Golden Compass, I decided to give it another shot and this time ordered a handsome printed copy that incidentally came with an introduction and a chapter by chapter commentary by Pullman. This time I forced myself to read everything aloud and that was the trick that let me crack it.

Wow! Poetry should be read out loud!

For some stupid reason this had never really occurred to me. Soon I had gone through and understood the first several pages. Suddenly I could read and understand the poem with surprising ease. It was the most sudden zero-to-hero experience I'd had in my life. I was covered in goosebumps and tears were welling up in my eyes. I was forced to stand up and pace around the room for several minutes to calm myself down. Finally I sat down and spent several hours memorizing the first couple of thousand lines, despite my generally awful memory for such things. After that experience I found I could read through a book of the poem at the rate of one per night. Not all the allusions made perfect sense on a first pass, but now that I had learned to sound out the words the syntax was no longer an obstacle at all.

The poem possessed me so completely that I eventually memorized the first two books of the poem almost effortlessly, possessing the poem in turn. And when I say memorize I don't mean by rote but by the kind of compulsive rereading and rethinking that only passion can motivate.

Anyway, that was my epiphany. I decided to try Shakespeare next and bought a copy of Midsummer Night's Dream. That was a lot harder! The word to word, sentence to sentence writing of Shakespeare is so far beyond the likes of Marlowe, Milton or Melville in terms of subtlety and modulation. All of those guys are at their best when they write majestic rhetoric blasting on all cylinders. That's not to say they lack subtlety in other areas; both Milton and Melville are incredibly deep.

Anyway, I slogged my way through Midsummer Night's Dream. It was fun but it didn't deliver Milton's spine-tingling bombast in the same way. It was subtle the way the world is subtle. If you read it superficially you are likely to leave with little else but the plot and a general but vague sense of the characters. Shakespeare's plots are expendable and the strength of his characters are in their variety, distinction and depth, so this way of reading him does him poor justice.

I put aside Shakespeare for another year or so. My next attempt was with Hamlet and that was what cracked Shakespeare for me. It had all the gratifying high rhetoric I so loved about Milton but it also had complexity in subtlety and depth to last you an infinite number of rereadings. From there I eventually went on to read all of his works. Unlike Paradise Lost, several of them took me several rereadings to love. They rarely give away their secrets on the first stroll through.

Holy crap was that ever long. Anyway, I present this as antidote to your criticism. What this and a few similar experiences has taught me is to never assume that something is a worthless sham simply because you don't get it right away.

<comment><USER4> "You are in effect making the argument that "tough learning" has no place in the curriculum.... you seem to have a particular problem with canonical literature."

No, which is why I explicitly mentioned Swift, Dickens, and Melville. We read both Gulliver's Travels and A Modest Proposal in school (nowhere near a complete list, of course), and even if it wasn't everybody's "favorite literature ever", at least it was _comprehensible_ to people. You could send students home with a reading assignment, and they might actually do it, and then when they come to class the next day, they can _discuss_ the text, and find out the deeper meanings, rather than having to come to class to find out just what the hell it was they just read. This is not a valuable use of literature time. Also, as I said, I read quite a bit of the "canon" of my own free will and could read Ye Olde Texte far better than most people of my age. More than once I was given a school assignment to read a book I had already read.

Your story is not a counter to my point; it proves my point. You had to be interested and bring a _lot_ of effort to the task of reading those things. This makes them grotesquely unsuitable to be using them in high school where those things are not generally available. That you found it rewarding is great. I too have done that, both in the computer domain and the literature domain. But an awareness of what it took is precisely why I say it's a terrible approach to general education.

I never said it was a worthless sham. I said it _wasn't worth teaching in high school_ , and that it should be left for college. Based on the age you had your experiences, you hardly provide a counterexample for that. It would be way better to simply _allude_ to Shakespeare's genius and let motivated people find it, than spend valuable time jamming it down students throats only for them to see an incomprehensible mass that they are never again motivated to look into. There's no way school could every have given you that experience. (Maybe a really motivated teacher, but even then it would have been a personal interaction, not a schooling thing.)

Basically, my position is that you have to educate the students you have, not the students you wish you had. And, if we are going to build our education system around teaching masses of children, building our core curricula around something that will maybe appeal to one or two kids per year, and then only several years later, is such a waste when we could be teaching people things that might actually _encourage_ rather than _discourage_ the pursuit of further learning.

So often the choice is presented as "Shakespeare or nothing", or "only incomphrehensible literature or nothing". You sort of have those memes in your response too, when you read into my post that I'm against literature. But when I spell that out, it's obvious that there are tons of other choices. There's thousands of other school-worthy candidates that won't have the side effect of firmly squashing all future interest in "literature".

(There's also a frequently unrecognized assumption that we have infinite time for students to learn in, and therefore it is sufficient to establish that something is "good", and therefore we then put it in the curriculum. That's not even close to true. Our time as students is _terribly_ limited compared to what there is to learn, and it's getting worse every year. The question is not "Is Shakespeare worth learning?". Duh. Of course it is. But then, there's enough "stuff worth learning" to fill your lifetime and beyond. The question is, is it the _best_ thing to learn? _That's_ what I'm challenging. Given the enormous liabilities that teaching Shakespeare... oh, and by the way, C, since the parallel continues to hold(!)... brings to the table, including the very- real risk of completely destroying any interest in literature/programming that may have been present, why shouldn't we teach the things that don't carry that risk?)

</comment></comment></comment></comment></comment><comment><USER6> The key to this (lovely) story, IMHO, is the iterative nature of the lesson. Each step was very small, built on the previous one, and resulted in clearly visible changes to the program. For novices, this is absolutely vital- especially in a classroom environment like this. Now, re: the appropriateness of Ruby's somewhat-odd syntax for this lesson- I agree that there's definitely some weird arbitrary-seeming stuff in Ruby ("darkred" vs "dark red", etc.). But, let's face it- ALL languages have a few weird syntax things here and there, especially from the standpoint of a novice programmer. It's just part of the way that computers and programming languages work, and anybody who wants to write code is going to have to learn to deal with it. From my experience teaching novice programmers, however, I can tell you that a lot depends on HOW

The problem is that, with most languages and in most intro lessons that I've seen, the student is exposed to all of them all at once without much context. I feel like the lesson described in this article did a great job of introducing the students to these language "features" one or two at a time and in a relatively sensible order. Eg, the students first got used to the idea that the computer was picky about upper- vs. lower-case letters. Next, they got their heads around the idea that attributes often involve a weird sequence of characters ("=>"). Then, they were exposed to the "identifiers-can't-have- spaces" rule. Etc., etc., etc. By the time they were exposed to each new oddity of the syntax, they'd had a chance to absorb the one that had gone before.

</comment><comment><USER7> I'm amazed as well that it went that well. And its really great to hear that they weren't put off, lets hope the coming generation can help improve the genre ratio within programming workplaces. I like you approach, and choosing shoes probably was a good choice. Instant gratification should be #1 goal in classes like this, its all about building excitement, which it seems you nailed.

"Another girl dug into the documentation she had seen me use, and figured out how to use the color picker to let the user select a color for the line!" I'm impressed. Figuring out manuals can at times be hard even for us who use them on a day-to-day basis.

</comment><comment><USER8> When it comes to pedagogy, the sigils and syntax of Ruby have always bothered me. The author notes the girls having trouble with _dark red_ versus _darkred_ , or getting the hash syntax right. I imagine a number of them wondered why _"center"_ and _"xx-large"_ went inside quotation marks, while _saddlebrown_ did not. Furthermore, special string values like "center" and "xx-large" somewhat undermine the teacher's assertion that quotations demarcate "a sort of free zone for text."

Are there better languages or tools for this task? Has anyone here tried to teach programming to non-programmers using a visual language like Scratch or a simpler language like Scheme or Basic? How did it go? Shoes is an incredible platform, I just wish the underlying language had a more immediately comprehensible syntax. (PLT-Shoes, anyone?)

<comment><USER9> > simpler language like Scheme

I really think Scheme is only simpler if you have a _really_ atypical kind of mind.

<comment><USER8> Actual development in Scheme is a bit mind-bending, but sexps are wonderfully simple.

      (fn arg1 arg2 ... argn)

Add quoting, and you're pretty much done explaining the language's syntax.

What sort of facilities can you build on top of that foundation to create an easy, accessible learning environment like Shoes?

<comment><USER9> The average high-school girl might not be able to instantly see the elegance of the idea that functions, programs, data structures, numbers, etc. are all interchangeable and that it all can be represented recursively with lists all the way down. (High school girls who have read Hofstadter, maybe.)

They might be able to understand "later, rinse, repeat", or making a cake by making three layers, and then making the frosting, where each becomes a separate modular and reusable problem.

<comment><USER10> His point isn't that Scheme's syntax is elegant, but that the syntax is _simple_.

There's no need to worry about the precedence of operators, or indentation, or why you use one character to separate function parameters and variable declarations, but you use another character to separate the arguments of a "for" statement, which happens to be the same character you need to use at the end of statements (but as a terminator, not a separator), but don't forget that control structures are not separate statements even though they're on separate lines, etc.

Each of these points is an obstacle between the beginning programmer's intent and a running program. Scheme's almost total lack of syntax gets a beginning programmer started quicker.

<comment><USER9> That might seem simpler to you, but you have the kind of mind I was talking about:)

To the average person, to make a toy car, it's straightforward to take a box, put on two axles, then put on two wheels. You could do all that with just toothpicks, but it's not really simpler in practice.

</comment></comment></comment></comment></comment></comment><comment><USER11> Totally awesome. It's so cool to hear such great success stories about teaching programming, especially to those who are unlikely to take it up otherwise. I wish I could upvote this 100 times.

</comment><comment><USER12> I am surprised how well that went down to be honest - they must have been some bright kids!

I don't think you'd have as much luck with first year uni students in a three hour segment.

Great work and I love the extension ideas (including USB drives!)

<comment><USER13> I had similar success teaching Shoes to fourth & fifth graders in a series of 3 1-hour lessons. I liked Sarah Mei's lesson even better, but for another datapoint, here's my report: [http://teachingkids.railsbridge.org/2009/07/20/fifth- grade-s...](http://teachingkids.railsbridge.org/2009/07/20/fifth-grade- shoes.html)

I think kids find programming more natural than adults do. They are used to learning many strange and arbitrary rules :)

</comment></comment><comment><USER14> Instead of just having the program files on USB memory stick could you have a had a custom bootable linux environment instead - then getting the environment set up would just be a case of plugging in to a PC and booting up. Sometimes getting to the point where you start programming can be hard.

Just a thought.

<comment><USER15> I'd say it's one of the biggest barriers to entry out there. Getting a dev environment set up for anything other than hacking a Wordpress template is non-trivial for most.

I use the stuff at PortableApps.com quite often with consulting clients who need Firefox/Audacity/Gimp in a hurry. Could there be a portable Ruby/Python/Django dev setup that could get a novice going quickly without having to install a bunch of files?

<comment><USER16> <http://www.instantdjango.com/></comment></comment></comment>


Show HN: Git-bars – Utility to visualize commit counts on the terminal - <AUTHOR> http://github.com/knadh/git-bars <comment><USER2> This is a lot of fun. Can this make the weekend a special color?

<comment><AUTHOR> Thanks :) Weekend bars are now blue.</comment></comment>