Hiring Without Whiteboards - <AUTHOR> https://github.com/poteto/hiring-without-whiteboards <comment><USER2> We use a 3 step process, sans whiteboard: 1\. First interview, the candidate interviews us. What market we serve, what our development processes are, what our technology stack is. If they express interest by being prepared and asking good questions, we send them home with

2\. a programming task. Choose 1 of 5 tasks. The tasks are not abstract problems or puzzles, but come out of the designs we've implemented. We ask for 100 lines of code and not more than 2 hours. They take as much calendar time in days as they need, most have full time jobs, and let us know when you're done. The candidate then comes back, typically in 2 to 7 days and hosts a code review in front of our team. I give strict instructions before hand: The purpose of the review is to learn how they thought through the problem and how they solved it, not to criticize their style and approach.

3\. If we decide we like them to this point, the third interview is with managers from other functions. How well does the candidate communicate, come across to non-developers, express interest in the company and role, etc. It's a check point to look for concerning weaknesses, as well as get buy in from the broader organization.

We like this approach because it allows the candidate to code in a much more natural environment. They can take the time they need and comfortably solve. No one spends their professional career coding on a white board. This approach so far has yielded excellent results. We ask developers how much time they took in the programming task and why they chose the one they solved. The programming task is telling, not in the quality of their code and how long they took, but how much did they get into it? We have had the range from candidates who did not complete it at all and opted out, to candidates who stumped 40 year veterans with elegant code. In every case, we learn how well they can express their thoughts, and importantly, their level of love for the discipline. This is as important to me as any other attribute.

<comment><USER3> How do you avoid awarding candidates who violate your request of no more than two hours and spend considerable time coming up with something amazing? They may still produce 100 lines of code — but those 100 lines may end up being unusually elegant, beautiful code that stumps 40 year veterans. How do you avoid punishing someone who took your requested time constraint seriously (or otherwise didn't have enough free time to violate it anyway)?

That's not to say I don't think this process has promise, but I'd need to be convinced that the above two issues can be fairly dealt with first.

<comment><USER4> I reject entire the premise of this question. Anyone spending huge amounts of time is either going to produce way too much or they just took too long. There is nothing magical about programming where taking 8 hours to solve a 2 hour problem is going to stump 40 year veterans or be unusually beautiful. That's just unrealistic.

<comment><USER5> You are taking it to the other extreme. Sure, in 8 hours I will not write something "unusually beautiful". But if I spend two hours solving a problem and hacking together a solution, I come to better understand both problem and solution. Hereafter, I then can rewrite my solution in another couple of hours and improve it significantly with respect to maintainability, readability, naming, messy choices, &c.

If I want this job, I would spend this extra time to submit this better solution because obviously that will improve my chances of getting the job. But that means that these take home assignments might not be such a good indicator of efficient proficiency. You might be overvaluing people spending more time on these tasks and undervaluing people spending the allotted time on these tasks. For example, a better developer who did write a solid solution in under 2 hours might get passed over because her solution seems not as good as mine where I did spend another hour to rewrite my initial messy solution.

</comment></comment></comment></comment><comment><USER6> I interviewed with one of the companies on that list a few years back (noredink). They gave a timed hackerrank style coding question as round one (so I guess technically not a whiteboard). I passed that, then I had an interview with an actual person.

He asked me vague question, like what is architecture, I started to reply with what design tradeoffs I made on the app I was working on. He literally laughed at me, and said that is not architecture. I was shocked that someone would laugh at a candidate but said ok, what does architecture mean to you? He responded, I ask the questions here not you. I immediately ended the interview. Worst interview I have ever had. Every time I see that company brought up, I think back to that experience.

<comment><USER7> I've had one such interview go really bad. I am normally a very trusting and confident person, but that interview destroyed me for days. This was for an internship on my first year of uni when I just started learning to code C and C++. The position was for a Junior programmer (which I was).

At some point the interviewer asks which I would choose in a project, C or C++ and reasons why. Basically answered that it depends on the project, but that both languages had their pros and cons, and it all depends. I told him I did not that many projects under my belt at the time.

Interviewer got somewhat pissed off and kept asking me about specific languages constructs which I was not yet familiar with. In a way felt like he was trying to ambush me and make me feel incompetent. I mentioned to him that I was not yet experienced enough to answer these questions, but was eager to learn. Well he did NOT like that either and ended the interview promptly.

It seems like the guy really did not like me (which is fine), but also wanted to turn this interview into some type of power trip ....

Had this been today, I would have politely declined and left. But I was a young and eager cub. Live and learn!

<comment><USER8> I rarely see it talked about just how much ego plays a part in interviewing. In fact, I'd wager it plays one of the _biggest_ parts of the interview process. If you get even one interviewer that is bitter, unhappy, on a power trip, or otherwise a sociopath, you're not getting hired. Many folks hate interviewing candidates; it can put them in a bad mood. They're going to take it out on you because it's safe.

<comment><USER9> Ego issues sometimes apply to the interviewee as well. I once had to interview someone that clearly thought the interview process was beneath them. This person answered every question as if it was a stupid endeavor and we should just hire them since it was just obvious how great they were.

<comment><USER10> What would have been a way to find out s/he had that mindset, before having the interview?

<comment><USER7> A crystal ball?

All jokes aside he was not foaming at the mouth or unpleasant when we first started the meeting, and I'm not the quickest one to pick up on subtle body language cues. So your guess is as good as mine.

<comment><USER10> interesting that weren't any "signs" before

<comment><USER7> What type of signs do you have in mind? Maybe some examples of your own experience will help me understand missed clues.

</comment></comment></comment></comment></comment></comment></comment></comment><comment><USER11> This is a great list. Thanks for sharing it.

I actually had the pleasure of interviewing at one of these companies before. They had a take-home project, which was to choose and implement and couple enhancements to a toy app. In the subsequent conversations, we discussed how I approached the problem, details of my design, technical tradeoffs, etc -- all the sorts of things you would expect a professional sw engineer to be able to do on the job. It was challenging but fair, and I was struck by how reasonable the process was. It left a wonderful impression on me.

Take-home projects are not a perfect solution to the interview problem. A major issue is that they require candidates to have free time outside of work to complete them. Personally, though, I'd much rather spend a few hours per application working on projects than countless hours prepping for stressful whiteboarding puzzles. As a developer who prefers to digest problems slowly, I look forward to the demise of on-the-spot whiteboarding interviews.

If you work at a company that subjects candidates to such high-stress algorithmic riddles, and you have the ability to make your process more sane, please consider following the lead of cos on this list.

<comment><USER12> I hate take-home projects more than I hate whiteboard interviews.

I think the best approach would be giving the candidate a broken app and have they debug it live or in a take-home debug-and-fix or debug-and-improve exercise. Could also have some refactor assignment.

Has any company tried something like this before?

<comment><USER13> I don't understand how take home interviews are being held up as a better solution. They have two major flaws:

They lower the cost for the company to zero. So they can widen their pool, and take a chance on more candidates, with no extra cost incurred for them. If you take this to the extreme – every company asking every candidate to complete a take home – then the result is far worse for candidates.

There is no feedback mechanism for candidates during the process. I can say "you shouldn't spend more than two hours on this" until I'm blue in the face, but candidates think they need to polish far beyond what is reasonable. On occasions where we had a system for capturing progress, I found that most candidates far exceeded the time recommended to them. Some were spending up to six hours. The frustrating thing was the extra time never made a difference to the outcome. I could tell from their initial hour whether they were going to pass.

As a hiring manager I've used them, but work hard to be considerate to the candidate. I'd prefer to design a better in person interview than take the easy route that is disrespectful of candidates' time.

<comment><USER14> At Caltech, most exams were take home with a time limit of 2 hours. Some were "infinite time". The students hated the latter, because it means there was no end to the time you needed to spend on the exam, meanwhile there were other exams, etc., to take.

With a 2 hour exam, there's an end to it, and you know how much time it should take to do the problems. Much better.

It's even worse if you're being graded on a curve, as you would be in a job interview. You're competing against others who will spend infinite time on it.

<comment><USER15> Like you said, "there were other exams, etc., to take.". Presumably other competitors have other exams as well? You'll thus be competing against people with different time management skills.

</comment></comment></comment></comment></comment><comment><USER16> This is yet another tech interview discussion, so I'd like to highlight a subthread from the one earlier this month:

> I'm actually a little scared to leave my current FAANG gig for that exact > reason tbh. I'm fairly certain I wouldn't make it back in the door without > more leetcode grinding + repeated loops than I'm willing to do at this point > in my career.

>> I'm in a similar position - a tech lead role at a Big N - and have recently been doing some interviews for senior roles at FAANG. The most frustrating thing is knowing I can't really leverage anything I've learned over the past 7 years of my career in the interview, at least at the early stages.

>>> I feel extremely little of what I'm doing in my real, actual, job, helps me in advancing in my career. Unless maybe if I choose to stay at my current company until retirement lol. Otherwise, why bother doing anything more than the bare minimum to get by at work? It would be a far better investment of time and effort to grind leetcode and practice for interviews, instead of going above and beyond to excel at my job. At least until I get into an "endgame company" where I feel it's worth staying long term.

[https://news.ycombinator.com/item?id=23848717](https://news.ycombinator.com/item?id=23848717)

Even as we debate the validity of the metrics measured by technical interviews, it's also worth considering the metrics used to measure engineer worth in the industry at large. This existentialist "fear and loathing in FAANG" discussion was quite illuminating, and a little sad.

<comment><USER17> > I'm actually a little scared to leave my current FAANG gig for that exact > reason tbh. I'm fairly certain I wouldn't make it back in the door without > more leetcode grinding + repeated loops than I'm willing to do at this point > in my career.

Makes you question if this is the precise point of it.

<comment><USER18> Disagree. It makes zero sense for Twitter to make it harder for them to poach Google employees.

</comment></comment></comment><comment><USER19> The advantage of leetcode is that you study once and then it applies for every job you interview for. Like democracy it's a horrible system except for all the others.

Take home project? There goes 4-12 hours per company that gives you one and sometimes more. Companies have no incentive to cut it down or not give it to even marginal candidates so you'll get a lot more of them than full in-person interviews.

Pair programming? That's basically white boarding with a better white board or, in the remote interview world, just regular white boarding. Sure the problem is more real world, in theory, but then you get issues of knowing the same frameworks, etc. So to do well you need to spend a lot of time beforehand studying their specific frameworks and code bases if they ask you to do an issue on their open source project.

<comment><USER20> In the past we had some candidates complain about take-home problems. The primary complaint was that candidates weren't interested in dedicating 4-5 hours of their own time unless they felt that someone from the company was putting in an equal amount of their own time. Some candidates also had suspicions that we were trying to use them for free labor (Not true, we used the same toy problem for every candidate). One candidate even thought they should be able to bill us for their time spent applying (We passed).

Eventually I changed the strategy and offered candidates the choice of the take-home problem or an on-site whiteboard style interview where they solved a subset of the problem in pair-programming fashion with us. Zero people chose the on-site interview. Everyone chose the take-home problem.

No one complained any more after we reframed the take-home problem as the candidate's choice rather than the company's.

<comment><USER21> So instead of doing a 4-hour on-site interview, you make a hiring decision based on a 4-hour take-home problem?

<comment><USER10> Seems you think that 4h test was all, wondering how you got that idea

<comment><USER21> So candidates actively chose to add a 4h project on top of a 4h in-person interview?

<comment><USER10> There's probably a one or two hours soft skills interview and more that they didn't mention.

</comment></comment></comment></comment></comment></comment><comment><USER22> No one enjoys whiteboard interviews but it's not clear that these companies have better alternatives. Many of them do take home projects which are time consuming. I'd prefer the employer look at my GitHub profile as there's plenty of code there already. Furthermore, I'm not a fan of "interviews" in general, I find that more information is transmitted by eating lunch with the team and talking like normal people.

<comment><USER23> As an interviewer, how do I know that you wrote that code in your GitHub profile?

<comment><USER24> Ask them some questions about it over lunch?

What's your threat model here? People making up elaborate lies about code they've written?

<comment><USER25> > What's your threat model here? People making up elaborate lies about code > they've written?

Likely people making up elaborate lies about code they didn't write. It's trivial to push code of which you are not the author.

<comment><USER26> Which is why the interviewer should ask specific and thoughtful questions about the project, so they can gauge the candidate's level of understanding. This concern could also be addressed by introducing a trial period.

<comment><USER19> >This concern could also be addressed by introducing a trial period.

Why would I leave a stable well paying job for a job that basically says they don't trust me enough to not fire me in two weeks?

<comment><USER25> > Why would I leave a stable well paying job Well, why are you doing that in > the first place? Maybe the new opportunity is worth it or the current one is > so bad that it's worth it.

</comment></comment></comment></comment></comment></comment></comment><comment><USER27> Here we go again perpetuating the stereotype of the whiney privileged engineer put in a position of mild discomfort by the villainous abusive white board interview boogeyman. I don't mind whiteboard interview exercises. They're not great, but they often give a very good and reasonable signal that often has a high corollary to work performance. I think there's bigger fish to fry in refining fair and objective interview processes, I'll never understand the perpetual moaning around this aspect of technical interviews.

<comment><USER28> > they often give a very good and reasonable signal that often has a high > corollary to work performance

Do you have any statistically significant data to back this claim?

<comment><USER29> There is statistical data saying the opposite: [http://chrisparnin.me/pdf/stress_FSE_20.pdf](http://chrisparnin.me/pdf/stress_FSE_20.pdf)

That whiteboard interviews are actually testing how you perform under stress rather than how good you are at your job.

Additionally, in my opinion, it measures how good the interviewer is. If you get a bad interviewer or even just one in a bad mood, just pack your bags because you aren't getting it.

Whiteboards are incredibly arbitrary in my opinion, if taken alone. If it is part of a complete assessment, ehhhh.

<comment><USER30> >There is statistical data saying the opposite

Interesting study and really good discussion on HN at the time, but the _opposite_ of "[Whiteboard interviews] often give a very good and reasonable signal that often has a high corollary to work performance" would be something like "People who do well at whiteboard interviews tend to be worse performers at work." I don't think that's what the study said at all; rather that people do worse at whiteboard problems when they're being watched vs. not being watched.

<comment><USER31> Also people act as though software is a job without anxiety. There is plenty of anxiety in this job. Sometimes under pressure you really do have to perform or push back and its good to know if people can do it.

<comment><USER32> > Also people act as though software is a job without anxiety.

Not. All. Anxiety. Is. The. Same.

Ask a firefighter to give a speech in front of 1000 people. This is an entirely different kind of anxiety than running into a burning building. Ironically, it may be the former rather than the latter that makes the firefighter sweat.

Writing code is generally not a performance art. We're not stage performers. Interviews are an unusual situation that aren't anything like our daily jobs. Stress on the job is not the same as interview stress. Work emergencies are not the same as interviews.

We need to take into account human psychology. These factors are all well known, but the tech industry seems to act in deliberate ignorance of humanity.

<comment><USER29> Also stress on the job isn't the same as stress getting a job.

One you are getting paid no matter what, the other you aren't and are 'performing' in the hopes of getting paid.

</comment></comment></comment></comment></comment></comment></comment><comment><USER33> I am starting to think the interview is a waste of time. Sure, you want to make sure the person is more-or-less who they claim to be on the surface, but you will never really understand their capability for reasoning with your problems until they are working in your process.

One of my earlier jobs dealing with code involved a very brief interview (no whiteboard involved) followed by a 6 month contract offer. The deal was that I basically had 6 months to prove myself, and if everyone felt like it was working out (myself included) this would be bumped into full-time with benefits. The crazy thing is, I was actually the one who decided I did not want to continue after the 6 months was out while they wanted me to stay very badly. I think this can be a really useful checkpoint tool for both sides.

Anyone can misrepresent themselves in an interview and dupe a room full of people for a day or 2. No one can keep up an act for 3-6 months continuously when working with complex code tasks. You will get found out before the term is up, assuming everyone is using this process as an evaluation tool and following up on it regularly.

<comment><USER34> > The deal was that I basically had 6 months to prove myself, and if everyone > felt like it was working out (myself included) this would be bumped into > full-time with benefits.

In the UK, we sort of have this in the form of probation - in most permanent jobs, you don't gain full employment rights until you've done a probationary period, which is usually six months. You get full salary and benefits during that period, but you can be fired with no notice at any time (or something like that).

In practice, though, the culture is not to use the probationary period as an extended interview, but more of a backstop against really screwing up hiring. You'd only let someone go after probation if they turned out to be absolutely incompetent or unsuitable, not if they were just disappointing.

It would be interesting to try using probation differently, though. Make the interview more of a screen, employ twice as many people as you need, only retain half of them at the end of probation, and be explicit throughout that this is the plan. I suspect you could implement and communicate this such that it wasn't absolutely horrible for probationers - active and transparent review during probation, one or two months' golden parachute, what else?

<comment><USER35> In Ontario, the letter of the law says you can be fired during probation for any or no reason (modulo human rights legislation), but in practice an employer can still be sued for wrongful dismissal if the employer terminates an employee without demonstrating cause.

It's best to have experienced employment law advice when devising any scheme around using probation as a kind of hiring scheme, as opposed to as a last resort if things go very badly.

I will not give legal advice, but for example I would be very careful before implementing a "Survivor: Tech Island" scheme where the company hires more people than it genuinely expects to keep following their probationary period.

I have been told that if I hire someone I think is questionable with the thought that "if it doesn't work out, we'll just let them go during probation," that can backfire as I make a conscious choice to hire them despite my reservations.

Again, INAL, but I would tread carefully around using probation for winnowing candidates out.

<comment><USER34> IANAL either, but my understanding is that _wrongful_ dismissal is breach of contract, whereas _unfair_ dismissal is illegal [1]. So if you write this procedure into your contracts, and then follow it, it's not wrongful dismissal. To be unfair, for a probation period of less than two years, it would have to be meet one of the criteria for being "automatically unfair", which are basically about the employer doing illegal things, or be discrimination [2].

I would also be very careful before implementing a scheme like this - as you say, you would want to have it thoroughly checked by legal and HR experts.

[1] [https://my.ucu.org.uk/app/answers/detail/a_id/280/~/unfair-d...](https://my.ucu.org.uk/app/answers/detail/a_id/280/~/unfair- dismissal-and-wrongful-dismissal)

[2] [https://www.citizensadvice.org.uk/work/leaving-a- job/dismiss...](https://www.citizensadvice.org.uk/work/leaving-a- job/dismissal/check-if-your-dismissal-is-fair/#h-3-check-you-ve-been- dismissed-for-a-reason-that-s-definitely-unfair)

<comment><USER35> I think you're right about the distinction between wrongful and unfair dismissal, but if I understand Ontario common law correctly, even in cases where the dismissal during probation is not illegal, and even when you have it in writing that the candidate can be dismissed for any or no reason, you may find yourself on the wrong side of a wrongful dismissal judgment and forced to pony up money.

It's probably easier to structure such a thing as contract-to-permanent, and I suspect this is why there are a lot of such contracts around. Or at least, I have seen a number of such things in Ontario.

And now the counter-example: My brother is the principal Tuba with the National Arts Centre Orchestra. He had to win multiple auditions, resulting in two finalists being selected. Each of them then went and rehearsed and played a concert with the orchestra, they spent some weeks on contract, expenses and compensation paid. They selected my brother.

Full-time job? Nope! He then went on probation, either for a full year or two, I can't remember. I'm sure he'd laugh at my worrying about 90 day probationary periods.

</comment></comment></comment></comment></comment><comment><USER36> One thing I've been wondering lately about _take-home tasks_ is how everybody complains about time estimates, and yet here's a bunch of "small projects" which someone pretends to have very precisely estimated.

Unfortunately, most companies fail to honestly estimate the cost of the tasks they assign for these processes. Many of them because they just don't take into account numerous small tasks that the candidate will need to deal with.

I've seen a lot of these tasks ask for setup instructions, _good_ unit tests, explaining any decisions you make, etc. In case there's any visual user interface (e.g. some web component) they also ask for it to be _nice_ , as in "we don't want a designer but we need you to make it somewhat nice". And then they'll say it should take just a couple of hours.

I've also seen a fair number of others casually say their task should _only_ take about 8 hours. And I can do that on my free time on five consecutive days, after having worked already 8 hours each day. Sure thing.

<comment><USER37> > One thing I've been wondering lately about take-home tasks is how everybody > complains about time estimates, and yet here's a bunch of "small projects" > which someone pretends to have very precisely estimated.

When I interviewed people and gave them a take home assignment I was very confident in my estimate for one simple reason: I completed the tasks myself. We had 4 tasks for this for the 4 different types of jobs we hired for. The "estimate" was just double the time I needed to do the task myself. We got feedback from some of the candidates that our estimate was very good.

If I could always estimate the time AFTER I complete the task instead of before then estimation would not be hard at all.

<comment><USER36> Good for you; not everybody does that :)

If you don't mind me asking, how big were those tasks?

<comment><USER37> We gave the estimation as follows:

\- Junior developers should be able to finish this within a work day (8 hours) \- If you are already familiar with all technologies involved then it should take about 2 hours even with all optional tasks completed.

As an example our backend dev task was to parse a CSV file (supplied with the task) and provide 2 API endpoints (return format and url defined in the task) that access this data (free choice in language and libraries). The optional tasks for this was to include unittests and to use docker.

That really isn't much work. We did get some good feedback about this estimate both from junior developers and seniors after they were hired or rejected. Most seniors seem to complete this task in about 1.5h to 3h and juniors do seem to be able to complete this in about 6h.

</comment></comment></comment></comment><comment><USER38> Oh God, here we go again.

Coding on a whiteboard in an interview is not "broken". What _is_ "broken" however is making the problem the candidate needs to solve "hard". I can't stress this enough: a whiteboard coding test is nothing more than a _negative filter_. It's to filter out people who can't turn a _simple_ idea into code as these people exist.

This is why FizzBuzz was such a simple problem.

But interviewers fall into the trap of thinking "this problem is too easy" and make it hard (eg figuring out a problem is g union-find on the spot and then implementing it) or, worse, they make it a crap shoot of whether you know the "trick" or not (eg reversing bits in O(log n) or the tortoise and the hare). This actually makes the test _completely useless_.

A negative filter is simply a filter whereby if you fail it, you almost certainly aren't a great hire but the reverse doesn't apply: there should be no A+ on a whiteboard coding test. It's straight pass/fail and an easy pass at that.

I realize some people have anxiety about doing this. Having helped or coached a bunch of people in the past about this, I can say that mock interviews and practice help the vast majority of these people. Some may be helped by doing this on a laptop in a shared doc instead, which is fine.

A few will struggle with the anxiety of that, even with coaching and practice. While I feel bad for those people, I do wonder if a reasonable-sized organization, which will generally require a certain level of communication skills and presentation ability, is the right place for you.

<comment><USER39> I agree with this. I work at a FAANG that does whiteboard coding interviews (opinions my own, etc etc).

When I interview people, I give pretty straightforward questions. My goal is not to run some algorithm guessing game. If I'm interviewing for a senior FE position, I make sure they know enough about async patterns in JavaScript that I feel confident that they could build an event handler without dropping promises on the floor.

A back end position? Can they use (not implement, just use) a hashmap efficiently and do they think about boundary conditions in input sets (you'd be surprised how many people ignore punctuation when breaking sentences into words - they just do a split(' ') and call it a day.

This is really basic stuff. I'm always able to (a) filter out people who just need more time learning to code (or are too nervous to perform well, which is still a problem with even these whiteboard activities) and (b) prompt a discussion about engineering practices.

My criterion is simple: do I think I could see this person submitting code to my project without me needing to constantly fix what they write? It isn't about memorizing algos or anything.

Aaalso: I didn't study leetcode, and managed to get my job just fine. I know there's a reputation of the process being just algos, and I'm sure a lot of the interviews are. But not all of them, even at the FAANG.

<comment><USER10> It'd be interesting if you know about any similar hard/easy level questions on the web

Id take a look :-)

</comment></comment></comment><comment><USER40> The issue with LeetCode style interview is that almost EVERYONE can solve those questions after studying a couple weeks/months.

The only thing that LeetCode questions predict is if the candidate has been training for LeetCode questions.

The fact that Google, Facebook etc still use it as a gatekeeping mechanism makes me believe they want to find cogs that will specifically spend hours studying for it. Making sure that those engineers will be ready to execute their mindless coding tasks at the company.

<comment><USER41> Right but in some ways this is what they need. Someone who can stare at computer science material, that is probably boring, for long periods of time. Someone who has a good memory for details, can remember routines. Can "solve" problems that have been solved before by applying theory. Someone who is willing to follow the rules and go through the process.

<comment><USER42> I think you made a logical somersault. Leetcode doesn't imply any of those, neither good memory for details or that you can "solve" problems that have been solved before by applying theory.

You missed the whole point of the OP's post it seems. The only thing that leetcode implies is that you can solve leetcode tasks.

<comment><USER43> Leetcode implies all of those, and just like in mathematics competitions, being able to solve these problems implies a high level of ability.

<comment><USER42> Nope

</comment></comment></comment></comment></comment><comment><USER44> I live in Sweden and we seem to have a very different interview culture. I've never heard of whiteboards or trivia questions, here.

I've only gone to four interviews (across eight years), and been lucky to get an offer each time (three of which I've accepted). They have all been primarily discussions about personal experience, hobby projects, personal interests, would-be responsibilities, benefits, and company culture.

At one place (Ghost Games, an EA studio and the only company which wasn't purely Swedish) I had to complete a code assignment (create a clone of a classic arcade game in C++) ahead of the interview.

<comment><USER45> I saw both sides in Sweden. Including being the interviewer who discovered that senior applicants with over 15 years of experience could not for the life of them figure out how to reverse a string. Not something that I could have found out during our casual conversation about past experiences, of course.

<comment><USER46> So s/he failed the (interview) specific _anxiety_ test.

</comment></comment></comment><comment><USER47> Another tech interview thread. For the side that supports these types of interviews, I've never gotten a good answer to a simple question: in the year 2020, why are we expecting people to write compilable code on a whiteboard? It's just stupid at this point. Even a laptop that boots into some micro linux distro and has nothing but nano open would be better. Or just a fresh install windows laptop with nothing but notepad.exe open. Or a chromebook that is open to a HTML page that has nothing but a textarea element.

I'm not including a compiler or having build tools or an IDE. Just a basic simple text editing area that allows the basic functions of typing in text and editing it.

I'm against these leetcode interviews. But if you did nothing else but change this one thing, just stop expecting people to write code on a whiteboard or paper/pencil and allow them to write code the way it's actually done (a computing device with a keyboard and text edit area), that would be such a huge improvement. Writing code on a whiteboard or paper doesn't test anything. Think about how limited it is and how different it is (for example, you can't just press an up arrow and add a newline, have to find an eraser and start over).

It's yet another useless skill to learn just for interviews (writing compilable code on a whiteboard/paper) which also encourages rote memorization (because you have to get it right on the first try since editing the text is so painful and difficult).

I hope people start pushing back on this. The reaction should be, wow you care so little about your interview process that you can't get a $200 chromebook in here?

<comment><USER48> As somebody who has sadly had to make hundreds of candidates write on a whiteboard, here are problems I've run into getting the candidate a laptop:

* IT just fails: they're unwilling to supply it, there aren't enough, the current laptops on hand are junk, or the default password is wrong. These all happen with HR buy-in and funding.

* Recruiting just fails: they "have to move the candidate fast because of a competing offer," they tried to work with IT but IT blocked them, they got laptops themself using their own budget but then ran out.

* I failed: I didn't make time to prepare a non-whiteboard question, I didn't tell my manager in each and every 1:1 that candidate experience sucks, I had three laptops and I forgot to bring my personal spare to the interview.

In general the core problem is there is zero incentive for hiring managers to do anything but shotgun candidate pools like they do today. There's also no feedback loop to evaluators to help them improve. False negatives are completely tolerated despite there being a huge gap between CS jobs demand and CS graduate supply.

<comment><USER47> These are not good reasons and again shows that the company really doesn't care. IT gives a bad laptop? That's fine it doesn't have to be good, it just needs a text editor open. Get a USB drive with a linux distro and boot to it and open nano. Or bring your laptop and stick the USB drive linux distro in there and let them use it. Again there is zero excuse for a company to not have a laptop ready for an onsite interview.

All these companies expect you to write code using a site like hackerrank during the phone screen. And yet they can't have a chromebook open to the exact same site during the onsite? If that's too much, they can't even get a cheap laptop, stick in a USB drive linux distro, open nano and give that during the onsite?

<comment><USER48> I totally agree: the excuses are poor and companies don't care. There isn't much candidates can do. Hiring managers need to fundamentally change how they approach hiring.

</comment></comment></comment></comment><comment><USER49> One of the most telling questions I ask when interviewing is "what was your favorite or most complicated project you've worked on and how did you solve the problems you encountered?" I don't need the interviewee to show me a perfect implementation of a red/black tree. Hell, I don't remember it. But I want to know what they found challenging and how they went about solving it. Often times they will give me all the technical details I would have asked for, but in context. I don't care that you know the exact name of that one function in PHP but if you mention it off hand in relevant context, I will place that much more trust in the fact that you actually used it instead of memorizing it.

Other questions I ask are things like do you work on your own projects and tell me a bit about the tech behind them, what's your favorite pieces of open source or free software (I work with almost exclusively FOSS stuff though my own work is not FOSS), what do you do for fun, and if we were talking on the phone and I had never made tea, walk me through the process (this is a fun one). Then take them out to lunch if possible and watch how they treat service workers. I think this has so far worked well for everyone involved. At least as far as I know nobody I says yay to has been a bad hire.

<comment><USER50> I'm not sure why you're being downvoted for asking about their previous work and watching how they treat people.

<comment><USER49> Eh. I don't care about downvotes. But yeah if people want to throw the same ten dozen puzzles at each other and then wonder why the people don't work out, it's definitely not time to try anything different :)

<comment><USER10> @eximius,

> service workers ...

> watching how they treat people

They are not people though. But yes, agreed that we should treat them well

</comment></comment></comment></comment><comment><USER51> Personally, the anxiety of whiteboard "coding as performance art" makes my brain freeze. I see it as a broken process. It certainly selects against me.

I think you can learn more useful stuff about a programmer by showing them a function prototype and asking how to black-box test it. Then show them complete code, and ask them to white-box test it. Test cases or a test plan, depending on scale. Then a discussion of test tools they have used.

The people who can produce quality code for your organization will show up pretty fast.

<comment><USER52> In higher education there has been a lot of walk back from tests like the GRE in most fields. It turns out, being good at the GRE only correlates to your ability to be good at taking the GRE, and doesn't correlate with actual graduate school performance or outcomes.

Performative interviews occupy a similar space. You spend all this time and effort studying for something at is ultimately irrelevant outside of the interview. Just a pure waste of brain cells and calories, with a few new gray hairs for you.

</comment></comment><comment><USER53> If I were a large company with money I would simply ask all qualified applicants if they would be willing to work on a small part of a real project as a contractor with the possibility of getting hired. Applicants get paid for their time, productivity is achieved, jobs might be offered, everybody is happy.

<comment><USER54> Sure - That would neatly eliminate majority of qualified candidates :-)

Very few people who are successfully employed would take a risk / have a luxury to forsake their certain employment for an extremely short term contract (with all the registration, taxation, regulation etc headache this implies!) and extremely limited chance of gainful employment.

(it's perspectives like these that make me realize how much of a grouchy old man I am and have always been - always being a buzzkill by thinking things through next 3 steps! :P)

<comment><USER29> >with all the registration, taxation, regulation etc headache this implies!

It's not if you are in the US.

You will get a 1099 from where you worked, you write it on your 1040 and include the 1099 with your taxes if I recall from when I was an independent contractor for a while.

It takes maybe like 10 minutes extra and 1099-ing is incredibly common nowadays (unfortunately) so not a super weird case.

And it's not regulation. You don't need to worry about anything besides getting the 1099 like you would your W2s and appending it.

<comment><USER55> What if you get a 1099 from another state? Multiple states? Don't you need to file taxes with those states now? (Genuinely curious, I am not a CPA so I really don't know).

What if you don't have any other 1099s (so you have to pay self-employment tax just because of that one interview, at least I had once when my moving expenses were paid with a 1099).

Not even getting into possible illegal work for the candidates on work visas.

<comment><USER56> My city considers 1099s as businesses due to some shady contractors in the past. So my local government would try to collect a chunk of any contractor earnings (with that chunk getting huge if you didn't register with them first).

<comment><USER53> What city do you live in? I've never heard of 1099 tax collection on a city level before, is that common?

</comment></comment></comment></comment></comment></comment><comment><USER57> Companies like google and facebook are paying 200k for entry level engineers. They aren't going to have any issues getting people to apply even with the whiteboard interviews.

</comment><comment><USER58> I prefer the take home text. Less stress and I can really show what I know. Sure it's time consuming, but I always try to have fun with it. If I get a take home test, and it's writing some code, I know I have a shoe in the door.

I interviewed at one of these companies in the list. I'm nobody special, did the test, did the interviews which were more about doing the work and how it gets done and I how I think about it, and they made me a really good offer which I accepted.

</comment><comment><USER59> What you really want to know is whether the take-home assignment really decides anything, or whether it's subjectively evaluated. Is there a rubric? Or is it the basis of an interview? A lot of take-home assignments are cargo culted; the people offering them don't have much faith in them, and still select candidates based on interviews and resume screens (or "pair programming" projects that are really just multi-hour or multi-day interviews).

I have a hard time believing that this many firms have take-home assignments that really matter, and aren't just another hoop candidates (or, maybe, disfavored candidates) are forced to jump through.

<comment><USER60> I've gotten to the point where it's simpler than this. Firms get 16 hours to evaluate me. If the take home isn't a huge component of that I know they've cargo-culted it. I won't do it if they can't promise that isn't the case.

I say this as someone who was an extremely early adopter of take home tests and think they are the highest signal filter you can reasonably have in the US.

So many places have added take homes in the worst possible way without removing all the other nonsense. It's as if orchestras started demanding high pressure auditions and then threw out the results if the person talked a good game in an interview.

</comment></comment><comment><USER61> Identifying top tier tech talent is not that hard. Identifying yourself as a non top tier employer and setting your hiring bar appropriately - that is hard.

If you're only hiring candidates who can pass Google interviews, what are you offering that Google doesn't? (Hint: "we give you a 10 hour take home, then ghost you" is not a competitive advantage.)

</comment><comment><USER62> I agree that the cliché whiteboard interview is just awful – I've had my share of terrible ones. However, there is one form of whiteboard interview that I really like – it's where the interview is about system design and architecture rather than coding problem nonsense.

I think the first time I saw this was at Google, where the question was pretty much "We are going to design Google Maps from scratch – what does it look like?". The interview is about evaluating the candidate's ability to ask sensible questions when designing a system, checking that they know how to analyse tradeoffs, and understand where they need to think about issues like performance or scaling.

I liked it so much that I've built it into our own recruitment process for our most recent role – it's a process that we go through a lot, even when making changes to existing systems. We do it as a one-hour long collaborative system design process with a short specification for a system that we want to build. We let the candidate lead the design, and help by providing feedback, discussing any ambiguities, and asking questions about specific areas of concern. It's been really interesting to see how different candidates have approached the process.

(Anecdotally, the biggest indicator of a good candidate so far has been the use of abstraction rather than concrete technologies - i.e. "I would have a queueing system here and an object store there" rather than "I would have Kafka here and push to S3 there").

<comment><USER10> I like these interviews, seem like fun.

> a system that we want to build. We let the candidate lead the design

They never got upset and thought they were working for free?

If you were to use their designs and ideas, then what if they later claim you've infringed on their IP?

</comment></comment><comment><USER63> I think the problem is similar to the problems in systems engineering, where techniques and processes can de-rail real engineering progress. This has been observed back in 1969 by Robert Frosch, who served as NASA administrator:

"I can best describe the spirit of what I have in mind by thinking of a music student who writes a concerto by consulting a checklist of the characteristics of the concerto form, being careful to see that all of the canons of the form are observed, but having no flair for the subject, as opposed to someone who just knows roughly what a concerto is like, but has a real feeling for music. The results become obvious upon hearing them. The prescription of technique cannot be a substitute for talent and capability, but that is precisely how we have tried to use technique."

\- [http://origins.sese.asu.edu/ses405/Additional%20Reading/Fros...](http://origins.sese.asu.edu/ses405/Additional%20Reading/Frosch%20on%20System%20Engineering.pdf)

Current technical interview processes with its whiteboards and technical questions will undoubtedly lead to hiring the first type of music student.

</comment><comment><USER64> I am probably one of the very few who prefers whiteboard problems compared to knowledge based problems. In my career spanking over 12 years, I have worked on ASP.NET, Silverlight, Blackberry, Android, Node.js, HTML/js/css, Xamarin, .NET Core, Code generation, Java microservices , Native Windows and AWS services.

The result of working on so many things is that I am not an expert in any of those topics. While I do well in generic system design interviews, I lack depth in literally every single thing that I have worked on. So inspite of having more than a decade of experience, I don't satisfy the criteria for companies who ask for 'x' years of experience in 'y'. So whiteboard and algorithm problems work well for me, because there's not really much to learn and memorize in those topics.

</comment><comment><USER65> This seems to promote Google Docs > whiteboard, what? At least you can brainstorm or draw diagrams on a whiteboard.

IMHO the worst of these options is coderpad/hackerrank/etc. - it's robotic, discourages pseudocode, and perpetuates the idea that rote memorization is required to pass an interview.

<comment><USER66> Brainstorming with a candidate is great, when you can get them to let you in to their thought process. It easily can turn a technically incorrect answer into a strong answer.

</comment></comment><comment><USER67> To a large degree, these look like no-name companies who probably have to resort to this kind of thing to make up for worse pay and benefits as compared to FAANG.

<comment><USER29> "oh no, this 'no-name company' is _only_ going to pay me $100k+ / year while I live in a low-cost of living area, boo hoo hoo"

Scrolling through there are a lot of high-quality companies (Auth0, ASOS, Blue Bottle, Basecamp, Buffer, Canonical, CircleCI... and more and more).

Just because Google or Microsoft aren't on the list doesn't mean these are backwoods operations. Those companies don't need to bring in new candidates by being on these lists.

<comment><USER28> /raises hand

And Elvie! (If you make it to 'E')

We're hiring right now :) [https://elvie.workable.com](https://elvie.workable.com)

</comment></comment></comment><comment><USER68> A thought I had recently was to shift from having candidates write code to have them review code. I'm sure it's not an original thought, but it comes with a few advantages. 1. Reading someone else's code is usually harder than writing your own and in most projects you spend more time reading code. You're seeing how they'd handle the more common work. 2. You can still pick how deep you want to go in the interview. Like some of the algo problems where interviewers keep adding new requirements, you can discuss API design, performance trade-offs, other code qualities. 3. You get to see how respectfully they can discuss a future colleague's work and how well they can communicate their own ideas.

<comment><USER69> Some of the very productive developers I have worked with are not great at code reviews. It's a different skill from writing code and not everyone who is good at coding is good at reviewing other people's code.

Considering most of us spend about 90% of our time coding and a very small amount of time doing code reviews, your best signal in my experience is just giving candidates something to code.

</comment></comment><comment><USER70> As somebody who has been interviewing recently, the format I've preferred is from TestDome. I've never used leetcode or others but I imagine they're similar. I had 4 coding problems for which I had to pass a set of test cases which were explained from a high level. The first 3 were fairly easy and had time limits of 10 minutes or so. The last one was more involved from a design and functionality perspective and had a limit of 45 minutes. Then, it was followed up by a conversation with a developer where we discussed the problems and approaches. The time commitment was finite and reasonable, and the opportunity to explain my code was appreciated.

</comment><comment><USER71> The best interview test I've done was a take-home test with a fixed time window. I was given a working development environment with an existing app and asked to fix some tests, implement new tests and then implement a few new features with tests.

It was good because:

1\. There was no faffing about to set up the environment 2\. There was some small, easy wins at the beginning which got me engaged and thinking about the task. 3\. It was fixed time - I had to specify when I would start and then the test was emailed to me 15 minutes before that time. I then had 4 hours to return it via email.

This solved most of the problems I've seen with take-home tests - namely that they take far too long and ask the candidate to do all sorts of pointless admin tasks before they can start coding. Unless the candidate happens to be super familiar the exact environment of the test (including setting up a new project - how often do most people start new projects?), you're either making them waste hours of time before they can start, or disregarding good candidates.

Ultimately you want to find candidates who can solve problems and understand software architecture. Unless you're interviewing for a devops position, making them spend hours setting up a build environment seems like testing for the wrong thing.

</comment><comment><USER72> I haven't seen portfolios mentioned yet, but that's another option. Pros: do it once, use it at multiple interviews. Cons: more difficult to make sure it's original work, difficult to come up with ideas. I did CV screening and very few people had anything, it's not like a portfolio project needs to take more than 2-3 hours to create.

Portfolios are a common requirement in other fields where creating a portfolio is even part of the educational process.

<comment><USER73> What kind of portfolio worthy projects take less than 3 hours to create?

<comment><USER72> Scraping Kickstarter projects with certain keywords and sending yourself an email, a tool for slicing csv files, reimagining various basic command line tools in new languages, a tool to scan a file with an unused filename in the current directory, a program to sort music by idv3 tags, an http media server, etc...

All of these would show 1. basic language ability, 2. some simple algorithms, 3. ability to do something to completion.

</comment></comment></comment><comment><USER74> I just started working at a company that didn't do any whiteboarding or technical challenges. As far as interviewing me goes, they broke all the "rules" that I used to follow when interviewing candidates at my old job.

So far, so good. They seem to have an attitude of seeing the best in everyone, and everyone seems talented. Maybe I can learn a thing or two from them!

</comment><comment><USER75> Maybe we should give options to candidates. If I were to hire software engineers, I would offer every option to them:

1\. Whiteboard interview

2\. Opensource contributions

3\. Take-home assignment

4\. Becoming contractors for a short period

5\. Etc

So if they fail whiteboard interview, they can redeem themselves with their opensource contributions. If they don't have opensource contributions, then they can take home our assignments. And so on.

So it's fair (I think).

<comment><USER76> That's not necessarily a bad approach aside from the cost to the hiring organization to maintaining so many workflows. It lines up with, e.g., Triplebyte's and Google's studies about how a candidate's success in a role is tied much more closely to the moments where they impressed you the most rather than any kind of minimum bar you need to meet or some kind of average performance. By giving the candidate the opportunity to impress you however they're best able to do so you could plausibly get a stronger signal.

</comment></comment><comment><USER77> A. this is a good list. B. Lets not forget that whiteboarding is not all evil. If used correctly, it can filter out a lot of people that don't have good CS fundamentals. By fundamentals, I don't mean inverting a binary tree. At the end of the day, as a technical manager, the goal is to hire smart+hard working people.

<comment><USER52> But are you really hiring smart+hardworking people or just people who are really great at working on a whiteboard?

</comment></comment><comment><USER78> Here is how to do a tech interview that (I think) will work for everyone. I am sharing in the hope that this will become the standard, it was certainly the most pleasant technical interview that i have had, and somewhere where our industry is badly broken.

I was asked to bring in my laptop with some of my own code - I had a side project at the time, so no problem. We talked about the code, decisions I had made. I was asked to implement a simple feature. I did.

No time consuming take home exercise. No whiteboard riddles. No need to have a github full of work (lets face it most peoples work is in house and not easy to show off). Code I was familiar with, so less stress. The only downside would be not having code available, but then it will be the equivalent effort of a take home exercise to produce something.

<comment><USER79> > I was asked to bring in my laptop with some of my own code - I had a side > project at the time, so no problem.

Seems like a good solution _if_ the candidate has a side project. However, many candidates might have their majority of the "good" code they've written owned by their previous company.

> I am sharing in the hope that this will become the standard

Hopefully it can become a standard _option_. There's probably never going to be a standard technique which fits for all candidates (or employers).

<comment><USER52> >However, many candidates might have their majority of the "good" code they've written owned by their previous company.

Which is a huge problem in this field that stymies wider innovation in technology by limiting discussion around technology through literal gag orders. You should be able to reuse whatever you write. It's silly. I'm not stealing cash out of the register, the company doesn't lose anything from me keeping a copy of my code snippet around for future reference. It's not the whole codebase, it's just what I've contributed. I swear, lawyers have ruined technology.

</comment></comment></comment><comment><USER32> Programmers change jobs very frequently. They come and go. So why are we so obsessed with "getting it right", as if hiring a programmer were like signing a 30 year mortgage?

Hiring is a crapshoot. You'll make bad hires. That's an unavoidable fact of life. Professional sports teams spend millions of dollars on talent evaluation and still get it completely wrong all the time.

No, a bad programming hire is _not_ devastating to your company. You'll live. A bad CEO hire could be devastating, but you can't do a whiteboard or a take- home test for that.

"The perfect hiring process" is classic premature optimization.

</comment><comment><USER80> I once applied for a company here in São Paulo where after the technical interview (no whiteboard) I talked on-site with the CTO and he passively- agressively asked a bunch of random questions and tried very hard to be a dick about it ("oh, you consider yourself a good self-learner? French? I bet you couldn't read some news in French if I opened any here right now. JUST KIDDING")

At the end of this 40 minute nightmare I had a bizarre homework to do at home that involved writing a list of uses to a paperclip.

They took a month to send me an email saying I wasn't accepted.

<comment><USER81> IMO you should have ended the interview the moment you detected that you were being treated poorly. Let alone not doing the homework.

</comment></comment><comment><USER42> Can someone explain to me why isn't it ever part of the discourse to talk about certifications?

Most of the software engineers are certified right from the start when they graduate in university. All these people hold a certification in their hand, namely their diploma which proves that they are actually deemed suitable to hold any software engineering job in the industry.

The whole hiring process could be replaced with a timely renewal of accreditation. Then anyone who passed accreditation could be deemed as passed the technical interview. And then case closed.

<comment><USER34> The best programmers i've worked with don't have degrees in computer science, or anything like that. Metallurgy, civil engineering, mathematics.

On the other hand, i've worked with plenty of people with CS degrees from respectable universities who were useless. A degree is not anything like a certificate, and universities simply aren't in the business of actually deeming anyone suitable to hold any software engineering job in the industry.

Even if you fixed this, so that somehow every competent programmer, and only competent programmers, had some certificate from a university, all it would do is let you hire graduate developers easily. It wouldn't let you sort the wheat from the chaff at senior level.

<comment><USER42> You completely missed the second part of my post: "The whole hiring process could be replaced with a timely renewal of accreditation."

The accreditation would be available to anyone. Just like anyone can become a lawyer in Japan for example. You don't need to hold any diploma for that. Although it might increase your chances to get accredited if you have formal education.

<comment><USER34> Fair enough, that does address the point about hiring seniors.

But if universities can't currently usefully certify people for fundamental development ability, after having them on campus for years, i don't see how they'd be able to accredit seniors for the more complex skills they need.

</comment></comment></comment></comment><comment><USER82> When there are hundreds of applicants for 1 role of which the top 10 are all equally qualified, no interview process will seem fair.

So how do we break the tie?

Whiteboard? No whiteboard? Pair programming?

If we accept that after a certain point the process is random, why not simply break ties with first come first serve?

In the final few rounds, is a candidate who can solve the exotic DP problem just because he's seen it before really any better than the others?

</comment><comment><USER83> I feel a better title could be "Hiring without CS trivia". Whiteboard is the most powerful tool I know to brainstorm high-level ideas quickly, such as timelines, diagrams, mock-ups, etc. What sucks is when the whiteboard is used as IDE and the candidate as a compiler.

</comment><comment><USER84> It's a funny cultural thing. Here is a caricature:

Hiring managers have a real problem to solve: "how do I hire?". They don't have the time to conduct actual experiments and work out what actually relates to better hiring. So they trivialise the solution into some assumption about how the information captured in an interview generalises to overall suitability. They then proceed to defend the process and it's results as if it was state of the art because bad hiring = bad management in the politics game.

To me it feels obvious that whiteboards, pair programming , take home tasks, panel interviews , etc are all subject to being highly flakey and highly presumptive not least because repeatable processes are just hard to enforce and train people on.

What really works is survival. Practically every company has a probationary period but it's rare that once you're in you'll get bounced out in my experience. IMO bite the bullet, over hire and then collect the cream based on evaluation of the total output and the teams impressions in the probationary period ...

<comment><USER69> > IMO bite the bullet, over hire and then collect the cream based on > evaluation

That sounds at the least unethical and perhaps illegal in some countries. You're messing with people's lives, their ability to pay their mortgages, support their children, and other commitments they might have. While hiring mistakes sometimes happen, you should not be using firing as a tool to make your hiring more cost effective.

And I also think word would get around pretty quickly not to apply to companies who did this. Per the article we are already making lists of who uses ineffective whiteboard tests. I very much doubt any company wants to be on the list of "hires you and then fires you instead of doing effective interviews".

<comment><USER84> Applicants should know what the process is; I'm not sure why you'd assume that would be kept from them? It's perfectly normal for hiring / firing sales, traders, and other performance oriented professions.

Nothing forces people to apply ... mortgages, children etc.

It really depends on whether as an employee you want a meritocracy or an easy time. Personally, I wouldn't be adverse to this sort of practice; and companies like Valve do something significantly more extreme.

<comment><USER69> > I'm not sure why you'd assume that would be kept from them?

Where did I assume that?

> nothing forces people to apply

Agreed. I already predicted that the reputation gained from this practice would have talented developers avoiding a company involved in such a shady way of hiring. I'm damn good at what I do and "we will fire the ones who don't work out to avoid doing interviews" is a huge red flag for me. You lose, not me. Companies are competing for developers; experienced developers don't need your company.

> it really depends on whether as an employee you want a meritocracy

A meritocracy is great. Your company just lost at it. There are far better companies I could work for where the interviewing process isn't based on laziness now and fire people later. Want to win in the meritocracy competition between companies? Learn how to interview. I know how. That's why I can hire great devs. Haven't had to fire anyone ever in my career. So that's who you are competing with in this meritocracy. Companies who get the hiring process right.

</comment></comment></comment></comment><comment><USER85> Best way to avoid this BS is to be introduced by an established insider to help bypass this.

<comment><USER81> There. That's it. All this is a far-from-optimal filter for the total strangers. Get a foot in the door with an established insider and you get right to the last interview(s). No WBs, no take-home projects.

<comment> non-entity Does this actually work in practice? From my understanding knowing someone might get you past the initial HR / ATS hell to a first round, but you mostly have to follow typical processes after that.

<comment><USER85> In some cases yes.

In my experience they were even apologetic "sorry, we have to do this, let's get over it".

Once the pull to get you in is in place - it will help to put a checkmark in internal bureaucracy checklist and minimize unrelated interactions.

</comment></comment></comment></comment><comment><USER86> I don't fully agree with the part of the title that reads

Companies that don't have a broken hiring process

Only because they don't currently do whiteboards. I feel like interviews in general is a tough problem

</comment><comment><USER87> I'm absolutely fine with whiteboards. What I'm not ok with is take home assignments. You are expected to work for free for 2-3 days on some project that will help no one.

</comment><comment><USER88> If you can't explain your thought process on a whiteboard you don't know what you are talking about.

Literally no company wants you to make actual production code on a whiteboard.

</comment><comment><USER89> I interviewed with crowdstrike recently and the interview was a whiteboard system design interview followed by 4 whiteboard lc easy problems.

So I'm not sure how accurate this list is.

</comment><comment><USER90> Is it simply a coincidence that people don't like whiteboard interviews _and_ believe they don't work? Seems pretty convenient to me.

<comment><USER42> Conversely: Is it simply a coincidence that people like whiteboard interviews and believe they work? Seems pretty convenient to me.

<comment><USER90> I don't like whiteboard interviews and I don't know if they work.

</comment></comment></comment><comment><USER91> Just curious, does work at these companies require any algorithm design skills or is it usual web/app/database development work?

</comment><comment><USER92> Are the STEM fields the only ones with asinine interview processes?

What does an interview as an accountant, sales or manager look like?

</comment><comment><USER93> They need to take Netflix off the list. All they do are algorithmic whiteboard interview questions.

</comment><comment><USER94> By no means an exhaustive list, so really just promoted listings.

</comment><comment><USER95> Thanks for this list. I'm going to need a new job soon.

</comment><comment><USER96> When the bar is low and interviews are easy, the bar is not only low for you, but also for your coworkers.

If the interview bar is too low, you will be exchanging hours of frustration by months to years of frustration.

</comment><comment><USER46> We need a similar list for companies without Agile.

</comment><comment><USER97> what's FB doing on this list?

</comment><comment><USER98> Adding my own personal experience.

At the end of last year I wanted to change jobs but I was paranoid about my raw algorithm knowledge. I've been a consultant for over 15 years. The last company I worked at for a decade. I've developed software given a specification, worked with teams to design applications, lead teams, presented to management of companies, and even was part owner in a company -- but I never was good at rote memorization.

I know my limitations and I know how to find answers.

So, for 4 months I practiced online programming problems, read interview books, and had my wife quiz me nightly. The nightly quizzes were whiteboard answers and I had to explain the solution enough that my wife understood.

In the end, I was interviewed at 4 companies: Daugherty Consulting, Google, Amazon, and Target. (For Google this was my second interview in two years. The first interview was a shock, I froze during the preliminary interview, and for two years contemplated if I'd ever quit my job.)

Daugherty never had me do whiteboard programming but did ask me some algorithmic questions. These were much easier to answer verbally. In the end I was told I didn't have enough experience in consulting working with large companies. (This was a bit of a shock but whatever.)

With Google, I never got past the first round. I felt very good with my solution coding in a Google Doc, but, they had wanted me to implement the Python bisect_left function. Instead I just used it to solve the problem.

At Amazon I made it onsite, but again, I failed to whiteboard a hashing function to their satisfaction. They told me it could have been overlooked if my architecture skills were stronger. They did complement me highly on my communication skills, which I appreciated. (I had worked for two weeks rewriting my accomplishments journal using the STAR[1] format.)

Target (where I work now), was completely different. I was given a choice of real-world-like problems to solve and a couple weeks to code. Two were pretty heavily algorithm/math-focused but the third was right up my alley -- implement a microservice backed by a data source and a different (potentially flaky) service. I took my time, wrote code I'm proud of, deployed it on Google Cloud, and explained my solution in detail to a Principal Engineer. There were still personality and experience questions (and I think also some algorithm questions) but nothing like my other experiences. It felt much more grounded in reality. Are you a solid developer, good communicator, and good fit for the company. In the end I didn't get the exact position I applied for but I'm still extremely happy.

My takeaways:

1\. Maintaining an accomplishments journal as more beneficial than I could ever imagine. I write down everything I'm proud of - when I'm proud of it even if it seems minor. I can always delete it later. Also, the STAR format is actually really good.

2\. Don't stagnate in learning. Technology and methodologies are changing all the time. I don't follow every fad or code in my spare time but I feel strongly taking some time periodically to maintain a level of expertise is a good investment.

3\. Knowing my strengths and weaknesses really helped me focus while preparing for my interviews.

4\. Learning from interviews and maintaining confidence was big for me. I took notes immediately after each interview of what I wanted to work on. I asked for as much feedback as I could get. These notes made it back to my journals and are things I'll refresh time-to-time because I know nothing is a given. Who knows what I'll want in another 15 years.

[1] [https://en.wikipedia.org/wiki/Situation,_task,_action,_resul...](https://en.wikipedia.org/wiki/Situation,_task,_action,_result)

</comment><comment><USER99> These companies might also not pay very well (like fb, google, etc). Something to keep in mind.

</comment><comment><USER100> How can more companies adopt this and give candidates more take home tests rather than leetcode

<comment><USER101> I really don't understand the idea that take home is somehow more fair or accurate than whiteboards. I've judged and taken both whiteboards and take homes. Take homes are more annoying for the candidate, and in my experience much more subjective from the interviewer. In my experience take home is the absolute worst way to pick a candidate.

<comment><USER102> When I do a take home I have access to google, StackOverflow, etc. I can spend 15minutes on it, then pause to do something else (while I still sort of think about the problem), get back to it for 30 to 60 minutes, rinse and repeat. Which is exactly how I work in real life.

Take homes can actually be compiled and executed against a test suite for correctness.

Take homes tend to be more realistic exercises than balancing a f-ing tree or walking some idiotic graphs.

<comment><USER103> When I interview (and yes, there's whiteboard time), I tell the candidate up front and very clearly, "I will be your Google. I know we don't work in a vacuum"

</comment></comment></comment></comment><comment><USER104> It's definitely important to prove you understand the core concepts required to be a successful developer, such as using efficient & scalable approaches to solve a problem, but writing quality code from scratch simply doesn't mix with high anxiety. And it's not representative of how programming work is done in the real world.

I think a good middle-ground is pair programming where the interviewer writes the code based on the pseudo-code provided by the interviewee. This way, the interviewee displays their thought process and communication skills without being bogged down by programming language nuances.

<comment><USER10> Interesting idea

</comment></comment><comment> non-entity Man I've said it before, but I need to get the hell out of this industry. Its pointless to argue against the whole whiteboarding / leetcode process, it exists for a reason, but I'm quickly realizing everything about the industry, including the industry process is just incompatible with the way I work.

</comment><comment><USER105> I am going to kill myself (exaggeration). I have maybe one of the best cv's (I am not saying that, it is feedback from industry) for last year student focused on low-level programming. But I cannot bypass recruiters. Why? because none of them even heard Boost.Beast or anything on my cv. Or any C++ thing I have done as project or my publication. They literally do bunch of find/search on the pdf. And because of that I am unable to get any internship, particularly now which there is no meet up or in person networking.

P.S. I used to have extremely successful in person networking before COVID19.

<comment><USER106> This feedback is well intentioned, so please don't take it as an attack.

As much as we'd rather it not be the case, even for programmers writing skills are incredibly important. This comment is contains a few grammatical errors and is quite hard to read - I checked your comment history and many others suffer from the same problem. As some who hires, if I read a job application with this quality of writing, I'd probably dismiss it.

If you're having trouble finding a job, it may be helpful to have someone read over any written material you submit alongside your CV (cover letters etc) just to make sure it's clear and easy to understand.

<comment><USER105> I _really_ apricate your feedback. Just one thing you should take into consideration, that is, as ESL I am not going to put that much effort on writing comments on HN to check for every grammatical error. I simply don't have that much time and energy.

I do understand what you say. But that is not the case sadly.

<comment><USER107> Keep in mind that people reading your comment might not have that much time and energy either.

<comment><USER105> Completely valid point and shows my shortcoming. I hope I will improve it. But with COVID19 and danger of going homeless, it has become hard for me to manage it all. But you are _completely_ right.

</comment></comment></comment></comment></comment><comment><USER108> We use a simple 2 step process that allows both the company and the candidate to have more certainty about the opportunity. This only applies to technical jobs.

1 - First interview. Does the candidate know about the company? Does he know anything about the business domain? Conversation about problems he has encountered in his past experiences. it works like a knowledge sharing conversation, you really get to know how well things have been thought of.

2 - Freelancing period of about 3 weeks, 20h/week - You get to know the technical skills, cultural fit, communication skills and all other aspects that you only arrive to practicing it.

So far it has worked pretty well for us - we've hired about 11 developers and refused/have been refused by 5. It's an empirical process that works for both parties.

<comment> non-entity Are all your candidates students or unemployed at the time of interviewing? How do people find that 20h/week?

<comment><USER108> the average is 15h-20h per week. Imagine 2h each work day + weekends. We don't enforce these hours, if they want they can work more or less.</comment></comment></comment>


Tesla Has Been Giving Some of Its Most Loyal Customers the Runaround - <AUTHOR> https://arstechnica.com/cars/2019/04/elon-musks-erratic-decision-making-leaves-loyal-tesla-fans-frustrated/ <comment><USER2> I'm one of the suckers who paid for FSD 2.5 years ago. Elon said "6 months, max" back then.

It's pretty clear that we are still years away.

IMO the only right thing to do to make things right is to refund our money for that feature, apologize, and still upgrade us to FSD when it's finally ready. It sucks, it's expensive, but that's the cost of making quick decisions and over-optimistic announcements.

Disclaimer: I'm a major Elon fanboi.</comment>


Blockchain and Serverless: A Marriage Made in Cyberspace - <AUTHOR> https://www.coinspeaker.com/blockchain-serverless-marriage-made-cyberspace/ <comment><USER2> I'd love to see a serverless product that could be paid for in cryptocurrency for this reason, with prices quoted in crypto currency (even if fluctuating based on exchange rates and actually pegged to fiat.)

<comment><AUTHOR> You could just use a crypto debit card as your method of payment for the FaaS service: [https://cryptoslate.com/cryptos/debit- card/](https://cryptoslate.com/cryptos/debit-card/)

<comment><USER2> I was thinking more so an AI could interact directly.</comment></comment></comment>


Why don't fish freeze to death in icy water? - <AUTHOR> https://www.global.hokudai.ac.jp/blog/why-dont-fish-freeze-to-death-in-icy-water/ <comment><USER2> Well, that's a funny thing, _some do_, or at least to the point where it makes no odds. I was touring around Iceland (a country I can highly recommend visiting) and we were at Jokulsarlon [0] (where the icecap carves into the ocean) and there were lots of what looked like dead fish there. They basically swim in from the ocean, seemingly catch hypothermia, and are a feast for seals.

The local "fishermen" came along, with wellies on, waded into the water, literally kicked the fish out onto the shore. One of the fishermen was kind enough to give us one of their catch. It was tasty.

Go to Iceland, it's awesome.

[0][http://blog.psdavey.com/2014/08/25/down-the-coast-to- jokulsa...](http://blog.psdavey.com/2014/08/25/down-the-coast-to-jokulsarlon/)

<comment><USER3> Having been convinced by my classmates to jump in with them, I can confirm the Jökulsárlón lagoon is very cold!

</comment></comment><comment><USER4> My dad says when he went ice-fishing with my grandfather they would put caught fish in a bag in the trunk. The fish would freeze on the drive back home but as they thawed back in their kitchen they would come back to life and start flopping around. Others have reported this, too (1). I can only guess their interiors were not frozen, and just in some sort of dormant state.

We have a small fishpond in our backyard just outside of Boston (kind of like this one, but a little deeper: [https://www.pondmarket.com/epdm-pond-liner- rubber-liner-coat...](https://www.pondmarket.com/epdm-pond-liner-rubber-liner- coatings-sealers/royal-coachman-preformed-pond)). We have a few comets and shubunkin. Every winter the fish go dormant, usually around late October or early November, sinking to the bottom half-hidden by dead leaves. The top eventually ices over a few inches. We keep the small pump going and try to break the ice bubble if it freezes over, but otherwise the fish are not active for 4-5 months (with one exception, described below), until warmer weather comes in mid-April. It's an amazing thing to see them slowly reactivate and start to take food again.

One winter the pond iced over and they were still swimming around a little (2) which I can't explain. It's the only time we've seen this in the past 6 or 7 years.

1\. [https://biology.stackexchange.com/questions/1661/fish- coming...](https://biology.stackexchange.com/questions/1661/fish-coming-back- to-life-after-being-frozen)

2\. [https://www.instagram.com/p/BAR4PG8pJrn/?taken- by=ilamont](https://www.instagram.com/p/BAR4PG8pJrn/?taken-by=ilamont)

<comment><USER5> When they thawed in the kitchen, were they perchance on a metal surface, or salted? If so, what you're seeing isn't resurrectionist fish, rather electrolysis inducing a galvanic response in the deceased nervous system.

<comment><USER6> I think it was more like this: [https://www.youtube.com/watch?v=zOJZaGUx9JM](https://www.youtube.com/watch?v=zOJZaGUx9JM)

<comment><USER5> Huh. That's a new one to me. This quora link sheds some light on what may have happened in both situations:

[https://www.quora.com/Is-it-possible-that-a-fish-is- complete...](https://www.quora.com/Is-it-possible-that-a-fish-is-completely- frozen-comes-back-to-life-after-submerged-in-liquid-water)

</comment></comment></comment></comment><comment><USER7> Hmm, interesting. There was a pond in front of the office at Denver which was created when construction diverted some of the Platte river, and it had 3 or 4 fish 'stuck' in it. In the winter the water froze solid and I walked over and saw one of the fish sitting there frozen in the ice. I was tempted to try to chip it out but it was too deep and the ice too hard. About 3 weeks later the ice melted and the fish was swimming around again. It was pretty amazing to see.

<comment><USER8> Depending on the depth of the pond, it may have not frozen to the bottom - water has the highest density around 4°C (~39°F).

<comment><USER9> If the pond isn't very deep, and there's no streams or anything else coming into it that would oxygenate the water, usually the fish would die because of that. Happens all the time around here if water levels change drastically (say a beaver dam lets go), followed by a hard, cold winter with deep ice.

</comment></comment></comment><comment><USER10> Why would water (inside the fish) freeze in water (outside the fish)? I mean, the fish are swimming in the liquid form of water which apparently also doesn't freeze so why is it considered strange that they don't freeze? It would be strange if they didn't freeze while being in a block of ice, or if they would be dead blocks of fish-ice floating on liquid water. A non-frozen fish in liquid water is not strange, or am I missing something? The article doesn't really clarify it for me.

<comment><USER11> All bodies of water are not created equal.

The specific, and level of, solutes in the water determines the temperature at which it freezes. Fish are not perfectly permeable membranes, thus the internal freezing point of fish could be higher than the external freezing point of the water.

Fish are also cold-blooded and are at the mercy of their surroundings.

<comment><USER12> Or in other words you can freeze non-salt water in liquid salt-water; the salt lowers the temperature at which the mix freezes.

Human blood freezes at about -2degC. Brine can apparently go down to -20degC (source: Wikipedia).

<comment><USER13> > the salt lowers the temperature at which the mix freezes.

If you make ice cream the old fashioned way you throw in rock salt to lower the temperature below 0C.

You can also chill beer cans sitting in water very fast adding piles of salt to water.

<comment><USER14> You don't actually salt the ice cream. You put salt on ice surrounding the unfrozen cream. Salty ice cream (with the amount of salt necessary to freeze it) would be gross.

Available soluble salt ions lower the melting point of the water to the point where the water-ice would very much prefer to be salt-water at the same temperature. But a phase change requires energy, too. The energy required to melt the ice and dissolve the salt is taken first from cooling of the sugar- water in the cream, then from phase-changing it to solid crystals, then from cooling the solid-phase materials. If the setup has a paddle, that is mainly to transport heat from the center of the cream vessel to the conductive walls, and thereby regulate the size of the crystals.

Cooling beer cans in liquid water by adding salt does not rely on freezing point depression, but the heat of solution for NaCl in H2O, which also contributes slightly to the ice cream system above. You want cold beer fast? Use potassium nitrate (KNO3, saltpeter), as it has nine times the enthalpy of solution as NaCl.

<comment><USER13> I know since I've done it I've made ice cream that way, I guess I didn't explain it very well.

<comment><USER14> Some people have never made it that way, or any other way. Anyone who ever has made it that way would immediately understand.

What a hand-cranked ice-cream bucket might look like in ASCII diagram world:

                       |
                  o----o
      ____________|____________
      \\.salt..|<comment>~~|..salt.//
       \\######|     |######//
        \\#ice#|cream|#ice#//
         \\####|_____|####//
          \\#############//
           \</comment></comment></comment></comment></comment></comment></comment></comment><comment></comment><comment>=/

<comment><USER13> The one I have has an electric more in place of the hand crank.

I also used to make it by putting the pot of cream mixture into a deep freeze still in the metal pot. Then get an electric hand mixer and mix it while it sat in the deep freezer. I'd do that over and over then for what seemed like 100 liters of milk and cream and hours of work I'd get 500ml of ice cream.

</comment></comment><comment><USER15> I'm going to ask a staggeringly dumb question here:

Since none of the fish on Earth, which survive these conditions, live in zero- gravity, what is the reason for doing the experiment in zero-G? Given the immense cost of sending stuff to the ISS, there's clearly something about doing the work in zero-G that's either necessary, or highly advantageous.

Clearly the effect that keeps the fish alive works just fine under normal Earth gravity, or we wouldn't have an interesting question to answer.

<comment><USER16> In the article they say that convective flow does not occur in micro-gravity which makes it possible to accurately measurement the ice crystals as they grow.

<comment><USER15> That makes sense, thank you. If you're already removing the flow that comes from the fish's own circulatory system, you may as well remove the remaining (comparatively small?) effects of the convective flow.

</comment></comment></comment><comment><USER17> Article says fish survive in sub zero environments such as below ice. But is the water below ice sub zero? I thougt it was at least zero as long as its liquid (at close to normal pressue). Is that the salt?

<comment><USER18> > Is that the salt?

Yes. Oceanic salt water freezes around -2C. (or possibly below as surface water freezes "pure", the salt content getting transferred to lower layers).

</comment></comment><comment><USER19> Some fish have antifreeze proteins:

[https://www.nsf.gov/news/news_summ.jsp?cntn_id=132798](https://www.nsf.gov/news/news_summ.jsp?cntn_id=132798)</comment>


Marketing Reddit: Have an Awesome Product and Tie it to a Fantastic Brand - <AUTHOR> http://thestartupfoundry.com/2011/03/17/marketing-reddit-have-an-awesome-product-and-tie-it-to-a-fantastic-brand/

<comment><USER3> As a college student that is working on a web app start-up, what Alexis said really hung with me. The fear of failure is present for me. Before watching this video, I had never considered the fact that I have the luxury of nobody caring/expecting my startup to be successful. This thought takes some of the pressure of my shoulders, and allows me to do what I do best: Get shit done.

<comment><USER4> Please don't be held back by your own fear of failure! At the risk of looking like a totally shameless selfpromoter, I spoke about a related lesson that's just as valuable: no one gives a damn about your startup/idea/thing - <USER2> maybe your mom - so you've gotta expect to convince _everyone_ that it's worth their time; that means making something really compelling and always hustling to improve it & build an audience.

<http://www.youtube.com/watch?v=pdDGVhqMuWI></comment></comment>


Video of a Man Hacking a Ring Camera Inside an 8-Year-Old Girl's Bedroom - <AUTHOR> https://digg.com/2019/this-video-of-a-man-hacking-a-ring-camera-inside-an-8-year-old-girls-bedroom-will-give-you-nightmares <comment><USER2> This is the worst kind of low-quality clickbait.

<comment><USER3> It is exactly what it claims though, to be fair.</comment></comment>


Ask HN: What do you hate about social media? - <AUTHOR> <comment><USER2> (I don't hate social media)

One specific behavior I dislike though is the very public display of .. thanking? People often seem to talk to their followers rather than the person they seem to be talking to.

They'll say "had a great!!! time at XYZ with you @someone see you next time!!"

Like, cool but why don't you just tell them? DM? In person?

Seems to happen especially often between high profile personalities within the given domain.

But yes, I get it, you're both using twitter as a tool and this gives you more visibility together, still I can't stand it.

</comment><comment><USER3> Anonymity.

In large groups without clear individual accountability there will tend to hide bullies and those who practice bad faith. There will emerge other types of preventable violence that doesn't happen in real life face to face small groups of adults. It also allows for bots, state sponsored propaganda, and phishing.

<comment><USER3> saw this great summary on a comment about minecraft servers and griefers...

"The lower the cost to get in, the more people who violated trust. The higher the price, the less people who violated trust. When personal reputation was on the line, no one violated trust."

Which sums up the issue nicely.

</comment></comment><comment><USER4> Lack of anonymity.

I hate that communities exist in FB but I have to not only join FB to engage but with a real identity. Try creating a fake profile and after 2 weeks or so it gets banned. During that time you are pestered to add photos, friends, etc. what if I just want to buy a second hand chair?

<comment><USER5> I don't sell any second hand items to random anonymous people. I just don't want to risk those kind of interactions.

That's why I exclusively sell second hand items on Nextdoor or to people who have filled out, legit looking profiles on Facebook.

I know a ton of people who think the same way. So the market is effectively filtering out anonymity regardless of if the feature is available or not.

</comment></comment><comment><USER6> How they try and centralise everything, and make it too easy for people to brigade communities they don't like. Something that's federated would probably solve a lot of the issues Reddit has for example, as well as a lot of the controversy over Twitter's ban policy (by letting communities have their own guidelines that don't affect the entire service).

</comment><comment><USER7> It allows the people with the loudest voices to seem like the dominant opinion.

</comment><comment><USER8> tl;dr That some people are forced to use it in order to have a social life.

I was talking to my friends about social media, and how it can be bad for some people. I said, "Ok, all they have to do is turn it off". The response was general outrage, and it was pointed out that I already have my social circle and if I don't use Facebook then it is not as if that goes away. But for many people, if they are not on Facebook / Instagram / Snapchat / whatever, then they lose access to friends. Picture kids in high school who don't talk to each other in-between classes and coordinate everything via a social platform. So it is bad that some people can't just "stop using it" because if they do then they'll be left out of activities.

</comment><comment><USER9> government involvement</comment>


Seeking freelancer: European rails dev with data and analytics experience - <AUTHOR> Hey everyone,
we are a Milan, Italy based company looking to build custom conversion tracking tools and analytics panels. We want to build the infrastructure to analyze the data behind our business and drive our growth in the future.
I'm Alessio, I have startup experience in the digital tech space, I'm a developer myself and am responsible for the tech part of the company.
For the sake of an easier communication we are looking for a fluent english communicator based in Europe. I really look forward to having a chat with you.
https://alessiosantocs.typeform.com/to/pmihT6
Recap: Contract: Freelance Located: Everywhere in Europe Experience: Rails dev who loves data :) <comment><USER2> Company url?</comment>


A Large Scale Study of Programming Languages and Code Quality in GitHub [pdf] - <AUTHOR> http://rayb.info/uploads/cacm2017-lang.pdf <comment><AUTHOR> _From abstract:_

In this study, we gather a very large data set from GitHub (728 projects, 63 Million SLOC, 29,000 authors, 1.5 million commits, in 17 languages) in an attempt to shed some empirical light on this question. This reasonably large sample size allows us to use a mixed-methods approach, combining multiple regression modeling with visualization and text analytics, to study the effect of language features such as static v.s. dynamic typing, strong v.s. weak typing on software quality.

 _From conclusion:_

The data indicates functional languages are better than procedural languages; it suggests that strong typing is better than weak typing; that static typing is better than dynamic; and that managed memory usage is better than unmanaged. Further, that the defect proneness of languages in general is not associated with software domains. Also, languages are more related to individual bug categories than bugs overall.</comment>