On asking job candidates to code - <AUTHOR> http://philcalcado.com/2016/03/15/on_asking_job_candidates_to_code.html <comment><USER2> I'm going through the interview process this week at two different companies. One of the companies I found through an ad on Stack Overflow whereby they publish two programming "exercises/puzzles" with the opportunity to earn $100 for a correct answer to each puzzle ($200 total for two correct answers). While I spent more than eight hours on both puzzles, I found them to be thought provoking and I learned stuff in the process so I don't feel like I wasted any time and I got a check in the mail for $200. My code was reviewed by their dev staff so they had immediate feedback on my programming ability and high confidence to bring me in for an on-site interview. I like this process and consider it worthwhile.

<comment><USER3> What these companies have successfully done is weed out all the great devs who value their time very much. Those who are desperate for a job will spend the 8hrs it takes on these silly tests and make it thru the process. Good developers who can take their choice of jobs will just ignore these companies who think they are being clever to require a test, and just move to some other company what is not showing signs that they will be a pain in the ass to work for. IMO, the more difficult the interview process, the more of a pain in the ass the company is going to be once you're hired. If an interviewer is not capable of determining my knowledge level by a normal interview that tells me right away something up, and I probably don't want to work with them because they aren't too smart themselves.

<comment><USER4> _What these companies have successfully done is weed out all the great devs who value their time very much_

I'm not sure I understand this. A dev who valued their time very much would already be working at a job they liked enough not to be looking. As job satisfaction declines, time opens up to work on whatever you want to call this, a technical feeler, perhaps. A dev who values their time wouldn't get snowed under by a ton of these because they wouldn't be applying willy-nilly, only to those places they want to work.

<comment><USER5> Not for those of us who have experienced the pain of terrible management and burnout caused by this. The last search I did I was also working 50 hours a week under very poor project management. I had to rule out any prospects that involved meeting with recruiters or these screens. I got a job I liked.

</comment></comment></comment></comment><comment><USER6> A couple of months ago I did a 'homework' assignment for an interview (it involved writing a simple REST service in go, even though I have at least one personal project demonstrating exactly this skill -- to an even greater degree -- in my github profile). During the phone-discussion after turning in the project, the mid 20-something lead developer couldn't find anything to nitpick (I could have) and even told me he thought I was a better programmer than him. The conversation went very well and I felt confident I would receive an offer.

Now, what do you suppose happened next? _I never heard from them again_ and all of my attempts at communication were ignored. I'm starting to think my age is becoming a factor (mid 30s) -- and also these kids have no sense of respect and professional courtesy. I'd like to say this was an isolated incident, but that would be a lie.

Stupid me just spent his Saturday doing another such project, though at least this one presented a more interesting problem. I swear, if I find out I had my time wasted again I'm going into consulting. Or maybe I'll go to truck driving school. Or open a cafe. Screw this "employee" stuff.

The fact that so many employers treat candidates like this tells me that the whole "it's hard to find good developers" line is a _lie_.

<comment><USER7> I've noticed that your github profile doesn't even get looked at until you jump through all the standard phonescreen/homework hoops.

Even after than many interviewers look at your resume like 10 minutes before the interview there is no chance that they would read through your code on github. github as your resume is overrated.

I am curious to know if people hire or get hired purely based on their github profile without a technical interview.

<comment><USER8> A couple of quick notes:

* If you don't provide a link to your github profile, I'm not supposed to look at it. This an HR requirement. Other companies may use this same policy.

* I definitely look at github, but it's not always easy to tell if a project is original or not. Quite often I'm presented with either 30 forks, or a few projects that seem like they started as some boilerplate.

* Don't just provide github, also provide a specific project to look at. Even better, point out a particular section that you are proud of.

* Even with a well-stocked github, I will still ask you to code. This likely isn't to tell if you can code, but to figure out what it would be like to work with you.

<comment><USER9> My issue with the whole "github resume" thing is all my best code is in private projects because they are related to serious side projects I intend to monetize. I suspect the same could be said for any other developer who is serious about their side projects.

<comment><USER10> Your situation is just one of the many reasons for why hiring by GitHub is bad. It selects for people who are willing and able to work for free.

</comment></comment></comment></comment></comment><comment><USER11> I definitely wouldn't call myself an expert, but in my career I've probably interviewed a few hundred people. I think people way over-complicate this stuff by trying to be too clever.

To me, the most useful question is this: "here is an example project that we'd be likely to do, how would you build it?" Then you just let them walk through the steps they'd take.

From there you can launch into why they made the choices they made, trade- offs, philosophy, etc. When you hear people explain _how_ they would make something real, you start to learn a lot of things about them without even needing to ask, and so you're less likely to get scripted answers. They're also probably in their comfort zone at this point, so you get to see them in a more real way.

I don't necessarily mind "homework" style projects, but I tend to shy away from that sort of thing as it's not always respectful of the candidates time.

Two questions I usually avoid (but see getting asked all the time): puzzle questions, and language trivia. I don't ask puzzle questions because I've never seen it correlate to something useful. Most puzzles require an "aha" moment where your subconscious bubbles up some kind of answer, but the easiest way to short circuit that part of a persons brain is to put them in front of strangers with a time constraint. Puzzles might give you a clue as to a persons overall IQ and confidence, but it won't tell you a lot about how they can do the job.

Puzzle questions are also "expensive", in that they usually put a candidate on edge, and they set off the candidates bullshit detector. Let's keep in mind that we're going to reject the majority of candidates, statistically speaking, but you still want them to think highly of your company. If they feel like they're being rejected for a BS reason, you've just created animosity towards your company. It's important that the people you reject still feel respected.

I don't ask language or framework trivia, as I don't think it's useful. I might lob a softball about a framework just to make sure they've actually used it, but I'm not going to ask something hard.

<comment><USER12> +1000 This is how a proper professional does it.

I always have turned down homework assignments. It just feels insulting and disrespectful. Even whiteboard crap is useless but I'm OK with it considering they've paid for my travel expenses etc.

If you think about it ... homework assignments are a classic case of B's looking for C's. A's wouldn't need to resort to such silliness. They can identify (as an interviewer) and exhibit (as an A interviewee) with just an intelligent conversation.

</comment></comment><comment><USER13> Interesting. I applied to DO about ten weeks ago. I had an initial phone screen, and they sent me a code challenge. I completed and turned in the challenge. Then I heard nothing for two weeks. I reached out to my contact there, and she got back to me a week later saying, "I've been waiting on feedback - sorry it's taking so long! I just pinged the manager again this morning." That was the last I heard from DO.

I feel like I wasted the half-day I spent on their code challenge. I don't expect a job, but a simple "Thanks but no thanks," would be nice.

<comment><USER14> Nice. I had the same experience with DO last September...The same month Braintree asked me to write a ruby project that took about 3 days and then on the call only asked about python sqlalchemy database queries. I don't know what the disconnect is with these larger companies and their HR staff? By the time that third one asked me to write a node.js rest api from scratch. I just chuckled and said thanks for your time...I doubt I will commit more than a few hours in the future to any throwaway project for just a call back. Why can't we get paid for the time spent writing sample code like a freelancer...If it isn't good enough pay for my time (up to a max) and both parties can move on. Time is more valuable than the risk of getting passed that initial callback/more screens.

<comment><USER15> You should send them an invoice.

</comment></comment></comment><comment><USER16> Weird.

I've created from scratch: new CRDT data-types for geospatial problems, new highly-available transaction patterns that provide useful causal history and atomic visibility, large-scale messaging platforms (400k ops/sec), high- throughput machine learning pipelines, lock-free lightweight-process mailbox implementations, and an end-to-end build pipeline for generating bare-metal rumpkernels to be parallel deployed to N-number of Minnowboard Max devices. Among other things.

If you asked me to do homework to prove to you that I can "program", I'd have a brief discussion with you about how misguided that seems, thank you for your time, and walk away.

I mean fully "hundreds of connections" to a socket pool? Are you serious? If I submit a solution that can do "hundreds of thousands of connections" that includes QuickCheck tests for your protocol and my implementation, Ansible scripts for deployment and orchestration, and a Makefile that builds it, tests it, bakes it into a unikernel, and drives the Ansible scripts to push it out and start the end-to-end load test, do I get to be CEO or something?

Why is tech hiring so full of strange, meaningless obstacle courses? Do we just collectively not have better ideas?

<comment><USER17> Because there are a lot of shit candidates. You might know you're worth talking to, but try hiring and you'll see why people want to separate the wheat from the chaff.

<comment><USER16> I have tried hiring. Many times. As recently as yesterday.

Using a combination of (semi-)rigorous personality profiling (time constrained), and mapping the abstract problem domain of their prior work to the problems I actually have for them to work on, it's been possible to align new people with the work such that they're intrinsically motivated, and make sure all the bases are covered for the necessary roles (hacker, inventor, finisher, watchdog, etc.) to carry a very large greenfield project from start to finish.

I've never once given a cute riddle, puzzle, or coding challenge as a part of the interview process, and actively dissuade others from doing it as well.

<comment><USER18> Can you elaborate more on your middle paragraph? It sounds very interesting.

<comment><USER16> Sure. To start with there are many, many models for personality profiling. I tend to stick to "The Big 5" mostly because it's foibles don't sufficiently affect the outcomes I care about, and I've gotten proficient enough at it that with the right set of questions in the right environment I can typically get a pretty good read on at least 4 out of the 5 factors, and given a follow-up interaction I can usually clear up anything that looks like contradicting signals by getting additional context.

Like for instance, if you were to meet me, you would probably immediately peg me as high on extroversion. While understandable given the outward traits I project, you'd have needed to really thoroughly created a controlled environment during an interview (requiring a series panel, a day-long engagement, and an intentional highly-social event in the middle) to figure out that I'm actually introverted, but have just learned how to benefit myself by projecting extroverted traits.

Now the richness of any assessment is bound by time, planning, training, and practice. If you want to get really good at it, you learn to do it in realtime by assessing people you know well, assessing yourself, then moving on to acquaintances, and eventually onto strangers. In each step you need to be aware of where your assessments are drifting from the model. Like if you assess yourself or someone else as high on agreeableness because you find that you're very typically conciliatory or trend toward diplomacy and keeping the peace amongst your peers, but you also are prone to "blow-ups" where you drop the hammer on people in rare, but consistent, circumstances (when you've "been pushed too far"), then you're probably not as high on agreeableness as you think, and are probably higher on neuroticism than you think you are.

Given enough inputs and hypotheses you can get pretty good at this. Good enough to provide useful points for making decisions given otherwise limited information. More importantly you can get good at how you setup your environments and your interactions to get higher quality signals from your limited information. For instance, say I wanted to assess someone's "Openness" because I wanted to bring them onto a team where their primary responsibility will be in driving phases of invention. The simplest pass would be to just ask them about personal activities and look for signals of risk-taking. However, one needs to control for the difference between risk-taking due to openness to experience and risk-taking due to self-destructive traits. The former would be awesome for the job role and the latter will bring down the whole ship. To control for that you'd want to put additional emphasis on assessing neuroticism and conscientiousness. Doing that can be as simple as leaving a lot of awkward pauses in the conversation and see how they deal with it. Does it make them uncomfortable? Do they feel compelled to fill the air with something? What do they fill it with? Relatable stories, trivia about their work history, etc. When they talk about anything, not just work-related things (sometimes specifically not work-related things), what level of detail do they provide? Is it consistent? Is it always TMI, always 30k ft view, or can they move fluidly from one to the other?

Things like that. Different kinds of job roles require a different kind of personality profile to be successful, and more important than that, the composite of personality profiles filling those job roles is a requirement for a healthy team where each person is competitive with themselves, but cooperative with their peers, and has the right mix of people to own each phase of the development and business process.

<comment><USER18> Hey, thanks for the reply, your insights are amazing. Totally with you on the whole "being an introvert who learned the traits of extroversion" thing too.

I've played around with the Big Five model ages ago and found it to be a really good way to look at personality. I especially liked its usefulness for self-assessment - like you say it's easy to self-deceive yourself into thinking that you're much more open and non-neurotic than you really are, but so long as you are honest with yourself it can be a good tool in understanding your own strengths and weaknesses. I hadn't really tried practising it on strangers, though, that would definitely be a useful exercise!

I'm actually in the process of setting up a crowdfunding campaign for a personal project that I'm working on, and if that goes well then I'll be thinking about bringing in more people on board - hence why I asked you about this.

I have to say, the way you're going about this is really impressive in terms of methodology. Most high-level advice in this area that I've seen is full of hand-waving and vagueness, but you've got the whole thing down to a science. Especially the part where you talk about counterfactuals! So many people overlook that.

If you don't mind answering another question, I would appreciate more of your thoughts on this. You've said that you look for particular combinations of traits to fill in particular roles (hacker, inventor, finisher, watchdog, etc.) - what are the traits that you look for in each of them and how exactly do you envision each role's responsibilities within the company? (Is there a set number of such roles that you discovered or do you find yourself inventing new ones as the company evolves?)

Also, it sounds like you've been doing this for a while and probably have quite a few people on board, which would make the dynamic of hiring someone a bit different than if you were just starting out. If you were back to square one and had to consider hiring your first employee or even finding a potential co-founder, how would you approach that?

<comment><USER16> Sure, do you want to take this conversation offline? This is just going to get buried deeper and deeper and eventually become locked when it's stale enough.

<comment><USER18> Yeah, good idea! My e-mail is zans.lancs at gmail dot com

</comment></comment></comment></comment></comment></comment></comment></comment><comment><USER19> I really hate having to solve these kind of programming challenges. I've already sent you my CV, with references from former/current employers and co- workers. Call them up if you are worried that I'm lying on my résumé, don't waste my time by making me work for the chance of maybe getting a job as only payment. Doing multiple interviews already takes up enough of my time, seeing as I'm the one who has to commute to your location.

<comment><USER20> The problem is: Writing good resumes and having a lot of experience with programming is a good heuristic for "experience" and "knows how to play the game" but not meaningfully useful for "excellent at programming".

Work-sample tests measure the latter.

A worker's experience is a heuristic for programming skill, but a heuristic of a heuristic is like an average of a set of averages: Useless.

<comment><USER21> Just to play devil's advocate, being excellent at programming is also only a heuristic for working as a professional programmer.

<comment><USER20> Sure, but it's O(n) rather than O(n^2).

Hiring decisions are hard to generalize, because different companies have different needs at different times.

<comment><USER22> Which makes it extra frustrating that hiring practices are the most often cargo-culted practices in this industry. I feel like the root of hiring problems is that most companies know neither what they need or how to identify those features in a candidate.

<comment><USER20> Sounds like a lucrative itch to scratch.

</comment></comment></comment></comment></comment></comment><comment><USER23> I just want to create a well documented simple resful API for applicants to use. If an applicant wants to apply for a job they have to write code in the language of their choice to apply using the API. Upon successful application, request their SSH public key and supply a git repository they can check their code into. A hook script will email me upon successful commit.

A moderately experienced dev will not spend much time to apply, and frankly would probably enjoy the interface better than most job application websites. :-)

A lesser experienced dev will end up spending more time, but really if they are successful we would still want to look at them.

<comment><USER3> And the GREAT devs like me will not waste their time on your antics and will look elsewhere for a job at a company that is less of a hassle to deal with.

<comment><USER24> Great devs never apply for any job using the traditional job application (except maybe as an after-the-fact HR formality); they are always recommended and hand picked by their network.

As a intermediate (after so many years, should I just replace that with 'mediocre') dev I think a test like this would be a fantastic way both for a company to judge me and for me to judge what a company would expect from me.

There's a huge variety in dev roles ranging from challenging to code monkey and it's sometimes hard to figure out from a job posting or even conventional conversational interview where along the spectrum the expectations are, but establishing those expectations on both sides is critical.

<comment><USER25> That isn't true for all devs.

<comment><USER24> I shouldn't have said always and never. .. how about "most of the time"?

</comment></comment></comment></comment></comment><comment><USER26> The real problem here isn't coding interviews; its how we (hiring managers) approach the entire question.

The goals of a technical interview should be: \- Confirm the credentials presented aren't complete BS \- Validate they know enough to be immediately helpful \- Assess their long term potential (brains, social skills)

Coding challenges tend to be good at the first two points but very questionable for the third. Incidentally, given that I'm seeing a 50% washout rate from a basic SQL code question (supported by confessions from candidates that they fabricated credentials/experience), you would be silly not to include some form of practical test.

The third point (potential) is best measured by talking with them about a project they really cared about. For those who cannot share work examples, pick a good side project. Don't get hung up on the content - focus on their passion. You'll get a better view of my capabilities if you chat with me about building an ad server, mobile optimization, or SEO analytics (real world projects I care about) than with a contrived example.

As for the github profile of code _shrugs_ why? Seems like a bit of a cult to me. I'm happy to sharing anything that's useful (and do so on my blog) but don't have the free time to spend developing code for the sake of "work samples". My work samples run on production servers as commercial projects, thank you very much..... Suspect many other good candidates are the same.

<comment><USER27> "you would be silly not to include some form of practical test"

Yes, I'm coming to the realisation that it's idealistic to assume that anyone with N years experience will be at a certain level of ability.

"(potential) is best measured by talking with them about a project they really cared about"

I agree. I've always thought that once you've made sure that a candidate's programming knowledge isn't fabricated, the best way to evaluate them is to get talking about code. Whether that's proprietary code they've written but can't share, or it's on a GitHub profile for all to see shouldn't make a huge difference.

<comment><USER28> One of the best interviews I 'gave' was when the interview was for a position programming on a topic I knew almost nothing about. I had the person bring their laptop and walk me through their implementation of the topic at hand, then asked how they'd scale/improve it given more cpu/memory/etc and what kind of constraints the larger environment would impose upon the work.

It was super neat. Totally hired that person.

</comment></comment></comment><comment><USER29> Like the author, having run a hiring program for several years based on coding challenges designed to mimic the work the team does (ie, work sample tests), I do not understand why any tech company does developer hiring in any other way.

I particularly like how this team iterated on their uploader problem, abstracting it out to a socket server and providing a test suite for it. Reacting to ambiguity by _abstracting_ the problem was clever. And, of course, a great illustration of how a standardized hiring tool can, like any software project, be iterated on and improved --- unlike a free-form whiteboard interview.

<comment><USER30> _> (ie, work sample tests), I do not understand why any tech company does developer hiring in any other way._

Because _every_ method of hiring has its own biases and tradeoffs. Requiring work samples biases against programmers who aren't interested in doing a "homework assignment". For example, I don't enjoy whiteboard interviews but I'd rather write some fragments in front of the interviewer and "think out loud" instead of working on a multi-hour (or multi-day) coding project. Writing on a whiteboard is just not that big of a pressure cooker to me.

One could argue that the _majority_ would prefer self-paced work samples instead of whiteboard interviews and you're optimizing for a wider candidate pool. That's fine but it's still a tradeoff. I think you're running a smaller scale boutique firm so work samples makes sense for you.

For a company like Google Inc who is more selective[1] than Harvard University, I don't see a compelling need for them to rely on work samples. That company is so desirable for programmers that any "work samples" Google tries to standardize on would get leaked and endlessly discussed on stackoverflow (see FizzBuzz for precedence.) If Google has to keep changing out the work samples to stay ahead of the street knowledge, you've lost the ability to _compare_ work samples over the years which was one of the motivations to use that method in the first place.

At their scale and selectivity, the "study your algorithms book and prepare to be grilled on the whiteboard" seems to work best for them. Yes, they reject a lot of false-negatives but it's offset by ... their scale and selectivity. They can _afford_ to reject false-negatives. If their screening methods were truly terrible, I don't see how the label "ex-Googler" would have the currency it does in the marketplace.

[1][http://thenextweb.com/google/2010/09/14/one-half-of-one- perc...](http://thenextweb.com/google/2010/09/14/one-half-of-one-percent-of- google-applicants-get-hired-heres-how-it-works/#gref)

<comment><USER29> The "no homework" thing is such a red herring.

Companies that do work samples _minimize on-site coding interviews_ , because the whole point of offsite work sample testing is that most candidates can't provide an accurate picture of their aptitude in an on-site coding interview.

No interviewing software developer prefers an extra 6 hours of on-site interview to an off-site coding challenge.

The problem developers have with "homework problems" is really a problem with bullshit companies that won't provide timely feedback on their submission. Nobody wants to do homework that goes into a black hole, only to find out 6 months later that they were in consideration for the role. That's a problem with _all_ hiring companies, and it is orthogonal to the structure of the interview itself.

<comment><USER30> _> No interviewing software developer prefers an extra 6 hours of on-site interview to an off-site coding challenge._

Sure, Google/Microsoft have the legendary "6 hour" marathon interviews with multiple teams with lunch in the middle but for smaller scale of companies, they don't have bandwidth to mess around with all-day interviews. It's ~2 hours. In this thread, it shows I'm not the only one who prefers onsite whiteboard over homework projects so the "no developer prefers" claim is too absolutist.

 _> The problem developers have with "homework problems" is really a problem with bullshit companies that won't provide timely feedback on their submission._

Regardless if the feedback is received within 10 minutes or 10 days of submission, if I apply to 4 companies, _I don 't want to do 4 homework projects_. I simply don't. At least with 4 onsite interviews, I see glimpses of the office and meet the interviewers.

<comment><USER29> This is just not true. I talk to interviewing developers every day, and very few of them are interviewing at Google.

The final round of on-site interviews usually eats a whole day, and is preceded by a whole battery of time-wasting phone interviews, which themselves eat as much time as a coding challenge and often involve coding _with someone else over the phone or Skype_.

There are companies that are not Google that have _multiple rounds of on-site interviews_.

<comment><USER31> Are they not only getting very desperate candidates ?

<comment><USER29> No. Everyone interviews this way. The very most marketable candidates leverage personal brand and connections to dodge a lot of it, but you have to be good at sales to do that.

<comment><USER31> Perhaps in the US, but here in the UK, in my experience, companies have shorter onsite interviews. (except for the American ones...)

</comment></comment></comment></comment></comment></comment></comment></comment><comment><USER32> I am in the midst of my job search, fortunately I'm not currently working (due to an international move) so I have more time.

So far, I have completed around 12 1 hour phone coding tests, the same amount of 30 min chats with recruiters, I have 8x 5 hour on-sites scheduled, have spent about 4 long days on one take-home assignment, 6-8 hours on another, and have two more assignments to complete, both of which I estimate will take over 8 hours each.

I haven't had a weekend for the past 2 weeks because I've been working on assignments and I'm pretty stressed and exhausted.

This process has made me consider changing careers because it's so exhausting. When companies give you take-home assignments I don't think they realise how many things you have going on at the same time.

Additionally, I've found that a one hour coding test has allowed companies to move faster in the process with me than those who have given assignments. When I receive an offer from a company I am happy with then I will tell those companies with assignments I haven't finished that I won't be moving forward with them and they will miss out on a great candidate!

<comment><USER33> > This process has made me consider changing careers because it's so > exhausting.

Me too. I'm so put off by the entire process but it seems that to stay relevant, you have to get lucky and end up at a place that keeps your skills up-to-date with opportunities to learn and use the latest tech, or you jump ship every 2-3 years meaning you're once again dealing with this insane pressure of interviewing. Throw in ageism once you get past your mid-30s or so and the mere thought of staying in the industry starts to look quite depressing :-/

</comment></comment><comment><USER34> Sorry but I will not write any code for you.

Furthermore, after providing ZERO feedback in the past when I have provided github links, offline code samples, personalized cover sheets, etc. to prospective employers this also will not be done. I'm not going to worry and beat myself up because you can't be bothered to provide feedback even when requested to do so.

Homework for programming jobs is just like homework for school: it is not used to judge competency but to demonstrate that your time is not your own as it belongs to someone else. In the case of programming jobs the homework assignment also acts as a filter to weed out employees who will not be properly subservient to the employer.

</comment><comment><USER35> Last week, I finished 2-3 hours of on-site interviews which required a flight and an overnight stay after an initial phone screen, a phone technical interview, and a short technical challenge online.

I was surprised how a couple really basic concepts evaporated from my head during the in-person white boarding. The questions though were reasonable and the interviewers were friendly. All in all a positive experience just an incredible investment of time it seems for all concerned.

The take away is I need to practice solving problems on a white board with just 3 hours of sleep. The sleep deprivation part might be the most important factor to simulate.

Hm. Maybe bourbon? Or a good cider?

<comment><USER36> Linden tea, orange blossom tea or mix of both. Have a nice deep sleep.

<comment><USER35> Noted. That sounds good. Thanks!

</comment></comment></comment><comment><USER37> On paper (or blog), having a candidate write code for an interview is better than a BS whiteboard session.

In practice, it means I spent 20 hours on my last piece of sample code I submitted to a job I was excited about. Only to be told no thanks, with the feedback of:

"Your app meets what we spec'd out, but it differs from our style."

And to be clear, after I repeatedly asked for what they were looking for in this code, they replied "we are purposely providing nothing except functional requirements to see what you produce".

I could have given them exactly what they want, but I guess they'd rather just wait and hope that someone magically codes the same way they do?

Honestly I should have billed them after that.

</comment><comment><USER38> A startup in Berlin gave me a code challenge that would take at least 20 hours, I had even to deploy in specific version of the libraries, they give the datasets and etc.

I was very suspicious of being an actual feature development. I've said I would only do that if they would pay me to.

They answered that they feel very sorry for me because a lot of the candidates feel the coding challenge very challenging because they can learn new stuff and they love it

go figure

<comment><USER39> I had a pretty famous startup in Toronto decline a second interview because I flat out refused to do a take home test that probably would've been at least worth a day's exploration.

In comparison, a YC company (who eventually got acquired) actually responded very positively to my suggestion that I get paid to do the test they required, not dissimilar to bringing in a freelance contractor for a day.

</comment></comment><comment><USER40> > As brain-dead as I was after eight hours at my project, [..]

Is giving out an 8+ hour programming project as part of the application process really considered good hiring practice? If you apply to five jobs, you're expected to spend an entire unpaid work week writing code that doesn't benefit anyone? If a company receives 20 applicants, their ideal candidate selection wastes a collective work month?

I don't really understand this ideal. What's the point of maintaining an online portfolio and a github profile full of code if companies prefer to waste your time writing a pointless uploader?

<comment><USER41> I got laid off recently and was applying for all the jobs I could find. So I had to do a bunch of these programming projects. I don't mind them per-se, but it takes so long to do each one and you have to be ultra-clean with each one. I don't think companies take into account that you might not have that much free time. It's particularly stressful when you don't get through.

On pair programming tasks, I had a pair programming session with a guy, he gave me an empty project and a poorly specified brief, verbally, then gave me 15 minutes to crank something out. It was pathetic. I didn't get the job and the feedback was, "he didn't know some of the keyboard shortcuts I'd expect." In hindsight, I'm glad I didn't get that one but at the time things were getting desperate.

The best process was the one where I finally got a job. We talked on the phone for an hour and a half and by the end you could just tell we had that, "developer bond". His approach was the best I'd seen. He just talked through some of the problems he had, I proposed solutions and we talked through pros and cons. He treated me as an equal from the start. After that we had the face-to-face HR questions (how do you resolve conflict, etc.) which they have to ask because it's a big company, and another one with another manager but by then it was pretty much a done deal.

And I think that's the way to do a decent hiring process. Don't behave like an authoritative dick, and use phone interviews to screen candidates. Programming projects don't work because it's going to take you forever to review them, and pair-programming doesn't work because there's not enough time, and it's kind of horrible for the candidate.

<comment><USER42> I'm actually in the hiring process at a place currently, and their process is reasonable for someone in my position coming from freelance:

1\. Phone screen amounting to about 45 minutes.

2\. Skype interview with small code samples between two interviewers at 45 minutes each.

3\. 2-week paid trial doing real work.

I had #2 yesterday and am currently waiting to hear back from them about moving onward.

<comment><USER43> It's #3 that makes all the difference: a task that means something, and makes it worth one's while.

Duolingo wanted a functioning nontrivial (albeit sparse) app, expected to take 10 hours to write. I found it amusing on its own, but interviewed & got multiple offers elsewhere in less time.

<comment><USER44> That's actually harmful for getting quality candidates - "hot" prospects nope out of the process when they get an offer elsewhere.

A clever option is emailing the recruiter at Duolingo with something like "hey, I've got a competing offer, so the 10 hour coding challenge is really not something I have time for at the moment. Is there any other way to proceed through the hiring process?" Either way they answer, you're okay with.

<comment><USER45> I did that before during an interview with a pretty good company overall and the employer refused to accelerate the process or make an exception and just stopped the process.

</comment></comment></comment></comment></comment></comment><comment><USER46> In my experience, asking candidates to read and explain someone else's code is a much better indicator than letting them do some whiteboard programming.

<comment><USER47> Similarly I've found that talking through some deliberately broken code, and what improvements could be made to it, gives a good idea of what a candidate would be like to work with on real problems. It's more about the discussion than the code changes they come up with!

<comment><USER48> Agree 100% especially since that's most of the daily job is fixing up old code that was written in a hurry or for a different circumstance.

The ability to see what some code was trying to do and how to fix it and make it extendable and or understandable for future generations is invaluable.

</comment></comment></comment><comment><USER49> I think a coding test is the wrong approach. Typically a coding test is too simple to be meaningful, or too difficult to be practical.

Instead I would have a Mock PR in a github style discussion forum. Have it be a contrived example of some things that could be improved or some things that could spark really good discussion. Give the mock PR link to the candidate and ask them to Peer review the code. Make the discussion and comments of the the candidate the basis for which you determine their aptitude. If they have a lot of depth in their critique then you can be relatively confident how they would interact with the team in a real work scenario.

<comment><USER50> This is a great idea!

</comment></comment><comment><USER51> I can see the value from a hiring perspective in asking candidates to code, but it's no panacea. The bias in the interviewing process that so many acknowledge still bleeds through into many of the coding projects in the form of prescribed languages or frameworks. I suppose there's some merit to that if the intention is to screen for experience in a particular language or framework, but it does run counter to the commonly-expressed sentiment that companies are looking for smart developers who are able to learn new technologies rather than developers who have happened to use particular technologies.

As a candidate going through the job search process now, though, I have to say that it gets tiring to do a project or programming test for every different company. I can juggle more traditional applications (phone screens, on-site interviews) at a time than applications that require coding projects as I'm limited by the number of 4-hour (or 8-hour or whatever) blocks of time I can devote to completing a project for each one.

</comment><comment><USER52> I just instituted a coding assignment with our hiring (exceptions made for those with extensive github/similar open source contributions), and so far only have had three candidates do it, but I'm a fan.

Mine is crafted so a really competent developer could get it done in 30 minutes, and intermediate 2 hours and a junior probably not finish. I tell them to send what they have in after two hours that I'm as interested in seeing the structure and thought process than the final result. It seems much better than having someone do it at the office or one a whiteboard for accessing basic competence.

Yes it sucks, but after you get bit once or twice by hiring people who knew all the right words but can't actually produce something useful you get a lot more open to it. I'll gladly take the time to take you out to lunch to discuss the job and code assignment and give feedback afterwards, investing a couple of hours into finding your next job shouldn't be this huge ordeal and gives off a diva vibe I don't want in my organization anyway.

<comment><USER53> _> I tell them to send what they have in after two hours that I'm as interested in seeing the structure and thought process than the final result._

While this seems like a good idea on paper, you _know_ that people are not going to stop after 2 hours unless you force them to (i.e. by doing the assignment on-site).

 _> get it done in 30 minutes, and intermediate 2 hours and a junior probably not finish._

What kind of exercise could a junior programmer _not finish_? Give up or produce miserable code sure, but not being able to finish? Do you require some really strange domain-specific knowledge?

PS: The website in your profile doesn't seem to be working.

<comment><USER52> well with 3 attempts one went over and kept working until he got a solution (took a while), another stopped at around 2 hours to submit and asked for feedback, 3rd finished in time and is a badass.

You've be surprised. Maybe I expect too much, but the assignment is essentially network aware k/v store with a single get command and set command. Extremely poor mans redis and a step up from an echo server but for some people doing anything over a network that isn't HTTP breaks them.

Thanks for website, my consulting/blog site that pretty much fell into disrepair after taking architect position at a startup.

</comment></comment></comment><comment><USER54> A lot of people here seem to suggest it is unreasonable to ask someone to do some coding in an interview given that you've shown a series of projects and provided references. Unfortunately I've been through a variety of interviews where I feel the conditions which I was asked to code were unreasonable (whiteboard, overly complex problems while under pressure, using a language I was not comfortable with or just academic problems that are rare in the real world), but I really think confirming their ability to write code, beyond things like github repos etc, is an important step of hiring a developer (because i've seen great github repos and references, but ended up with developers that are very, very average)

I've only hired tens of people, and while that is not a massive sample size, I'm still astonished by the simple questions we ask that people get stuck on. The one that continues to shock me:

"In any language of your choice, write a function that takes an array of ints and returns the sum of those ints (If you use a language such as php do not use array_sum)"

I'm just looking for something like:

function sum($ints) {

        $sum = 0;

        foreach($ints as $i) {

            $sum += $i;

        }

        return $sum;

}

Then we follow up with something like: "lets say your input array contains [4,-7,0], walk me through the code"

And we finally ask "talk to me about some things that could go wrong at runtime that might cause issues with this function". Expecting people to talk about overflows (maxint?) or if its a dynamic language like php, checking for non-integer values.

I estimate that around 50% of people get ruled out during the "coding" exercise mentioned above. It is astonishing that these people even apply for developer jobs.

<comment><USER55> Imagine there are twenty candidates who can't code at all, so they apply everywhere. There are also twenty candidates who can code, but are presently unemployed because they all worked for SpoonRacket.

Now the entire lot apply for jobs at your company, and you hire two of those who can code. Within 3 months the rest of the good ones have found jobs, but 30 others have temporarily been unemployed and none of the bad ones have been unemployed.

So you interview again and roughly 40% of the programmers you interview can't code. But that doesn't mean that even 5% of the programmers out there can't code, it just means it is the same bad programmers that interview in lots of places.

It is the same when employers say they are getting ten or twenty job applications for every open position - that doesn't mean anything because everybody who is unemployed applies to more than one place, but from only one side of the fence it sure looks like that.

<comment><USER56> I'm not sure I understand,

> but 30 others have temporarily been unemployed and none of the bad ones have > been unemployed

Is this 30 new programmers plus 20 of the original candidates who can't code? Should that be "none of the bad ones have been employed"?

<comment><USER55> Yes, sorry.

</comment></comment></comment></comment><comment><USER57> If a company asks me to do 8 or more hours of artificial coding test I usually refuse, unless I really want to work for this company (because it is well known in the community, which are few).

However, I'm happy to work on any real task for a reduced or limited fee. It's a win-win: the company gets to know the candidate under real conditions, has an actual task solved at a reduced price, and the developer does not give away his/her time for free. Plus, if it works out, the candidate has learned already the first steps about the company's system and process and can directly build on it.

I'm always wondering why not more companies do this.

</comment><comment><USER58> The process I've seen working best to this day was at Pivotal Labs: pair for a half hour to one hour with 3 different people in the team you're being hired for. You get to measure everything, from communication skills to cultural fit. Perhaps pair on an internal tool, or even better, an open source project (in which case the candidate gets an extra incentive for not wasting their time).

If you're dealing with too many candidates, issuing a simple 1-hour coding challenge will weed out the uninterested (and won't favor people that have 4-8 hours to devote to a huge coding challenge).

<comment><USER38> I've interviewed there. Only two people I've paired with, one in the morning and other on after lunch. I had two problems:

* The 2nd guy I've interviewed with was completely non interested on doing it. He was checking the phone all the time and getting off and on. I felt so awkward because of his lack of interest.

* I didn't get _ANY_ feedback at all, just got refused. After 6hours in their office...

<comment><USER58> That's an orthogonal issue, though. I had the same experience during a whiteboarding somewhere else...

</comment></comment></comment><comment><USER59> I don't mind a 2 hour or even maybe a 4 hour assignment, but a project estimated at 8 hours?

Yeah, either you pay me for that time, or I'll just look elsewhere.

(And granted, that might exactly be what the company is looking for: a way to reduce the amount of applicants they have to sift through)

<comment><USER60> The problem is, there are no realistic 2-hour, or really even 4-hour, coding projects that faithfully map to on-the-job skill. 2 hours is too long for a data structures riddle, but way, way too short for anything of even a tiny degree of complexity. Let alone writing some unit tests, mocking up a design and then scrapping it for something that makes more sense.

Timed coding tests are just a waste of everyone's time, whether it's a weekend take-home thing, or a 30 minute whiteboard thing, or anything in between. They do not account for individual developers' needs, like privacy, their own editor or IDE, and they exist in this ether between bullshit trivia and an outright working internship that ought to be paid.

The other huge thing is that in a lot of firms (I doubt Digital Ocean is like this, but many firms are) "programming" is a low-status, junior activity. In these companies, if someone asks you to write code in an interview, _especially_ if you are an experienced candidate with a GitHub profile, references, and project experience, then even merely _asking_ you to code is more like saying "dance, monkey, dance" and setting the whole atmosphere of your negotiations and interviews in such a way where you are the lucky low- status person who should be so grateful as to receive a job.

Unless you get a really great cultural feeling from speaking with technical team members first, the best general advice is to simply reject employers who ask you to write code. If they ask you to write code before even speaking with you, absolutely reject. Full stop.

I also think all candidates should fully reject internet-based timed "exams" as well. Things like HackerRank are designed to commoditize software labor, which is a creative design activity and if it even can be commoditized at all, it surely can't be boiled down to data structure trivia that you have to code around in a browser-based IDE with a literal clock ticking down.

Anything like HackerRank immediately screams "quantity not quality" and is straight rejection-worthy, no questions asked.

</comment></comment><comment><USER61> When we hire, after a few phone interviews we give the programmer an in person coding exercise that is very basic. Open a file and parse data, etc... They can use the internet. It's very basic. 80% of the candidates cannot do that. I'm not exaggerating. Maybe I'm missing something here, but it's pretty astonishing to me.

<comment><USER62> Do you require them to do it in a specific language or let them choose which one they are most comfortable with? How do you decide which programmers to bring onsite? Have you considered giving them different options in the from of doing a take home test or coming in and coding inperson instead?

<comment><USER61> We require a specific language... the language we use for the job. The phone interview asks technical questions, a few design questions, and some personality geared questions. The technical questions are basic and cover some of the basic concepts of the language and basics of OO.

We haven't considered doing a take home test for 2 reasons. 1) We want to see how well the person can get around windows/keyboard. I believe this is important. 2) What we are asking to code is very basic. If we sent something home it would have to be harder.

Are you thinking that the in-person coding would make them nervous? I ask because we are clearly doing something wrong if it's hard to find candidates that can do basic programming. Even if the person is entry level, that would be fine as long as they are an A player at the entry level... if that makes any sense.

<comment><USER62> Valid points as far as the language goes, but I'm not sure I agree with your points about workflow... the primary goal should be to see if they can reach the solution using the tools required for the job over _how_ the tools are used in my opinion, and how much guidance they require to do so. I am thinking that the in-person coding might make them nervous. As a personal example for my specific line of work coding is more of a secondary requirement over having knowledge of how systems work, and despite having contributed code to some widely used open source projects and having enough knowledge to get by given language given enough documentation, I would likely freeze up in an interview if asked to code.

If I were you I would look for candidates that actually have some projects they can showcase that proves they know how to code, and offer them the chance to either do an easier in-person test or a take-home project. During the job search I personally jumped at the chance to do take-home projects since I view them as learning experiences, but for more experienced people I can see them being viewed as tedious. Most truly passionate entry level programmers looking for their first job and some direction I hope would jump at that chance to get some more directed experience as well.

</comment></comment></comment></comment><comment><USER63> One thing which I've wanted to see a company try is to give someone code to read and asking them to walk through how they would go about understanding it. I've done this once when I worked at MassChallenge and was trying to hire a more senior engineer to work above me and I felt like it gave me a good idea of his ability to talk through problems and work with others, but that was only one interview and I'd like to see a larger sample size.

</comment><comment><USER64> Calcado describes his wish that the candidates be "more T-shaped", and not just worker bees with narrow skills. But I see no mention of ways his more successful hires "branched out" and expressed wider interests or capabilities, presumably in in domains outside computing.

Here, do the T branches apply only within computing, like knowing more than one software tool? If so, I think that's not what Tim Brown meant.

Presumably an interview seeking signs of "T-branching" would propose higher level questions, like business acumen or domain curiosity or spontaneous Q&A to better understand or elaborate the requirements needed to solve a representative problem. Otherwise the "T" just equates to broad software skills.

</comment><comment><USER65> not sure why there is so much resistance here about doing an 8 hour test, I might be in the minority but for a traditional whiteboard interview I would spend a lot more than 8 hours revising algorithms and so on, and have a lot of stress due to whiteboard coding not being anything like real coding.

Being able to take a Sunday to work on a challenging problem in my own editor / computer / dev environment sounds great! I wish that all companies added in their job ads something like 'if you want to skip the whiteboard send us your solution to this problem together with your resume' where the problem is a representative problem of the type of work they do, then you spend the interview discussing your solution as opposed to playing algorithm-roulette.

It would also cut down the uncertainty when you see a job ad where they ask for all sorts of different languages / skills, if you have fun doing the coding test, it's likely you'll enjoy working there, and vice-versa (in which case you might decide not to even apply)

<comment><USER66> > not sure why there is so much resistance here about doing an 8 hour test

If you're applying to just 5 companies, that's a full (well 40-hour) work week of effort for a small chance of going forward with the next stage of the interview/hazing ritual. Might be good if you are already unemployed, but incompatible with candidates who already work.

<comment><USER67> Which means that, if you're hiring, _and you want only candidates who are currently unemployed_ , it's a reasonable practice. But the best candidates probably already have jobs, and you're not going to get them. So maybe that's not really the way you want to do things...

<comment><USER65> not necessarily, if I am not unemployed I am not likely to apply to tons of jobs, but be a bit more selective, and I don't see finding more than 1 job a week that I would be interested in... of course I am not a typical developer (C, python, back-end, security, networking, ...) so there are a lot less jobs in my niche than say if I was a front-end person

This means that taking every sunday to work on cool / interesting self- contained problems would be great, and much better than spending time revising red-black trees and avl tree rotations and sorts and so on for the n-th time to prepare for the whiteboarding

</comment></comment></comment></comment><comment><USER68> I like the 'simplified job description'. All too often a job description is simply a list of technologies the last guy claimed to be using/have used/knew something about.

</comment><comment><USER69> I've only had to do 4 of these, as they aren't that popular in my corner of the earth yet, but I didn't like them, I have a kid and stuff to do and I can't spend my time doing 4 hours extra on the weekend unless it is a job I really, really want.

First out of the 4 I did what was asked but got refused because I didn't clean up the code afterwards ( I thought I'd been cool for demonstrating some lesser known functionalities of the language and API I was dealing with). If I've spent 4 hours doing it, I don't want to spend 2 hours making it really pretty.

Second out of the 4 I misunderstood the instructions so I went in and 'fixed' their code to make it work they way I thought they wanted it ( was maybe too tired since it was late at night after a day of working on a personal project) This was actually a job that was close to one I really, really want.

Third out of the 4 I can't remember anything about it now.

Fourth out of the 4 they really liked what I did and we had a great interview, and then I got turned down because they thought I wasn't really that interested in their company (which was true, I was just fulfilling a job search requirement while getting my startup going)

Aside from that I interviewed at a social analytics startup one time and we ended up talking data mining and they asked me to do a 2 weeks project with them to do a POC of sentiment mining Twitter/Facebook. I gave them what my consulting fees would be, but they seemed really interested in me doing it for free as conditional for pursuing employment there. So I never got to try that coding exercise.

</comment><comment><USER70> I wonder if maybe instead of asking people to code on the fly, it would make more sense to ask people to review, analyze, and explain some existing code.

</comment><comment><USER71> A few years ago I was given a programming test after an initial phone screen with a HR person. They said do it in any language, and it had a fancy framework that let you edit and test the solution before you sent it.

I chose Clojure since I was learning it at the time. The problems were easy and I made working solutions, but the framework wasn't working. I gathered some call stack data and sent it to the people that make the tool. They thanked me for the feedback and later fixed the bug.

I sent the completed code by email to the company with a note about what had happened. Showing clearly that my code works in the form of some tests.

Then, on the basis of this they didn't proceed.

</comment><comment><USER72> > we would ask you to use just your language's standard library, no third- > party libs or frameworks... With the problem description, we sent to > candidates a functional test suite, a binary that when started would try to > connect to the candidate's server implementation, open lots of sockets, > sends lots of messages, and verify the results against what the problem > description stated. The candidate was instructed only to send their > submission once it passed the functional test on their local box.

Incidentally, this is exactly how the "Sun Certified Java Developer" certification was structured. You were only allowed to use java.* and javax.* packages provided as part of the JRE, and even among those things like the SQL packages were off limits. Instead of sending a test suite, they provided an interface you needed to implement and then when you submitted it, they had some sort of test suite that tested your code for conformance to the spec (and if it failed, you failed). If it passed, someone would manually review your program to make sure it made sense with respect to the requirements (which were intentionally somewhat vague, like real requirements) and if you passed that step, you had to write an essay about WHY you made certain design decisions (like sockets+serialization vs. RMI, etc).

This was by far the most challenging and fun task of its type that I've done, and to this day if I see that someone has that certification I know that they have what it takes to get things done. This methodology should really be utilized more, although for many companies the overhead of designing such a challenge would be way over the top. Makes me think that things like [https://www.stockfighter.io/](https://www.stockfighter.io/) have a future in our industry.

</comment><comment><USER73> Well, you can't give people stuff to do offline because they cheat (you'll see the whole thing show up in no time as a Stack Overflow question). And you can't avoid coding questions during interviews because it is astounding how much people lie on résumés; seriously, even people who describe their background as "Advanced Knowledge of $LANGUAGE", can choke when asked to do the most trivial things in $LANGUAGE.

I try to ask questions that aren't completely ridiculous, but definite tests of knowledge. And in my mind I judge correctness not so much on remembering obscure syntax but on how they approached the problem, and how much time it seemed to take them to start doing something. I _always_ ask about things like how they approach debugging, testing, etc. because these are important too and they tell you a lot about how likely the person is to be able to handle any problem you come up with.

</comment><comment><USER74> Good for initial screening, but should not take more than 30 minutes. Else IMHO laziness or incompetence asking a programmer to do an assignment. Any good manager/developer/interviewer can sit with a candidate for 1-4 hours and determine if his skill level and personality match the job requirements.

</comment><comment><USER75> "we sent to candidates a functional test suíte, a binary that when started would try to connect to the candidate's server implementation, open lots of sockets, sends lots of messages, and verify the results against what the problem description stated"

Codility.com is a software as service company that automates this.

</comment><comment><USER76> we use this [http://play.elevatorsaga.com/](http://play.elevatorsaga.com/) because it's extremely easy to understand, is open to multiple solution and immediately sort out both who doesn't understand queues, priorities, closures but most importantly who has completely no empathy and win level pushing numbers instead of satisfying clients.

it also takes very little time to introduce and solve which is a plus since we prefer to give it out in-office. but we do tell the candidate beforehand that there will be a 15 coding challenge so they don't panic.

</comment><comment><USER77> You don't have to determine whether I am good at writing software. I put that right on my resume.

You just have to determine whether I am a _liar_ \--or, more generously, simply unable to assess my own abilities accurately.

</comment><comment><USER78> "It turns out that most people would put together a Rails app with more lines on their Gemfile than lines of actual code they wrote. Worse"

\- how is this worse then rewriting the wheel? write code only when you must

<comment><USER79> I would argue it contributes to huge bloat. People pull in huge powerful libraries to format a single date. In addition, a vulnerability in a library you pull in could compromise your application. Even deeper, a sub-sub-library (i.e. one you didn't even add yourself) could have a vuln that affects you.

</comment></comment><comment><USER80> This article sounds pretty good, we need more people like you, but I think the next step for DigitalOcean is not hire for resume, or at least giving a chance to make one of those coding tasks

</comment><comment><USER81> I like Jeff Atwoods approach[0], hire the candidate for a short project (2-3 weeks) let the candidate a feel for the product and team while getting paid. And at the same time let the team get to know the candidate. I would love to go through a hiring process like that :)

[0][http://blog.codinghorror.com/we-hire-the-best-just-like- ever...](http://blog.codinghorror.com/we-hire-the-best-just-like-everyone- else/)

<comment><USER66> Unfortunately, that approach filters out already-employed people whose employers do not permit them to work simultaneously for someone else and/or impose IP ownership rules (We own everything you write inside and outside of work).

</comment></comment><comment><USER82> I don't usually ask candidates to write code for free. They're professionals and I expect them to _review_ code with me -- some of theirs and some of ours -- but not write it. I find that asking a candidate to write code for no true economic purpose that I will lightly read and then throw away does not inspire somebodies best work and I can gain more insight by an exercise where i have them read an explain code to me.

</comment><comment><USER83> That article is the best advertising for not working for DigitalOcean. The management doesn't understand what makes a good programmer.

</comment><comment><USER84> "Language standard library" seems like a very arbitrary standard to impose. Some languages have a very large one, others a small one. Wouldn't it be better to specify what "primitives" you wanted the code to use? (e.g. must use raw sockets vs. may use a http library vs. may use a basic json-parsing library vs. may use a full json object mapper)

</comment><comment><USER60> As I mentioned in another comment on this thread, one of the biggest issues is framing programming as a low-status, commodity activity.

I doubt Digital Ocean does this since they want good people, but a lot of firms absolutely design their hiring process to create an upper hand and lower your status before you even walk in the door -- especially if you're an experienced engineer.

Short timed tests or whiteboard hazing are the worst version of this. These are the most blatant attempts to commoditize software labor down into a fixed set of "primitives" like data structure trivia or riddles. This is what organizations like HackerRank exist to do: allow big companies to suppress wages by creating these sorts of commodization filters. They don't attempt to capture the value of creative problem solving at all -- because the company is not pricing the value of creative problem solving into the budget for the position, since it's a rank-and-file commodity job.

For this reason alone, you are better off flat out rejecting anything like a HackerRank test or similar online, interactive, short timed test. I would absolutely go so far as to even refuse to write code on a whiteboard if asked to do so in an in-person interview. It's fine to _talk_ about how to solve a problem and spec things out. But the minute it becomes focused on actual fucking syntax, it's game over. You are now a cog.

Longer-form tests are a lot harder to evaluate from the candidate's point of view. Now you are asking for a significant chunk of my time, without paying me. And I am still at the mercy of whatever you happen to think constitutes a valid test. _I_ might look at your take home test and thing, "wtf? this has nothing to do with real world work." Where does that leave me? Of course I want to impress the hiring staff, but I also don't want to get roped into a bad job, and a team that hires me to do real world work by using contrived coding examples is probably not a good place to be.

Personally, I think it's a lot more useful to talk about code that already exists. Either example code form the candidate, or example code that you give to them along with some time to study it.

Engineers read code more than they write code anyway, and right when you hire someone, even if you're a bleeding edge start-up, their short-term impact is a lot more predicated on their ability to read code and quickly learn a new system, not so much writing a bunch of stuff from scratch.

It's also pretty hard to fake competency when reading and discussing actual code. If you don't know how something works, you just don't know. You probably can't just google how the internals of some highly-specific ad hoc code works, unlike data structure trivia (which is yet another reason why it just doesn't matter how much data structure trivia you have memorized).

Basically, my overall conclusion is that when someone asks you to code in a short, timed setting, they are basically lighting a cigar, fingering their handlebar moustache, kicking their feet up on the table, and saying "dance monkey dance."

Unless you are literally desperate for the job, you should reject it right away. You are being positioned so that no matter how well you do on the test, you are but a lowly code typist and they will not take your attempts at negotiation seriously.

</comment><comment><USER85> My main hope when talking with a recruiter person is that we avoid the kabuki theater. at this point: I'm good at some stuff, poor at others. My resume is an adequate description of things I am good at. I'd like to talk personality fit now, thanks.

</comment><comment><USER86> Flavio is a nice Guy! He wrote a great scala Database library, lately. (quill)

</comment><comment><USER87> Not writing code for a potential employer or being asked to in an interview is a warning flag to me personally. I feel better if they do, as they've seen the goods to influence their decision on the hire. It's more about "How does this person think, how do they approach the problem". How can you measure that unless you're there? I see a lot of "I shouldn't have to do that". Well yeah, ok. But if you're any good, you shouldn't have a problem with that either.

<comment><USER88> _Well yeah, ok. But if you 're any good, you shouldn't have a problem with that either._

Whiteboarding, sure. Homework assignment that takes 8 hours, as the author experienced? No way. I'm good, my hourly rate reflects that, and I don't work for free.

<comment><USER89> See I'm the inverse. I'd take the 8 hour project over the whiteboard.

Thinking in a quiet, relaxed environment beats trying to reason through a problem while also thinking about how you are "performing."

</comment></comment></comment><comment><USER90> I understand the appeal of having candidates write code as part of the application process, but in my view it's useless and worse, it will drive off some of the best potential candidates you could get.

First, why is it useless? Because it is an artificial and contrived exercise and not real work. Even if you design your requirements to mimic your real work environment, there's something key missing; the candidate has no knowledge of your team and your environment. How I would write code for my current team vs. how I would write code at any other job I've been at is different. You're basically asking the candidate to take a guess at what kind of paradigms your code reviewers like and what they don't like and hope it works out. One team might find use of closures smart and great, and another might find it unintuitive and adding unnecessary complexity. Your applicants have no way of knowing and just have to hope they write code your team likes.

On top of that, one contrived exercise is not a good way to get an idea of a persons' actual skill and knowledge. I'd much rather have a 30-90 minute conversation with someone where nobody writes any code than to review a contrived example. I get the appeal of the programming task, it can be done asynchronously and thus doesn't require any developer time until you go to do the review. But is it really saving you much time? Anyone who's code doesn't pass unit tests or any other automated level of screening wouldn't probably make it very far into a conversation either.

If I have an applicant spend 1-4 hours writing a code example I've had them use at least as much and probably more of their time for far less benefit than if I had simply called them up on Skype and talked to them about programming. The conversation will reveal not only a lot more to me about their actual skill set, but also their personality and how they might fit into my team. Now I realize teams doing exercises aren't skipping the conversation step, but I'm advocating for skipping the programming task step and going right to the conversation if someone's resume is solid and they can intelligently answer some basic questions (a handful of questions that should take no more than a couple minutes to answer, not an essay.)

Second, why would I say it drives off good candidates? Because I know a lot of really good developers (including myself) who skip any job posting that requires writing code as part of the application process. Why do I do that? Because it tells me the team behind this application is immature. They haven't yet gotten to the point in their careers where they realize these exercises are a waste of everyone's time. That means there's probably going to be a lot of other young dude bullshit in the team that I'm not interested in dealing with.

</comment><comment><USER3> I think the reason so many companies resort to asking candidates to do a programming project for the interview is because they are lazy and don't want to take time to interview someone, or they suck at interviewing, or both. Bottom line, if you can't judge someone's ability by talking to them it's YOUR fault. You suck at interviewing. Last time I was looking for a job a company I was interested in gave me this massive assignment and I just told them to take a hike. I'm not spending all day writing some code for you for free sorry. Sure there are lots of horrible programmers out there, and you'll end up hiring them if you suck at determining what their skill level is via a conversation. But I hope other GREAT developers follow suit with me. Refuse these stupid programming challenges.

<comment><USER91> Have you considered that part of the purpose of the challenge might be to filter out particularly arrogant applicants who might be problematic in a team setting?

<comment><USER92> Could be, but that'd be an assumption at best, and "negging" at worst.

Also, one could just as easily see a "massive" assignment as arrogance from the employer.

</comment></comment></comment><comment><USER93> I recently spent 4 weeks of my life interviewing for one of the bigger video games companies for their web dev team.

After those 4 weeks, I didn't get the job. Around the 3rd week I felt like my time was wasted. I want to work for this place, but if I want try again later in life, I probably won't because it took up way too much of my time with little to no value.

It sucks that I'm required to sacrifice my personal and vacation days to find potential new employment.

</comment><comment><USER94> Watch the recent coreos post/video, the CTO is probably a smart and nice guy but he looks about 16.

<comment><USER95> HN is not a place for discussing personal details about somebody, like how they look. Moreover, that weird "but" makes this comment read like an attack. Please don't post anything like this again.

We detached this subthread from [https://news.ycombinator.com/item?id=11290180](https://news.ycombinator.com/item?id=11290180) and marked it off-topic.

<comment><USER94> Sorry, wasn't meant to be an attack, just noticed that almost every startup about page and video makes it clear that grey hair is not welcome, even at exec team level. Can't delete now.

<comment><USER95> Ok, but singling out a single company, let alone a specific person, isn't the direction to go. Think of HN threads like a Markov chain: each thing you add biases what others add next. A comment like the one you posted leads to an ugly place.

<comment><USER96> > Think of HN threads like a Markov chain

Fascinating! It's like Blindsight in reverse.

</comment></comment></comment></comment></comment><comment><USER97> The best candidates are too busy kicking ass and getting lured away with huge sums of money - and not waste time working for free.</comment>


Facebook custom targeted audiences - <AUTHOR> https://m.facebook.com/business/a/custom-audiences <comment><AUTHOR> Wow, this is shocking. I'm surprised this hasn't been in the news before. The fact that you can do this and that Facebook is so open about is surprising.

Has anyone used this before?</comment>


I want to learn C/C++, but really shouldn't - <AUTHOR>

As a web developer I've always been a scripting guy, and I enjoy the advantage.
However, recently I began to find C/C++ code to be beautiful with all their dangerous manual memory allocation calls, garbage collection and cryptic compilation logs. It just fascinates me, especially since I tinkered with Fossil.
But as it seems it can really make productivity suffer as higher level languages are more tailored toward the web. Please give me some advice to mute the hardcore hacker inside me. <comment><USER2> If you find yourself asking others to make your decisions for you, the correct answer is inaction, since you'd only do it halfheartedly anyway.

<comment><AUTHOR> The problem in fact is too much enthusiasm. I wanted to hear some guidance on how to find balance in business vs technology.

Anyway, I appreciate your comment.

<comment><USER3> Use the right tool for the job. C is the right tool in a limited set of circumstances, but it's the right tool where other language choices simply don't exist. You literally cannot write what the development community would recognize as a working OS kernel, for example, in Javascript. That means that, for someone writing an OS kernel, C is infinitely* more productive than Javascript. Conversely, while you could implement a web browser that used C as a client-side scripting language, it would be a terrible idea for any number of reasons. As such you're far more productive using Javascript for the job it was designed for than if you used C.

You're trying to compare apples to oranges when you say that using C can make productivity suffer when compared to higher level languages. "Balance in business vs. technology" has nothing to do with this particular discussion when it's framed properly. Decide what you want to write and then, if a low- level language is necessary, the ? evaporates. If you can make do with a high- level language, the ? likewise evaporates because you'd be an idiot to use a tool that forces you to deal with unnecessary complexity like C does.

* for finite but extremely large values of infinite</comment></comment></comment>


Matching cryptographic primitive strengths - <AUTHOR> https://www.imperialviolet.org/2014/05/25/strengthmatching.html

<comment><USER2> The argument that 128 bits of security are enough is quite reasonable [3]. There are, however, a couple technicalities with using 128-bit security primitives and/or keys:

\- Multi-target attacks. In the case of symmetric ciphers, one has to take into account that the attacker may have many messages encrypted under many keys, and that success is achieved once at least one of them is broken. [1] exemplifies this effect with AES-128: with 2^86 time, a machine of size 2^32 has near certainty of breaking at least one out of 2^10 keys. Larger key sizes effectively render such tradeoffs moot. Note that multi-target attacks do not affect the speed of elliptic curve discrete logarithm computation: while it is possible to speed up a discrete logarithm after a previous one (by a factor of ~sqrt(2)), you still need to go through a full 2^128 computation first. So pairing (say) a 256-bit elliptic curve with AES-256 is not entirely unreasonable.

\- Quantum computers. The last paragraph mentions pairing AES-256, SHA-384, and 128-bit McEliece as a balanced choice. Quantum computers clearly affected this choice, with the AES key size chosen to thwart Grover search, SHA-384 being chosen to thwart the Brassard-Hoyer-Tapp collision-finding algorithm (although it is no better than classical search due to crazy memory requirements), and McEliece for its well-known post-quantumness. A little- known fact is that Grover also speeds McEliece breaking considerably [2]. Thus, in a quantum world the McEliece key size would have to be increased to get quantum balance (McBits's parametrization achieves 128 bits of classical security).

\- Security margin and operational costs. This is covered well already by Adam, but is worth emphasizing. With AES, the cost of going 256-bit instead of 128 is 4 more rounds. In the case of SHA-256, upgrading to SHA-512 actually speeds things up in typical 64-bit CPUs. Except in the most extreme of cases, this is a very acceptable cost to get a comfortable 128 extra bits of security in case AES or SHA-256 gets wounded. In the case of RSA or Diffie-Hellman, going from 128 to 256 bits of security gives us an approximate 25x slowdown, not to mention the increase in key sizes: 128 bits of security margin in this case is hardly acceptable. The case of elliptic curves is more murky: the cost of an extra 128 bits of security margin is around 4x, which may or may not be acceptable depending on the application.

[1] [http://cr.yp.to/papers.html#bruteforce](http://cr.yp.to/papers.html#bruteforce)

[2] [http://cr.yp.to/papers.html#grovercode](http://cr.yp.to/papers.html#grovercode)

[3] I made a similar argument not too long ago, on the specific case of SHA-256's collision-resistance: [https://news.ycombinator.com/item?id=7747545](https://news.ycombinator.com/item?id=7747545)

<comment><USER3> Thanks for that. I've updated the post to mention multi-target attacks and the quantum attack against McEliece (which I wasn't aware of, and is saddening because it appears to make McEliece a much less attractive PQ system).

I picked SHA-384 as an example of a case where one might want to spend resources on an "unbalanced" primitive because of the history of hash functions failing to meet design strengths, rather than as a quantum resistance.

Although, if you are looking to pick between SHA-256 and SHA-512/384 for speed on modern CPUs, SHA-256 (in an interleaved tree mode) wins for speed: [http://eprint.iacr.org/2012/476.pdf](http://eprint.iacr.org/2012/476.pdf).

</comment></comment><comment><USER4> The thermodynamic limit cited (Landauer's cost of erasure) to show that 128 bits is enough (barring algorithmic advances) doesn't strictly apply. Charles Bennett[1] pointed out that any irreversible computation (i.e. one that involves erasure) can be made reversible by saving all the intermediate results (rather than erasing them), printing the result, then running the computation backward and letting it eat up all of the saved intermediate results. While this approach to computation isn't directly applicable here, it does show that you can't count on the thermodymanic cost to keep you safe.

[1] C. H. Bennett, "Logical Reversibility of Computation" _IBM Journal of Research and Development_ 17:525-532 (November, 1973), [http://www.cs.princeton.edu/courses/archive/fall04/cos576/pa...](http://www.cs.princeton.edu/courses/archive/fall04/cos576/papers/bennett73.html)

<comment><USER2> There is another more fundamental limit, credited to Margolus and Levitin [1], which puts the maximum speed of a (quantum) gate at 2E / h operations per second for a system of energy E and Planck constant h. Given the energy of the sun quoted in the article, this puts the upper limit of bit flips per second at 2^193. Seth Lloyd's 'Ultimate physical limits to computation' [2] goes into further detail on this subject.

However, the notion of zero energy dissipation (scalable quantum computers will likely need error correction, which implies bit erasure) on any real system is ludicrous, and I think the Landauer bound models reality better than Margolus-Levitin.

[1] [http://arxiv.org/abs/quant-ph/9710043](http://arxiv.org/abs/quant- ph/9710043)

[2] [http://arxiv.org/abs/quant-ph/9908043](http://arxiv.org/abs/quant- ph/9908043)

<comment><USER5> Either way the limits are so far out from the computational capability of humans in the forseeable future (next few hundred years) that they're both massive overkill. When you really NEED that much security you have to worry about the loyalty of your nation's military and how good all those armed guards will be at keeping you from being waterboarded.

</comment></comment></comment><comment><USER6> Another point on the "2 bits per increment", you can only flip one bit at a time to get through all possible 128-bit states using Gray code.

<comment><USER7> Or unary coding...bringing register width into new dimensions.</comment></comment>


Tell HN: Please, please put your product's price on your website - <AUTHOR> All too often, my experiences with SaaS's these days follows the same process:<p>- Find out about new product<p>- Visit website, spend too much time deciphering what the product actually does<p>- Ctrl + F for "price/pricing" - not found.<p>- The all-to-ubiquitous pre-sales chat window pops up at the bottom-right of the website.<p>- Me: "How much is your product?"<p>- Them a few hours later: "It is $XX/per user/per month"<p>- Me: "Unfortunately this is out of my budget, I wish I could have pre-filtered myself as a potential client to save us both time."<p>SaaS's, please, please put your product's price actually on your website! <comment><USER2> I understand companies in highly competitive markets not wanting to disclose their prices to lose an edge to the competition (though as a consumer I want more price competition to get better deals).

What pisses me off the most is when the price could only be discovered after having some sort of conversation/correspondence with a salesperson. I just want to find out a number. I don't want to talk to a person or schedule a time for them to "understand my needs and what makes my situation unique" before giving me a price. Maybe I am an exception, but I don't want the personal interaction when trying to make a business decision based on my own weighing of factors, including price. I would rather submit a recaptcha to see a price than go through all these hoops and spend extra time on it.

</comment><comment><USER3> I would just assume if you care about price it's not match. These kind of SaaS target price insensitive buyers that work for megacorps.</comment>


Heroku: Why Instant Deployment Matters (YC W08) - <AUTHOR> http://blog.heroku.com/archives/2009/2/23/why_instant_deployment_matters/

<comment><USER2> _Once every hundred years media changes!_

I wish people would stop conjuring up mathematical models to add drama to their observations.

Firstly, mathematics makes for poor theater. Secondly, the silly models distract from otherwise sensible observations: media changes and development gets faster.

But, in the end, the only sensible sweeping generalization may be _past performance does not guarantee future results_.

</comment><comment><USER3> I get your point: the deployment-to-development ratio has increased as we've streamlined development processes faster than deployment processes. I understand the underlying theme, Heroku is going to kickass in deployment. More power to you.

Here's what bothered me about this post:

\- Scrum fanboyism. It was unexpected and threw me off track while I was reading. I felt it negatively detracted from your point.

\- Capistrano bashing. You certianly gave Cap it's due, but calling it an "incremental improvement" compared to what you've got in store takes balls. We'll see how that pays off.

I know this is just a preface to some (hopefully) technical articles about your work, but on it's own, it's just fuzzy math and hype. Here's to some meat in the next one.

<comment><USER4> Awesome comment. Wow, "scrum fanboyism" is definitely not what I was going for there - just trying to poke fun at how these teams have changed over the years.

Yes, the technical details are on the way - we're excited to share them.

</comment></comment><comment><USER5> Are these numbers based on speculation?

Since when serious applications are written in 1 week in 2008? You could write 1-week applications since the dawn of computing. Haven't you heard? BASIC was written by B. Gates and P. Allen in 8 weeks.

</comment><comment><USER6> Has software development really gotten 160x faster in the last 10-15 years? It's a moving target, of course, since people keep trying more ambitious things as tools get more powerful. But that number doesn't feel right to me.

On the other hand, the main argument - that the percentage of effort occupied by deployment has grown - may not need that figure.

<comment><USER7> Considering there are things you can do with web apps now that you couldn't do at all in 1996, I'd say for some use cases web development is now "infinitely" faster.

<comment><USER6> Sure you could. You could have built the web and everything you needed and then written your app :)

"Development is faster" presumably means "relative to what people are trying to do". That's the sense in which the OP uses it, because he's talking about development going faster per project. The concept is problematic since what projects consist of has changed as much as everything else has.

</comment></comment></comment><comment><USER8> The figure that are stated in the article are quite obviously made up but conceivably close to real life scenarios. I think the author is way off on the last example, though: "It then takes just one person a few days to provision new resources from IT or a fast-moving hosting company, install the default web stack, and do the initial deploy." I can see that for an IT department but not for a fast-moving hosting company. With all of the deployment tools available, most of the deployments I've done have ranged from a few minutes (RailsMachine) to a few hours (barebones install). All that aside, though, instant deployment would be pretty darn cool.

</comment><comment><USER9> Comment I left on the post:

As a potential customer, instant deployment is not what appeals to me. If I knew that 1 day after my code was done, my app would be working just perfectly, I'd be a happy camper. Instant deployment = cool but not mission critical... at least not for me. (Now, if deployment took 2 weeks, I'd be pissed.)

What appeals to me about Heroku is exactly what I like about my MacBook Pro: It just glides. I don't have to think about it. Things just work (at least in theory.) It's not the time, but the thinking about deployment that I'd like to remove.

I just click the magic deploy button (command) and tada, "It's a live." And if I hit the jackpot and land a massive client or show up on TechCrunch, things will just keep on gliding smoothly. And hopefully, I'll make more money and serve more clients.

That is the ideal I'd like to reach.

Now instant deployment might be something I fall in love with (like git) but I'm not there yet. I just want things to "just work" for now.

<comment><USER4> Ah, bingo. Yes, "it just works" and "it glides" are exactly what we're going for. Our version of the magic deploy button is "git push". We've got some great detailed posts coming up about this.

The thing that we've discovered is that instant and "it just works" are tightly related. If something takes minutes or hours to do (even if automated), it's likely that there are many steps involved - each of which is a potential point of failure.

<comment><USER10> Your last sentence rings especially true. Each step is a potential point of failure. I tried to set up cap and passenger one day but got stuck on a couple of steps. Sure I could have plowed through and gotten it to work eventually, maybe switched to a different computer or whatever but time is precious. The word "easy" is thrown around too casually in tech circles. Today I've got three domains running on herokugarden and it still shocks me how short the procedure is. Thanks.

</comment></comment></comment><comment><USER11> The historic argument you guys are trying to push trough to enhance your pitch is shadowing the purpose of your heroku itself because of its controversial nature.

I will advise you remove the whole history thing from your argument and take a different approach of selling your tool which I think has good potential.

<comment><USER12> I agree. It's enough to say that development has gotten much faster, but deployment is still a slow pain in the neck, and you are the guys who are going to change deployment. Most of the time the smooth, concise story sells best, in my experience.

</comment></comment><comment><USER13> A valid point undermined by the extravagant attempt to prove it.

<comment><USER14> I think the numbers/graph are necessary. Of course they are made-up, but the point is the trend over time. I find this argument quite compelling, and I wouldn't have found it interesting had it just been a qualitative description of these trends.

</comment></comment><comment><USER15> I get the sense that your numbers are based on speculation rather than hard data.

I have several examples of sub 1 week development time to first release from projects we've done in the last few years. But then the initial deploy for those projects, on average, took about 20 minutes of one person's time.

Iterative development often goes hand-in-hand with iterative deployment. Somewhere during week 3, we'd spend an hour moving onto an automated build and deploy system, and from there on out it would be push-button.

Once, and only once, the site got big enough to need its own dedicated box at a colo, so that added an extra couple days building a server, driving it to the facility and moving the site. But by then we were 3 months into development, so the effort was nowhere near 25% of our total spent man hours.

Cool idea for a service though. But maybe you're painting the picture a little more dire than it needs to be.

</comment><comment><USER16> One thing his historical comparison completely neglects is there was a time when people actually cared about performance (or rather they were so constrained by it they were forced to care). On the other hand most of these one-week wonders suffer from "we'll just throw more HW at it" syndrome.</comment>