The Security Dilemma of Cyberspace: Ancient Logic, New Problems - <AUTHOR> https://lawfareblog.com/security-dilemma-cyberspace-ancient-logic-new-problems <comment><AUTHOR> "Operation "Nitro Zeus" illustrates this thinking. As Buchanan explains, in the early stages of the Obama Administration, U.S. hackers went on an expansive hunt for "zero-day" vulnerabilities in Iran's strategic infrastructures. The raid targeted the Fordo nuclear facility that Washington suspected was purifying uranium to weapons grade. But that was not all. The United States also penetrated Iran's financial, transportation, and air defense systems. The invaders acted in anticipation of the possible failure of diplomatic efforts to curtail Iranian enrichment activity peacefully. They crafted weapons of war even as they strived to avert it."

<comment><AUTHOR> I thought the author's idea that it is prohibitively difficult to create exploits in a vacuum, and need to know the specific configuration of your adversary's systems in order to be able to attack or counter-attack was interesting.

If you think it is appropriate for your nation to maintain the ability to respond, then, pragmatically, you support either your nation intruding in other's systems or your nation maintaining a weak deterrent response.</comment></comment>


Vanguard Founder Jack Bogle Says 'Avoid Bitcoin Like the Plague' - <AUTHOR> https://www.bloomberg.com/news/articles/2017-11-28/vanguard-founder-jack-bogle-says-avoid-bitcoin-like-the-plague <comment><USER2> "Bitcoin has no underlying rate of return," said Bogle, 88, who started the first index fund in 1976. "You know bonds have an interest coupon, stocks have earnings and dividends, gold has nothing. There is nothing to support bitcoin except the hope that you will sell it to someone for more than you paid for it."

Didnt quite understand this quote - is he down on gold too?

<comment><USER3> _There is nothing to support bitcoin_

That's what I've been realizing latey: there _is_ something to support Bitcoin, and that something is the blockchain. The reason banks and big financial companies are interested in it is _not_ for its speculative value. They are interested in the fact that it is both decentralized and immutable. As long as financial corporations are willing to exchange Bitcoin for USD, then Bitcoin is a reasonable store of value.

There's also value in Bitcoin because it's a good entry point into other cryptocurrencies that could turn out to be better general-purpose exchange media, like Litecoin.

It's a bit like US Treasury Bonds. Nobody buys them for the tiny rate of return they offer, but they have genuine, valid use as a financial instrument. Normal people, however don't typically buy them, and normal people probably shouldn't be buying Bitcoin either unless they have some spare cash to gamble on it.

<comment><USER4> Nah. Banks and big financial companies interested in the blockchain will spin up their own version without tying it to the unstable mess of bitcoin. e.g. [https://ripple.com/](https://ripple.com/)

And people do by treasury bonds for their rate of return, when the investment must be as riskless as possible.

<comment><USER5> There's no point to a decentralized database like a blockchain when you run it from a handful of centralized chokepoints which are easily censored or pressured by political actors. You may as well just use a SQL database that everyone syncs up periodically. Bankcoins like Ripple are pointless.

People buy treasury bills when they care more about return OF capital than return ON capital. Bitcoin and gold are also solid assets in this regard, particularly because they are bearer assets that don't represent a liability or promise-to-pay by some third party.

<comment><USER4> And yet, banks and financial institutions are vastly preferring blockchains that have been uniquely developed for their needs. There's very little appeal in simply accepting the transaction costs and times in bitcoin when you can spin up a version with whatever spec and interoperability you desire.

If by 'return of capital' you mean a propensity for an asset to retain its value, bitcoin is utterly inappropriate. Even gold is far too volatile and is only rationalised on those grounds against (very) long tail risk.

<comment><USER6> Their "preference" for "blockchain" is borne out of political necessity. I say this not to trivialize it, but to explain.

Banks are massive, massive institutions, and it is hard enough to coordinate people and resources within them, much less across them.

Back-office operations are indeed cumbersome and a major cost-center, and there is much value to be captured in simplifying all of it. If "blockchain" can serve a Schelling point to help get right people in these institutions to talk with one another so as to get on the same page, then there's a chance that some progress can be made towards solving these problems.

It's "blockchain" because "blockchain" is the new hotness, that's all. Any real world, inter-dealer distributed ledger will, in all practicality, be more like a cluster of SQL databases backing some kind of state-machine replication protocol than a cryptocurrency.

Happy BTC10k, by the way.

</comment></comment></comment></comment></comment></comment><comment><USER7> Bitcoin is becoming a value store. With slow transaction confirmation and high fees, what are the real-world applications for Bitcoin supposed to be now? And for those applications, are there not other cryptocurrencies more specifically targeted at being better at that application?

<comment><USER8> Nobody keeps bitcoin to store value. They keep it because they think it will appreciate in value. Why? Because of has went up in the past.

If people lose faith that it will keep going up, then it stops holding value.

It's pure speculation based on the greater fool theory.

<comment><USER9> A lot of people do use it to store value, its deflatory nature makes it ideal for that.

<comment><USER8> Price is based on supply and demand. It's only deflationary if demand stays the same. But if people stop believing bitcoin is going to MOON, then demand can/will fall.

There are a limited number of beanie babies too.

</comment></comment></comment></comment><comment><USER10> The biggest red flag is it's clear the exchanges don't have the liquidity for everyone to cash out. Since there are no banks to provide credit lines it's entirely dependent on people getting into the market to provide the fiat currency to cash out.

A run on these exchanges will destroy the value of bitcoins and cause a panic.

<comment><USER11> How is it clear that the exchanges don't have liquidity?

<comment><USER10> The way the system works provides little in any sense of a line of liquidity. Unlike normal banking the exchanges operate ledgers where the fiat currency they have on hand is directly tied to how many people have put fiat currency into the exchange network. Given the extremely rapid rise of the price of BTC, it's extremely unlikely they have anywhere close to the amount of fiat currency if large numbers of people cash out.

<comment><USER9> Exchanges are not market makers. They don't have to be able to allow everyone to "cash out", they are not banks.

Exchanges provide a venue for buy-sell to meet. If someone wants to "cash out", someone has to want to "cash in". Lately every time the price dropped folks saw it as the time to buy more, or just "finally buy bitcoin, because it'll go up soon".

A big crash happens when the equilibrium shifts drastically, when more people wants to get out than get in, it'll crush the price (because more people wanting to get out will lead to even more people wanting to get out).

</comment></comment></comment></comment><comment><USER12> Isn't the "value" of a currency in it's relative stability over time? That is, the confidence by those who use it that it will be exchangeable for roughly the same amount of goods and services today, tomorrow, six months or a year from now (accounting for nominal inflation, of course).

If you accept that premise of what a viable currency should be, then it seems like the volatility of Bitcoin and other cryptocurrencies have thus far made them failures. I mean, if a national currency had similarly rapid changes in value, wouldn't we see financial chaos, riots, and revolution?

</comment><comment><USER13> > There is nothing to support bitcoin except the hope that you will sell it to > someone for more than you paid for it.

Not everyone agrees with that statement. For example, see:

[https://news.ycombinator.com/item?id=15797380](https://news.ycombinator.com/item?id=15797380)

[https://news.ycombinator.com/item?id=15796880](https://news.ycombinator.com/item?id=15796880)

<comment><USER14> The second link does nothing to refute the idea that bitcoin, as an investment instrument, is purely speculative. The author simply argues that bitcoin is unique among money-like assets.

The first link describes the hypothetical utility of bitcoin, but does nothing to explain a natural reason that bitcoin should have returns or appreciation. In fact, since the author describes bitcoin as a "medium to store wealth," the assumption should be that every buying transaction has an opposite selling transaction, which means there would be little to no overall price impact from any of the described activities.

I wouldn't scramble to dismiss what Jack Bogle says. He may be 88, but he has earned a reputation as a clever, knowledgeable, and decent fellow.

</comment></comment><comment><USER15> "There is nothing to support bitcoin except the hope that you will sell it to someone for more than you paid for it."

Genuinely asking, isn't this the plan for many people? How is bitcoin different than other commodities or property in this regard? Would Bogle say the same thing about those investments?

<comment><USER16> There is a big difference. If you buy wheat futures and nobody wants to buy them, you can take delivery of a bunch of wheat. If you buy investment real estate and nobody wants it, you can live in it, or build things on it. If you buy stock in a random Fortune 500 company and nobody wants it, you can take possession of a bunch of desks or factories or inventory or whatever their deal is. And if you take possession of a bunch of US currency you can be assured that you'll be able to use it to settle debts or pay your taxes as long as the US government is around and has better guns than everyone else.

Bitcoin is different from all those things.

<comment><USER7> > If you buy stock in a random Fortune 500 company and nobody wants it, you > can take possession of a bunch of desks or factories or inventory or > whatever their deal is

Not really, the value of the stock approaches zero, and when the company closes up shop I doubt you get anything at all as an investor. Assets would typically be sold to pay off outstanding debts.

<comment><USER17> More fundamentally, the value of a share of stock is tied to earnings. When companies accumulate capital beyond their horizon of their operational needs, they issue dividends.

Yes, companies can go out of business, but that's not the point Bogle is making. He's not saying stocks are riskless. He's saying that their valuations are tied to notional future earnings and the dividends they will generate.

</comment></comment></comment></comment><comment><USER18> Forcing people into 401Ks was the greatest financial hack of the century. The bankers don't want competition for "their" money pile.

<comment><USER19> That is an interesting argument. These guys are selling financial products that are explicitly not pro bitcoin (yet). If they see value in bitcoin then they would probably pump them. That is most likely too soon. In a few years you will probably hear that bitcoin is a hedge against global instability. It might be rational for oil to trade in bitcoin. It's also a big way for the Chinese to get money out as well as the Russians. North Korea has nothing to lose with bitcoin unless it pisses off the Chinese and Russians.

</comment></comment><comment><USER20> > It's "crazy" to invest in the digital asset, he added. "Bitcoin may well go > to $20,000 but that won't prove I'm wrong. When it gets back to $100, we'll > talk."

One more in the many death's of bitcoin...

[https://99bitcoins.com/bitcoinobituaries/page/10/](https://99bitcoins.com/bitcoinobituaries/page/10/)

</comment><comment><USER21> Over the past 7 years, Bitcoin has outperformed every security and portfolio that Jack Bogle has recommended.

<comment><USER22> This is pretty disrespectful to Jack Bogle.

Vanguard is almost singlehandedly responsible for returning trillions of dollars of costs, in the form of fees and underperformance by active managers, back to investors. Millions of investors have benefited.

<comment><USER21> Bitcoin has been a bubble since $1 and $100 to these people.

<comment><USER23> What evidence is there that it isn't a bubble? People buy Bitcoin only because they think they can sell it higher. Eventually, you will run out of greater fools.

</comment></comment></comment></comment><comment><USER24> He has a point since I could always go down to the reserve and trade my dollars in for cold hard gold. Oh wait...

</comment><comment><USER25> The common sense for the wealthy investor should have been to invest 1% of his portfolio into bitcoin and other digital currencies. If he did that two years ago he'd get x15-30 returns, making Digital currencies over 10% of his portfolio. That would be the time to "cash out" or "scale out" the original investment back into traditional channels.

Is it late now? I think wait a bit and then start DCA that portion of your portfolio.

<comment><USER26> Why? Why was this common sense? What other things should people have invested 1% of their portfolio in? Is every thing that has the possibility of increasing in value dramatically worth investing 1% of my portfolio?

<comment><USER25> Diversification. Preferably your portfolio will have a tech allocation and crypto could be part of it. Certainly there are many things that will appreciate ten folds in a year; and most things that don't.

Had I opted for diversification in my investment, I'd have seen my wealth even triple that now. And this is not in crypto only.

<comment><USER27> I think the question is: given that you can find at least 100 long shots out there, how could you know this was among the <= 100 that you should put 1%+ into?</comment></comment></comment></comment>


Random startup tip #1: Picking a name - <AUTHOR> http://blog.mattrubens.com/2007/07/06/random-startup-tip-1-picking-a-name/ <comment><USER2> What I find more important now in retrospect is to agree upon a name as one of the first things when working together with someone. _Not_ because the name is important, but because this will be the first decision where you can find out how good you will work together when you have an argument. If you can't decide on a name, work with someone else.

<comment><USER3> This is a good point. We find it also works as a predictor of success at the level of whole startups. When a group is spending half their time agonizing over what to call themselves, that's probably a sign of deeper breakage.

</comment></comment><comment><USER4> Is this a troll, or did the author not realize that urbandictionary isn't exactly a reliable resource?

The entry in urbandictionary for "googe" tries to be funny but is clearly fake.

I suppose it's technically possible for someone to name their startup something like "Blumpkn", but I'd say this is a non-issue.

It's a good idea to think about a word before buying it as a domain, but that's common sense.

<comment><AUTHOR> Fake meaning what? Doostang is just a ridiculous name regardless, but it might have served Google well to see if 'googe' had any other connotations (whether or not they're in Merriam/Webster) before putting it in a logo.

<comment><USER4> First, the stem of the cherry was intended to be the L, but it's acknowledged that it doesn't look that way.

Secondly, "Googe" doesn't connote anything, unflattering or otherwise.

When I say fake, I mean that people try to invent new words on urbandictionary, and not words that people have ever used before. Don't believe me? Then just click a letter on urbandictionary and read the entries sequentially instead of by popularity.

To sum up: the examples used in the article are completely ineffective, but know what words mean before attaching them to the identity of your company.</comment></comment></comment>


Three hundred programming interviews in thirty days - <AUTHOR> http://blog.triplebyte.com/three-hundred-programming-interviews-in-thirty-days <comment><USER3> It's great to see some objective research being done on this, and I am very interested in following the results.

They mention evaluating the effectiveness of giving a candidate a project to do "in their own time." I recently had a interview that included this and I can share the result: I accepted an offer from a different company that didn't require it. I doubt my life is that different than anyone else's, with a full- time job and a full-time life outside of work. Spending that much time to qualify for a single job is too much to ask of anyone. If it were to pass a generic proficiency certification applicable to many positions, I would consider it, but this does not scale if a candidate is applying for multiple positions.

<comment><USER4> I did this once. After a take home quiz that took about an hour, I had a 6-7 hour "homework" problem. I did it, and sent it in. It took the company a month to get back to me, though I did stay in contact with the recruiter. The final answer, delivered by the recruiter, was "we've decided not to more forward at this time." No other feedback. I have no idea if anyone even looked at it - <USER2> never did speak to a dev at the company.

This experience left me embarrassed, honestly. I don't think it reflects well on me that I put up with it even once.

At this point, I will participate in tech interviews, but I won't do any more take-home assignments. I suppose this could change under different circumstances.

Gayle Laakmann McDowell, who wrote "Cracking the Coding Interview", wrote an insightful post her blog about this. It's titled "Companies who give candidates homework assignments: knock it off!"

[http://www.gayle.com/blog/2013/09/18/companies-who-give- cand...](http://www.gayle.com/blog/2013/09/18/companies-who-give-candidates- homework-assignments-knock-it-off)

She mentions something I see as a problem as well - these tests allow the company to burn a lot of a candidate's time, but not the other way around. I may have burned 6+ hours interviewing unsuccessfully at google, but they parted with 6+ hours of developer time as well (though I did spend a lot of additional hours reviewing algorithms and doing sample problems). This creates a natural set of checks and balances that don't exist with homework assignments.

She does have some good suggestions about how to use a take home if a company is determined to do it. To me, a particularly insightful one was to have a high (she says 90%+) passage rate.

I agree. If you're going to ask a candidate to spend a day on a homework assignment, you should be very close to an offer. If you're using it as an early filter, you are wasting a tremendous amount of developer time.

<comment><USER5> I've seen several companies giving out such homework to me or my friends back in college. And to be honest, none of the are among the "great culture company list" people are talking about. Beside all the issues you mentioned above, those tests don't have a clear PASS line. You spent hours on that and can be rejected for no reason. Back in college I've done a code challenge from Box and my solution ranked No.1 in their leaderboard (although nearly 100 people are No.1 with the same highest score) and I was rejected without an interview. Another company I don't remember rejected me because I used (x * 8) not (x << 3). (But I'm interviewing a web company and I was writing python, not C in some embedded system) Most of the time I just don't know what the hell they're looking for. So now I reject all those HW or code challenge unless they agree to pay me for the time I spend. (Obviously no one has ever agreed my term so far.)

<comment><USER6> Even in C I would point out that the compiler will likely make such a trivial optimization for you. That was clearly a company full of morons.

<comment><USER5> Regardless of the compiler, let's say (x << 3) is slightly more efficient than (x * 8), I don't see it make such difference since my job will probably be writing code for webapp sort of things. For the things I do I actually believe readability advantage of (x * 8) overcomes the efficiency disadvantage (which doesn't even exist with modern compiler).

Some interviewers (not just the "homework") raise their bar by not allowing you to make any tiny mistake. And I just don't get it. If someone is good enough to write something like a lite version of Hacker New website in hours, I'm not going to turn him/her down because of such mistake.

<comment><USER7> What the hell. A developer who writes x << 3 instead of x * 8 when writing code that is logically doing multiplication is not a good developer. Written code is designed for human readability; the human should not be having to convert that mentally to its math equivalent to understand what operations are taking place. In nearly every possible case, this is at best a micro- optimization. At worst, a symptom of someone who writes unnecessarily obfuscated code to show off their knowledge in a manner that only hurts the maintainability of code by a team.

If I found bit shifting in an interview code sample as a replacement for basic multiplication, I would ask the developer why they chose to do it that way. They'll either a) calmly explain that their computer science classes taught it that way or it's a habit they've adopted after writing code for embedded systems or similar where the optimization actually made a difference, or b) their ego will make an appearance with a "because I'm so senior" attitude. The latter is not a good sign.

<comment><USER8> Or it's idiomatic for that language.

</comment></comment></comment></comment></comment></comment></comment><comment><USER9> Imo, this post did not organize its data and findings into a coherent presentation.

For example...

 _> The fizzbuzz-style coding problems, however, did not perform as well. While the confidence intervals are large, the current data shows less correlation with interview results. [...] The coding problems were also harder for people to finish. We saw twice the drop off rate on the coding problems as we saw on the quiz._

I read that paragraph several times and I don't understand what he's actually saying. If those candidates "dropped off" on the fizzbuzz, were they _also still kept for further evaluation_ in the following extended coding session? A later paragraph says...

 _> So we started following up with interviews where we asked people to write code. Suddenly, a significant percentage of the people who had spoken well about impressive-sounding projects failed, in some cases spectacularly, when given relatively simple programming tasks. Conversely, people who spoke about very trivial sounding projects (or communicated so poorly we had little idea what they had worked on) were among the best at actual programming._

For the fizzbuzz failures to be non-correlative and counterintuitive, it means he did not reject them for failing fizzbuzz and they later ended up doing spectacularly well in the larger coding sessions. If that's what happened, then yes, that is a very counterintuitive result. What were the topics of the larger coding sessions?

<comment><USER10> Hey! Author here. Yes, we did not know that the screening steps were meaningful, so for the first 300 applicants, we interviewed everyone, even people who performed badly. We then looked for correlation between screening step scores, and programming interview results. Doing well on the fizzbuzz problems was not very correlated.

By dropoff I mean people who left during a step, and never logged back into our site.

<comment><USER11> I appreciate that you guys might not be statisticians, but if you're going to try and analyze data like this, you simply must address survival bias. As it stands, these data are meaningless unless you assume dropouts are completely unrelated to your screening.

You claim doing well on the Fizzbuzz wasn't correlated with interview performance, but you also said "We saw twice the drop off rate on the coding problems as we saw on the quiz."

An alternate explanation for your finding then is, more low-quality candidates drop out of the process when given FizzBuzz, leaving a relatively homogeneous pool of higher-quality candidates for the later interview. This effectively reduces the ratio of meaningful interindividual differences relative to noise, which will reduce the correlations.

In all likelihood, _both_ of your correlations are low, but the idea that coding is less predictive than a quiz could be __purely a statistical fluke __due to survivor bias.

<comment><USER10> All candidates did both screens (quiz and fizzbuzz). The correlations were calculated against the same population. Now, I agree that survivor bias could affect the quality of these results (we know nothing about the significant % of people who dropped out). But it's not really possible to solve that problem outside of a lab. I don't think it's an argument to not do analysis. For now we're simply trying to minimize the dropoff rate, and maximize correlation. The quiz was better at both.

<comment><USER11> Well, the candidates doing both screens is better, but it doesn't totally solve your problems.

It doesn't address the survival bias issue, and when you say a significant percentage dropped out, that's not reassuring. But it's not the case that you need a lab to solve the problem. Even a basic questionnaire of programming ability self-assessment might tell you if there are meaningful differences in the population that quits your process. At the very least, you should understand and talk about survival bias in your article to indicate you're aware of the issue.

Even if you still want to claim a difference between the quiz and coding exercise, you're not yet in the clear. For example, did you counterbalance the order you gave them to people? E.g., if everybody did the quiz first and the fizzbuzz second, that meant they were mentally fresher for the quiz and slightly more tired for the fizzbuzz, which could again create a spurious result. And this definitely doesn't require a lab to test.

Don't misunderstand me, I appreciate your attempts to quantify all this, and I actually think you guys have roughly the correct result (given the limited nature of fizzbuzz-style coding), but when you step into the experimental psych arena, you need to learn how to properly analyze your data. Given that your business is predicated on analyzing the results of how your hires do in the real world, you need to really up your analytical game.

</comment></comment></comment></comment></comment><comment><USER12> "as soon as we started doing them however, I saw a problem. Almost everyone was passing."

Why is that a problem? Maybe almost everyone is decently good (as evidenced by having a string of jobs, and presumably, references), and your interviews are creating tons of false negatives.Or heck, vice versa. You don't know.

You are presuming your conclusions. You have no basis to make conclusions yet, you just have incomplete data. It's _iteresting_ data, and I'm gleefully happy that somebody is looking at this in the context of programmers (too many studies are very broad, across many different career/job type, IMO). But I think all you have right now is data. Fishing for correlations at this point is nearly bound to lead you astray.

With that aside, I'm very interested in the eventual correlation with test performance and job performance. I'm biased - I dislike IQ tests, but I must admit there is a lot of research on them out there. For me personally, I perform spectacularly on this sort of test, pretty poorly in whiteboard tests, so-so in pair program to get a job, and generally top of the heap in actual job performance. It would definitely help me personally if these tests were true. Yet, still, I wonder, do they measure "get things done"? Do they measure "don't piss off the CEO/customer" skills? There's a ton of things that I think are important beyond pure cognitive skills.

<comment><USER13> If they want to market themselves as being great at screening candidates so that they send only the best to interviews at client companies, then a test that 90% pass isn't of much use to them.

</comment></comment><comment><USER14> Here's the part that really seems to matter the most.

        This does create some danger of circular reasoning
        (perhaps we're just carefully describing our own biases).
        But we have to start somewhere, and basing our
        evaluations on how people write actual code seems like a
        good place.The really exciting point comes when we can
        re-run all this analysis, basing it on actual job
        performance, rather than interview results.

Absolutely. Results on the earlier screens and results on the later interview aren't exactly independent variables, and neither is the one that really seems to matter - subsequent on-the-job success. There are all sorts of biases and confounding factors likely to be shared between them, especially since there's no indication that the later interviews were even done blind w.r.t. the earlier screens. Until then, we're just measuring correlations between different interview techniques, and it should be no surprise that two different kinds of code-focused interviews show the highest correlation.

<comment><USER10> Author here. We do do the interviews blind to earlier screening results. That's clearly vital. But you're totally right that job performance is the real thing that matters

<comment><USER15> So, the jury is out until a followup. How to measure productivity without bias? Applying Demarco and Lister's "Coding War Games" approach may work between companies, but how to apply within a single company? What would be the standard error? A dozen new hires is a lot different than 300 interviewees.

What kind of skill set? My previous company contracted with Galye Laakmann McDowell to coach the engineers before an acquisition due diligence. I thought it was unnecessary because we rarely used O() notation in our day-to-day. Since then, I'm designing a machine learning product and find sections in McDowell's book extremely relevant. My takeaway is that it all depends: are you hiring an architect or a hiring a construction crew? How do you measure such different skill sets? Or, how to measure with a job that combines both?

</comment></comment></comment><comment><USER16> I got a phone interview with Harj and wasn't considered for anything further.

I'm not sure what kind of hackers they were looking for, but I've been directly involved with creating the infrastructure used in marketing campaigns with the likes of CNN, McDonalds, Infiniti, and more. I've turned an idea into a company with 8 full time employees and have investors seriously interested in one of my side projects. I'm currently involved with leading a project that integrates with a large bank.

I'm a full stack ruby dev learning clojure in my spare time and heavily involved with self improvement. Anyone who watches me for a moment can see that I can solve problems very quickly. I didn't care much about being selected, I have a solid job and offers coming in.

Would anyone who got selected by Triplebyte care to list their credentials/achievements? My main motivation was to see how I compare against others at my current level.

<comment><USER17> I fell through at that stage as well, which I think was surprising only because their value prop is connecting you to YC startups. I'm a former early employee at a YC startup having contributed to nearly half of its codebase and other YC startups constantly attempt to recruit me.

Oh well, I'm at a non-YC startup now and very happy. I guess the discrepancy lies with Triplebyte selecting engineers that Triplebyte wants, not what is actually representative of what YC startups want. Overall it's still probably a decent signal—you'd have to expect some false negatives here and there (and I'm not a rockstar dev either).

</comment></comment><comment><USER18> Very interesting methodology, but it would be very nice to correlate this data with long-term job performance. Interview decisions (of which of course you get more than long-term results, and they are clearer to quantify) are hopefully, but not necessarily an indicator of whether an employee works out for your company. Otherwise you run the risk of optimizing the quiz/screening process around metrics that influence your interview (i.e. test for how you personally weigh performance indicators, not for how these performance indicators actually affect performance)

<comment><USER19> This is exactly what I came here to say.

It's all well and good that certain things (quiz) correlate more to performing well in a long-form remote coding task.

But part of the problem of hiring is that these test-style code questions are themselves not necessarily good indicators of future job performance.

The only way to truly analyze whether a given technique is working or not is to follow the candidates through into their jobs and see which ones actually become good long term employees.

</comment></comment><comment><USER20> One of the most enjoyable and mutually effective interviews I had included a one-hour pairing session which was language agnostic that required me to describe how I'd implement a basic data structure. The interviewer drove and implemented. This was then followed by a day long session of pairing on real problems with a number of different interviewers. Lunch was spent with some of the other team members, where we discussed basic things like culture, day-to- day affairs, and each of our histories.

This was a great approach to me, because it didn't particularly focus on anything outside of the present. We worked on solving real problems, and contributing to the project. It's a great, low-stress, method of gauging if someone has the chops for what is typically the "day-to-day" life at the given shop.

</comment><comment><USER21> > _In our first 30 days, we 've come up with a replacement for resume screens, > and shown that it works well._

What's the metric that shows it works well?

> _The really exciting point comes when we can re-run all this analysis, > basing it on actual job performance, rather than interview results._

And how _precisely_ do you measure job performance? If this is achievable, I've got a line of companies out my door that would love to pay for a service that systematically measures job performance.

<comment><USER10> Measuring job performance is hard. A good analogy is measuring intelligence. The IQ test is clearly bullshit (it does not come close to reflecting the complexity and high dimensionality of human intelligence). That said, it's a useful research tool. Across a large number of people in aggregate, it can be used to learn things (like that leaded gas was a really bad idea). We'll be doing the same sort of (very imperfect) measurement of job performance.

<comment><USER22> I'll be very interested in hearing what your measures are, and how you'll account for things like employees being in a bad environment/having bad bosses. It's a complicated thing to measure.

Are you guys planning on publishing any of this material in a journal or research venue, or will you keep the results to blog posts?

</comment></comment></comment><comment><USER23> I really think technical, or any other kind of hiring is broken. As you showed best indicators are questions (like your quizzes) or by showing "live" you can rather than a pretty CV, certificate or a fancy university name (I'm talking about tech not medicine, construction engineering or others that really require those).

I have been rejected many, many, many times because the first screening (CV check by non-technical recruiter). My last example was at a well know tech startup were I had to hack my way to get noticed in order to get the first interview. The funny thing is that I was the fasted candidate to get hired + I won a company-wide award for my work at the company just 4 months after joining.

I haven't finished a degree because I thought was boring and I was learning things I already taught myself before, but this fact makes my resume go down the list very fast. Because interviewers don't have time to lose and thousands of candidates to check I'm sure they will find very useful the use of technology on getting those good prospects in front of everyone else.

Something I've seen many times at my past jobs is having good technical applicants, some of them are even referred by one team member and are turned down later because culture. I don't know why but engineers and technical people are more likely to fail at those than others. The surprising thing is that they check culture as the last step because those who can run those type of interview are a few and can't become full-time culture keepers. This is an enormous waste of time and resources for the applicant, the interviewers and the company itself.

<comment><USER24> For people without impressive CVs networking is key. Go to conferences, hackathons, and meetups to meet people. This will get your CV past HR almost always.

</comment></comment><comment><USER25> >Suddenly, a significant percentage of the people who had spoken well about impressive-sounding projects failed, in some cases spectacularly, when given relatively simple programming tasks.

This indicates to me that either the "simple programming tasks" are not well- designed, or the the discussion about the past projects was not long enough. It still sounds like this interview process is only identifying candidates who are good at coding while someone is watching over their shoulder.

However, what I find to be the bigger issue with this article is that "success" is considered to be "passed the interview". Ultimately, all this article tells us are what currently correlates with qualities Triplebyte likes to see in candidates, not what correlates with good hires. To be fair, they do mention this at the end of the article.

</comment><comment><USER26> The skill of the interviewer in interviewing candidates should correlate just as strongly with everything else. How are you capturing that? I ask because our #1 determinant after an at home code sample is the #3 thing you find are not predictive.

Are they asking critical questions on what decisions and trade-offs were made? Their past projects, can they explain well the reasoning for choice of tools used? Can they talk about what types of improvements they wanted to see in the pipeline process of build-test-deploy?

I'm just surprised that this question is singled out as "poor."

</comment><comment><USER27> Preamble: not middlebrow dismissal, I really like what they're doing here and will pay attention to them going forward. This is just picking a nit that my hypersensitive self just can't resist:

"Fizz buzz style coding problems are less predictive of ability to do well in a programming interview"

I'm sure this is 100% true, but I thought the point of fizzbuzz-type problems were to weed out people who couldn't program at all? It's not to identify good programmers or even competent ones, it's to identify blatantly incompetent ones, which are surprisingly common even when hiring in SV.

I've never personally asked fizzbuzz when interviewing because my company's hiring process seems to do well enough to not require it. However, based on what I read here it's also very good for filtering out narcissistic divas (i.e., the occasional HN posters who pop a monocle when they get asked fizzbuzz: "how dare someone ask a dumb question that is beneath me?!? Needless to say, I walked out of the interview immediately! Harrumph!").

Maybe Triplebyte's article is using the term "fizzbuzz-type problem" to refer to any contrived programming problem, but in common usage fizzbuzz-type problems are bozo filters that serve no higher purpose than filtering out bozos.

<comment><USER10> Yeah, we're just using it to mean a asking a simple programming problem (of fizzbuzz difficulty).

<comment><USER28> But if you're not evaluating the ones who fail, then what's the point of it? you're just grading for style points?

What I take away from your article is that being able to solve a FizzBuzz- style problem isn't correlated with being able to do a more intensive programming task later, which surprised me. What I take away from your comments here is that among people who were able to solve a FizzBuzz-style problem, some variety of style point scoring isn't correlated with being able to do a more intensive programming task, which... okay, that's totally plausible, and much less surprising/interesting.

</comment></comment></comment><comment><USER29> I've done over 1000 interviews and my experience agrees with their findings that talking about a project is not a good predictor of coding. I usually left detailed resume questions for the end since so many candidates would bomb the coding part of the interview.

I'm surprised they didn't get stronger results from fizz buzz, but I noticed among the candidates I saw that the percentage of 'non-coders' is substantial but not a majority.

One thing missing from this investigation is a measure of solution quality. A good portion of candidates who actually finished coding questions with me ended without thoroughly understanding how their code worked and/or had code that would be hard to maintain. Other candidates would write top-notch code but were unable to explain their thought process to some extent. These are critical pieces to the interview that contribute much more 'color' than 'score' and are important to note.

<comment><USER14> I really don't mean to cast aspersions on your interviewing abilities, but the common factor in all of the interviews you've conducted is you, and _how_ you talk to a candidate abut prior projects can really matter. At one extreme you have "tell me about your last project" free-form. At the other you have highly-specific drilling down on a relevant technical issue (e.g. flow-control mechanisms in Ethernet networks) or behavioral characteristic (e.g. openness to collaboration). Which of these have you tried? Have you observed any difference in effectiveness between them? My own experience, from interviewing people who conduct interviews themselves, is that people who say "asking about projects doesn't work" just don't know how to do it right. Again, I'm not saying that applies to you personally, but what information can you provide to refute the observation?

<comment><USER29> I totally agree with you that the "ask about projects" question requires care. And, yes, I did see differing results depending on how the discussion went. When I did the question, I've run it as both a 1-minute quickie and a 15-minute conversation. I tried my best to dig an solid technical exposition out of the candidate if I could.

Typical cases where the project discussion was not helpful:

* Senior engineers could often talk in great detail about things they'd built, but then would most often bomb coding questions.

* A lot of MS/PhD-level new grads could give great talks (about almost anything) but again would bomb coding stuff.

* Lots of ESL candidates struggled tremendously to convey basic project details, but nevertheless were solid coders. From my TAing experience in grad school, I noticed that there are a lot of ESL CS students who have poor oral communication skills but are orders of magnitude stronger at written communication. (This made strong textbooks and written references crucial to the courses I TAed).

* Sadly we hired quiet a few senior / PhD-level people who could give great talks, were highly responsible, but were eventually fired for poor coding. (There were also some management disasters associated with those cases, but stronger coding skills would have helped them nevertheless).

Interviewing has both subjective and 'objective' parts. The most objective data one can collect are things like completion times, the actual code written, and (perhaps) raw contextual data like a measure of the candidate's mood (were they sweating like mad?). Project discussions rarely recover solid objective data. So, I'd definitely recommend against outsourcing those questions to a service like Triplebyte, and junior employees should probably also not be spending 15 mins on projects for /every/ candidate.

</comment></comment></comment><comment><USER30> It seems like they're only evaluating phone screen methods against their pre- designed coding interview problem? But what if there are issues with that problem?

There seems to be a big assumption that "our programming questions are going to be good and predictive, even if everyone else's are bad." What if being able to describe in-depth a past (real) project correlates just as well (or better) to on-the-job performance as being able to design and code one of their artificial ones? Or what if those artificial ones just don't correlate that well with on-the-job performance in the first place?

It is definitely harder to BS-detect/grade, though.

They want to re-run against actual job performance in the future, that's nice, but it seems like they're throwing ideas out awfully early, then.

</comment><comment><USER31> while this process feels like it's hitting the sweet spot for finding out who can write brilliant code, that's half or less of the battle in hiring people. personally (and as a hiring manager), i feel like a majority of the hiring process is dependent (obviously) on the environment you're hiring into.

hiring for that small startup? you'll want multi-hat wearing people first, brilliant programmers second.

hiring for a large enterprise team? you'll want to hire for "plays well with others" first, and brilliant programmers second.

that's not to say you should hire schleps, for sure. they should at least be competent programmers. i guess what i'm saying is (despite how it sounds), hiring someone who can program brilliantly is important, but not as important as hiring someone who can navigate your company's software-making requirements successfully.

firing the brilliant engineer who thinks he's more talented than everyone else in the small company so he keeps demanding to be put in charge? yup, that's a thing. firing the brilliant engineer who fights tooth-and-nail over some inconsequential feature the product team wants to change? that's a thing too. assigning a brilliant engineer to crap, meaningless work because no one else on the team wants to work with them? yuppers -- seen it.

in any organization, you are either the only one in charge or you're following someone else's orders -- both of which require different aspects around working well with others.

<comment><USER32> Hear hear, the "no assholes" rule is always good to apply, regardless of programmer brilliance.

</comment></comment><comment><USER33> Neat idea, especially for someone like me who's trying to get out of support and into development.

From Triplebytes' FAQ:

"When do I meet companies?

If we decided to work together after giving you feedback post our technical interviews, we'll start introducing you to the companies and guiding you through their hiring process."

So, just to be clear, first you quiz/screenshare/interview with Triplebyte, and then you still have to go through each company's search process? Or do companies partner with Triplebyte to fast-track candidates who've already been vetted?

</comment><comment><USER34> I interviewed with TripleByte and it was different from a normal job interview in some ways, but fundamentally also the same. There is just no culture fit aspect to the interview. Keep in mind that they are not addressing the problem that coding interviews do not predict on-the-job success. They are basically addressing the needs of a broken system.

</comment><comment><USER35> Correlation between testing and interviewing success tells you nothing about the correlation between either of them and actual work performance.

<comment><USER36> From the article, in bold:

 _The really exciting point comes when we can re-run all this analysis, basing it on actual job performance, rather than interview results_

<comment><USER37> How will they gather data about the job performance of the people who did not pass the interview? Without that, the study is only halfway done.

<comment><USER38> That's true of any hiring system though: how do you gather data on the job performance of the people who don't pass a resume screen? A coffee date? A phone screen? Missing counterfactuals everywhere.

To run a full analysis, you need to hire people randomly - both people who fail and pass your hiring process - and assess their subsequent performance. This never happens in real life.

However, you can still run an informative statistical analysis based on the variability in interview scores and performance scores. For example, the people who scored 5/5 on the interview should perform better than the ones who scored 4/5.

</comment></comment></comment></comment><comment><USER39> It's very nice to see hiring advice based on data rather than anecdotes. But I wonder if the process described in the article is pre-selecting for people who are out of work and desperate, rather than currently employed and casually looking for something better.

From the article: > Our process has four steps:

> 1\. Online technical screen.

> 2\. 15-minute phone call discussing a technical project.

> 3\. 45-minute screen share interview where the candidate writes code.

> 4\. 2-hour screen share where they do a larger coding project.

Then later:

> ...we can't afford to send people we're unsure about to companies

Does every applicant in this system really have to go through four rounds of screening before even talking to someone who works at the actual company? I can't imagine doing that unless I was desperate.

<comment><USER40> >> ...we can't afford to send people we're unsure about to companies

>Does every applicant in this system really have to go through four rounds of screening before even talking to someone who works at the actual company? I can't imagine doing that unless I was desperate.

And from the candidates' viewpoint: That headhunter made me waste 3 hours on screening and didn't even get me a phone interview. Why am I wasting time on this?

I realized another problem. You're trying to predict WHICH PEOPLE WILL GET HIRED BY YOUR CLIENT. That's a completely different outcome than trying to pick the people who are the best workers. If the employer's process is defective, your pre-screening is just reinforcing that bias (albeit improving your "efficiency" as headhunters).

</comment></comment><comment><USER41> Wait, how is this telling us anything? They don't seem to be predicting what determines the best candidate by looking at how he/she performed on the job. All this is doing is predicting how well they will do in a different part of the interview. Am I wrong?

</comment><comment><USER42> I personally hate the "take home test" approach to interviewing. I've had multiple such tests that take anywhere from 10-25 hours to complete because simply answering the question isn't enough; you need to give textbook correct answers and your code must be formatted perfectly with the requisite comments and documentation. In short, it's pretty similar to an upper-level college course's final exam; however, in college, you can get a good grade with a few mistakes; in interviewing, you get rejected for a few mistakes. I'm done giving a company 15 hours of my time just to get to a first interview; this is arrogant, condescending, and completely devalues my time.

The reality of hiring is you're going to make mistakes, like every other part of running a business. Even in an extended "interview" such as dating for a potential life partner, people make mistakes so I'm not sure how the hiring process can be quantified to remove said error. The interview process is so excruciating these days I often hate the companies I'm talking with.

While we're at it, the skills requirements listed with jobs today are astounding. My experience is that a company wants to hire a programmer with at least a journeymen's level of expertise in 6-8 skills. If you have 5 and are comfortable you can learn the other 3, you're dead in the water. Let's be honest, the latest Javascript framework isn't that complicated. The latest NoSQL database isn't that hard to learn.

The truly hard parts of joining a new company are learning how projects are managed, getting the political lay of the land, finding a sherpa to answer your questions in the first couple of weeks, and learning where you fit within the organization.

<comment><USER40> If only there was some way that people could spend a couple of years proving they had basic competence, so they didn't have to prove basic skills on every interview.

Why did I get a CS degree if every interview starts with the assumption that I'm an unqualified loser?

<comment><USER27> Because lots of people with CS degrees don't have basic programming competence?

<comment><USER40> Why are those people getting CS degrees? Shouldn't they get flunked out?

When schools allow students to do coding projects in groups, it's possible to graduate with ZERO ability, provided you can find someone competent to partner with.

<comment><USER27> Right, exactly. That's what I'm saying, I don't know what you're disagreeing with. Having a CS degree does not guarantee programming competence. Some people get CS degrees who shouldn't get CS degrees. Some students probably should get flunked out who don't flunk out. Some students graduate from perfectly fine colleges with good grades but zero ability. It happens, hence fizzbuzz. Some people are just naturally good at marketing themselves, and show up with multiple degrees from top-tier schools, vast lists of publications, with years (if not decades) of industry experience on their resumes in important-sounding positions doing impressive and difficult- sounding tasks, and can't code their way out of a paper bag. Hence fizzbuzz.

<comment><USER40> Why are universities graduating people who can't pass programming 101? Why are they so grossly negligent? They diluted the value of my degree, by graduating too many clueless people.

I did lots of problem sets, homework, and coding projects in school. Why do I have to do it again FOR EACH JOB INTERVIEW? Especially since the projects are usually less on-topic than the ones I did in school.

I.e., I do a homework project for a class, and if I don't get an A, I usually get a reason why not. I do a coding interview project, it goes into a black hole and I never hear from them again.

<comment><USER43> I think it's because CS and software development are different things with some overlap if you Venn diagrammed it. It doesn't necessarily mean the university is negligent or incompetent. It means that what constitutes the content of a good CS education doesn't map to in-the-trenches software development. It also probably means the schools reward and incentivize different things than a software development firm does.

</comment></comment></comment></comment></comment></comment></comment><comment><USER44> The blog post kindly shared here reports on a startup's experiments with offering a new kind of hiring screening service. "We launched Triplebyte one month ago, with the goal of improving the way programmers are hired. . . . Well, a little over a month has now passed. In the last 30 days, we've done 300 interviews. We've started to put our ideas into practice, to see what works and what doesn't, and to iterate on our process." They are currently validating what they are doing on the first steps just against what happens at the later steps in their process: "For now, we're evaluating all of our experiments against our final round interview decisions. This does create some danger of circular reasoning (perhaps we're just carefully describing our own biases)."

I agree with the blog post author that current hiring processes mostly show that "too many companies are content to do what they've always done." And the idea of a standardized, automated quiz of programming knowledge sounds interesting. But what has to happen next is to an actual validation study and find out if programmers hired by this process do better as programmers in actual workplaces than programmers hired by some other process.

Regular readers of HN are aware that I have a FAQ post about this topic of company hiring procedures.[1] Company hiring procedures are the research focus of industrial and organizational psychologists, who have almost a century of research to look back on with long-term, large n studies to provide data on what works and what doesn't work for hiring capable workers. It's a shame that most company human resource departments ignore basically all of that research.

[1] [https://news.ycombinator.com/item?id=4613543](https://news.ycombinator.com/item?id=4613543)

</comment><comment><USER45> "The really exciting point comes when we can re-run all this analysis, basing it on actual job performance, rather than interview results"

I'm not sure how this gets around the circularity arguments though, since you never get to evaluate the job performance of someone you selected out already. Only the tiny fraction of coders that make it past the initial test get evaluated, which could serve to reinforce the potential biases rather than ameliorate them.

The one case in which this would work is if they hired a number of coders that didn't work out well, and could add or update a feature as a negative predictor of job success.

I'm assuming that they're not at the scale of a larger company with thousands of engineers, and that the observations going into a regression model are relatively sparse. If this is a startup with a 20 hires, I'd be surprised if there was much to do to refine the model after a round or two of evaluations, but would be excited to learn otherwise.

<comment><USER10> Lack of insight into false negatives is (I think) the major flaw in most studies of hiring engineers. For the analysis in this blog post, we just passed everyone thorough the screening, to avoid the issue. Going forward, we'll be passing 10% of those who fail the screening, to be able to track our false negative rate. My dream is to reach a scale where can do the same thing with the final hiring decisions (well, less than 10%, clearly). Someone needs to do this.

</comment></comment><comment><USER46> The chart is kind of dumb IMHO. It's nice that they show correlation between different factors and their final hiring decisions, that doesn't indicate the actual quality of the people hired. All it does is reveal an unwritten formula behind their hiring decisions. In other words, the time to complete a programming task is being ignored in their decision process and so should be removed from the process. That's independent of how well it indicates a good candidate. Or am I missing something?

</comment><comment><USER47> Most of the problems of coding interviews can be solved by good reference checking. Code problems in isolation will never expose how a developer will do when faced with real-world issues, users, QA, and teammates.

Why are people so afraid to pick up the phone and talk to references? I'm always happy to give out my references, and always delighted to talk about the good devs I've worked with, with specifics about what they've done.

Standardized tests don't work for schools and don't work for jobs.

<comment><USER48> > Why are people so afraid to pick up the phone and talk to references?

First, references are chosen by the candidate, so they are almost universally expected to say good things.

Second, many companies explicitly prohibit their employees from giving meaningful references about former employees. Relying on a process that needs people to violate company policies or to have moved on from the role wherein they worked with the candidate is not really a good idea.

<comment><USER49> Not to mention that in the industry I am in people will tell you the exact opposite about people. It makes sense, you want your worst people to work for competitors, and your best people to not.

</comment></comment></comment><comment><USER50> I found the article very interesting, but it seems to me that the metrics are the wrong ones. You cannot treat hire and no hire outcomes equally. Let's not forget that the goal of the interview process is to actually hire people. A successful hire is worth millions, a correct no-hire decision is just avoiding further losses of time and money. On the other hand, in terms of efficiency, a candidate that wasn't rejected early is the most expensive error. Well, not as expensive as actually hiring the wrong person, but it usually takes months to figure that out. Whereas the fact that a whole man-day was spent interviewing the wrong person on site becomes obvious in a few hours.

Here are the important metrics in my opinion, in order of decreasing importance:

\- How many people were hired? Or what percentage of positions were filled? How long does it take to fill a position? Nothing in the article mentioned how many people were actually hired.

\- False positives (people making it to the most expensive stage of the interview, typically a day-long on site interview, and being rejected there). What percentage of people that went to on site interviews got offers? Personally, I have always advocated processes that eliminate as many false positives as possible, even if it comes at the cost of some false negatives. Of course, you have to be careful not to filter out people too aggressively, because then you're just not going to hire anyone.

\- False negatives (incorrectly rejecting good candidates early). By definition that's impossible to measure exactly. However, if you are not hiring fast enough, then maybe you have a problem with your screening process. At this point you could do an experiment and relax the screening process for half of the candidates and see what happens. But it could be just a sourcing problem, that is, you are not getting good candidates in the pipeline to begin with. It's very hard to tell whether you are being too harsh and not believing enough in people's abilities (or not willing to develop talent), or you are just not attractive to the kind of people that you want to hire.

Of course, all of the above is from the employer's point of view. If you are also trying to provide job seekers with a good service, then you can devise other metrics for success and for efficiency from their perspective.

(edited for formatting)

</comment><comment><USER51> Maybe I glanced over it but who did the candidates interview for? Were those just simulated interviews or are they working with companies that actually hire people? Or are they hiring for themselves? I'm very confused by what "success" means. The last paragraph indicates that it wasn't actually used for a hiring process? I see a commenter on their site has the exact same reaction.

Either way I applaud every afford to improve the hiring process. However I'm a tad bit skeptical. They should release the dataset (unless I missed it) because it's pretty convenient that the results seem to indicate that hiring can be improved by a quiz which they could build and sell.

I'd be interested in the following screening filter: Have a programmer at the company read through the projects the candidates supplied (as a replacement to "read CV") and then come to a conclusion of yes/no. No projects = no job offer by default. You can always think of a different approach for people with no projects if you feel like you should hire from that group. Possibly have multiple programmers read the code and discuss it.

</comment><comment><USER52> Most candidates can speak well about interesting projects they'd worked on in the past, but a significant percentage of those can't pass a coding test.

Also, not being able to speak well about a past project is highly correlated with doing well on a coding test.

Sounds like one or the other should be thrown out. (Or maybe only the small percentage who do well on both will go on to do well on the job?)

<comment><USER53> One or other should be thrown out. Or both thrown out. Or neither thrown out.

Maybe only the small percentage who do well on both will go on to do well on the job. Or maybe large percentage will do well on the job. Or maybe a percentage who do bad on both will do well on the job.

In short: there is no data at all about doing well on the job, so we don't know what to throw out and whom to chose.

</comment></comment><comment><USER54> Where I work, we ask candidates to do a take-home task after they've passed the first in-person interview. The take-home task is trivial, something most developers with a few years experience should be able to knock out in less than an hour. All components of it are easily googleable.

The advantage is that we're building a corpus of solutions to the same problem that we can compare against each other, which is interesting. More importantly, we're building a corpus of solutions that we can then pick from to have the candidate analyze in-person, and talk us through what they see, what they'd do differently, what they like/don't like, etc.

In short, we familiarize them with the problem via their own answer, and then ask them to analyze someone else's (anonymized) answer. Our sample set so far is too small to draw definitive conclusions from, but it feels better than our old ways of doing things.

</comment><comment><USER55> This is a noble project but I'm concerned that it may not be a methodologically valid blind study.

Specifically, in order to be a blind study of the relationship between the screening exam and technical interview performance, the technical interviewers should not know the results of the screening exam before they make their decision. While they do not state this clearly, it seems possible that since the same 2 people were conducting all steps themselves, that they were not properly blinded.

Thus we cannot rule out confirmation bias in the interviewers themselves, i.e. that they were impressed by good performance on the programming quiz, not that it was an independent predictor of good performance in the technical interview.

Now, maybe one person did the screening and the other did the technical interview with no information sharing in every case, but this would need to be clarified.

</comment><comment><USER56> Many fields require serious professional certification. You can't become a doctor unless you go through a board certification that includes simulated patient interaction. Likewise, you can't build a bridge or a dam without engineering certification.

IMO, this article demonstrates the need to certify software engineers; using a process similar to the interviewing process described. Therefore, when hiring, we can skip most of the "do you know how to code" and get down to cultural fit and mutual interest.

<comment><USER57> Professional licencing processes establish minimum standards of competence, training and education. They do not guarantee excellence and a license represents that its holder is no worse than the worst legally allowable practitioner. Half of all surgeons are below average. You don't want one of them if you have a choice.

<comment><USER58> Half are below the average of practicing surgeons. But the average surgeon is much better because you weed out a lot of incompetent people who can't get certified.

<comment><USER57> Surgeons are not number 10 heating oil commodities. The difference in quality between one surgeon and the next matters when it comes to an operation's outcome. Likewise - and this is the gist of my comment - licensing software engineers might raise the minimum bar, but it won't make hiring any easier because licensing doesn't establish relevant expertise or pertinent experience. Obstetricians not thoracic surgeons are what you want for Cesareans just as you don't want a J programmer writing your CMS.

<comment><USER58> What do you mean licensing doesn't establish experience or expertise? That depends entirely on what you use to qualify people for licenses.

Naturally, you won't hire a heart surgeon to do a brain transplant, but then you don't have a generic "surgeon" license, you have separate "brain surgeon" or "heart surgeon" licenses. You get a Cesarean from someone licensed to do it, and that won't be a thoracic surgeon.

And it _would_ make hiring easier, because you have a much smaller pool of candidates to look at. The problem with licensing is that you end up having to pay for a license, and technology changes to fast to keep licensing relevant. And so much of tech education is expected to be done without guidance or experience.

<comment><USER57> Again the license means a person is minimally qualified. It doesn't mean they are good at whatever they are licensed to do. As a licensed architect in real life, I am minimally qualified under the law to design a State Penitentiary. I lack the experience and expertise [and inclination] to do it well in accord with contemporary standards...i.e I'm not qualified to do that type of project beyond a general legal minimum for practicing architecture.

So when there's a new penitentiary in the works, there's an RFQ for design services.

</comment></comment></comment></comment></comment></comment><comment><USER59> This is interesting, but are they using "performance on a sample coding task" as the truth? If we knew that was a good proxy for being a good hire, we'd already be done.

</comment><comment><USER60> I've had mixed reaction to this take home exercise. I've done that some 4 times and 3 of them resulted in getting called in for an onsite (Amazon). If problem is interesting and there's something to learn. I tend to go for it. But couple of them were lame. Neither it was a good use of my time, nor it was an indicative of my skills.

In near future, I'll be requesting exercises that can be treated and used as a micro library. They are free to use it and I get to post it on github.

</comment><comment><USER53> > The really exciting point comes when we can re-run all this analysis, basing > it on actual job performance, rather than interview results.

Yes, this is when you have something real.

</comment><comment><USER61> "suddenly, a significant percentage of the people who had spoken well about impressive-sounding projects failed, in some cases spectacularly, when given relatively simple programming tasks."

Unsurprising. Turns out talk really is cheap and doesn't indicate one can do. I've even seen people able to maintain jobs over a period of years by talk alone without ever really doing much. Or even being able to do much.

</comment><comment><USER62> >Suddenly, a significant percentage of the people who had spoken well about impressive-sounding projects failed, in some cases spectacularly, when given relatively simple programming tasks.

I feel like this should be listed among the least surprising things in the world. Being a good programmer is about MAKING that hard project look easy, by approaching it in the correct way!

</comment><comment><USER63> There are different kinds of jobs and different kinds of work environments. Grouping all of the situations together and trying to find just a single way of interviewing people flies in face of diversity in human nature. While this research is great and startups in this field are doing well, I see this huge gap that no one so far is trying to address.

</comment><comment><USER64> I'd really like to see how references correlate with the other factors. Did they not call references?

</comment><comment><USER65> I should apologize to Triplebyte in the name of those of us, including me, who immediately created fake profiles and took the test just to see if we would do well against this new rubric.

</comment><comment><USER66> OPEN LETTER </comment><comment></comment><comment>====

Hi, I had a reading of the conclusions you made and I felt as if you defined a process of hiring machines to code rather than humans. So I took a few moments to read your manifesto([https://triplebyte.com/manifesto](https://triplebyte.com/manifesto)) (the premise on which your entire conclusion is made), and here is my take on it.

1\. /"Whiteboard coding and algorithm questions aren't good predictors of how effective someone will be at writing real code." Whiteboard coding show how someone really thinks. It illustrates the though process of the person and that helps the interviewer to judge him on his rational thinking and his logical approaches. Algorithms add to this by illustrating the problem solving ability. A person may not be able to solve an Algorithm actually, but the attempt on a whiteboards speaks more than his implementation on a online platform.

2./ "Candidates deserve a consistent experience and consistent evaluation". The entire USP of an interview is the diversity which allows the interviewer to judge if someone is able to adapt to new situations and come out of his comfort zone. What you are suggesting is to change the interview process into a GRE exam which will only in-turn develop the culture among developers to prepare for that exam for 2 years.

3./"Hiring decisions should be made using a clear scoring system, not gut feelings" Most of the companies have a 3round or 4 round interview process. It is obvious enough to remove the gut feeling factor. If you wanna argue that it may be possible that a candidate got selected based on the gut feeling of all 4 interviewers then my counter argument is that he is worth being selected if he could generate that gut feeling in so many people.

4./"The hiring process should be focused on discovering strengths, not uncovering weaknesses" Agree to the point. However, the irony is that you are trying to define a particular process to hiring. I wonder if it could actually perform the "discovery" part.

5./"Candidates should be told exactly what to expect in an interview, and be allowed to prepare in advance. " So basically you want to hire the guy who has studied the most over the smartest guy in the room. From my experience, I can surely say if companies like "Google" and "Fb" used to follow that practice, I wouldn't be even writing their name here.

6./"truthful feedback on how they did so they know how they can improve" Agreed. Something that should be adapted by all companies in their recruiting process.

7./"Good programmers come from all types of background." You enforce my point with this statement. Good programmers need not be just people who could quickly write a program for a search in a large of data using hash maps but can also be people who have brilliant problem solving ability and are slow in transforming that into code, or people who are amazing in thinking software design and scalability and probably cannot remember code syntax so well. A company needs a good blend of all these people. Then only a good ecosystem to growth is created rather than just making a team of 10 machines who could transform a pseudo code into Java in 10 minutes.

8./" The software industry needs to experiment more with hiring processes and figure out what really works." I think many are already doing that by doing Tech Hackathons, online challenges, weekend projects, open source contribution references etc. So, not something new which you guys figured out.

9./"Candidates are at a fundamental disadvantage in salary and equity negotiations" Not sure what kind of companies you have surveyed. I think most well known companies maintain clear standards of salary and compensation plan. Though people will surely be flattered reading this. :)

10./"Companies should not have to make recruiting a core competency" Now you are just trying to open the market for yourself by yourself. No comments. :P

Would love to hear your counter arguments. Mail me. :-)

</comment><comment><USER67> Great. So where can I do this quiz and these small programming tests? :)

</comment><comment><USER68> Gosh I hope they are controlling for time-of-day, day-of-week, and weather considering how closely packed together those 300 interviews were.

</comment><comment><USER47> Without any demographic data about who the interviewees were, this is basically useless. Are they all under 30, white and Asian men? Where?

<comment><USER69> I can see that that would be interesting information to have, but why is it absolutely necessary?

<comment><USER70> Fresh graduates are often better at programming / algorithm interviews than older experienced programmers who have been out of the interviewing game for a while. It's actually pretty important to know which demographic of users this data applies to.

</comment></comment></comment><comment><USER71> Another approach: [http://ortask.com/a-better-way-to-hire-developers-and- tester...](http://ortask.com/a-better-way-to-hire-developers-and-testers/)</comment>


Programming as a Foreign Language - <AUTHOR> http://nuts.redsquirrel.com/post/23200685522/programming-as-a-foreign-language

<comment><USER2> I wish more developers treated learning to program like learning a foreign language. When you're learning a foreign language, your #1 goal is to cast off any poor grammar or pronunciation and sound like an intelligent native speaker. It's been my experience that people often don't care very much about doing this with the programming languages they use.

Routines that are too long and ought to be refactored into smaller ones are akin to run-on sentences. Leaving out semicolons in your JavaScript is like speaking French in an Italian accent because you already know Italian and French people seem to more or less understand it. And people who refuse to follow prevailing code style guidelines? Those are your "Everyone there speaks English anyway" folks.

Such poor communication skills would be humiliating when trying to communicate in a foreign language, but for some reason people often wear them like a badge of honour when trying to communicate in a programming language.

<comment><USER3> "I wish more developers treated learning to program like learning a foreign language. When you're learning a foreign language, your #1 goal is to cast off any poor grammar or pronunciation and sound like an intelligent native speaker."

I'm not sure that's true at all. When I learn a foreign language, my aim is to be able to communicate. The extra effort required to be perfect far outweighs what I need from it. The native speakers I talk to are quite capable of taking my bad grammar and understand my meaning. Wanting to sound like an intelligent native speaker isn't just at the bottom of my list of needs; it's not even on the list.

When learning a programming language, I have no choice; I must have perfect grammar.

"Such poor communication skills would be humiliating when trying to communicate in a foreign language"

Not at all, in my experience. Simply having some simple grasp of the local language is met with delight; just having made the effort goes a long way, even if the locals themselves prefer to speak with me in English.

</comment></comment><comment><USER4> Makes me think about programming IN a foreign language. One time we had to fix some code a Turkish client had added to our system. The spec and the code were in Turkish (imagine Turkish COBOL if you will). We somehow got it done by trial and error (Is THIS what you want? No? How about THIS?) I have no idea whether this means anything, but we came away saying "Berklat bokari" to each other at random moments on future assignments. Those words or something like them kept appearing in the Turkish code.

<comment><USER5> For a lot of the world population, english, as common as it may be in programming, is still a foreign language. People less fluent in English are that much less likely to try scripting something. Not only would the language keywords make little to no sense, but the supporting materials and comments as well.

<comment><USER6> Thanks to the British Empire and partly to the United States, English is effectively the international language of Business, though.

</comment></comment></comment><comment><USER7> My girlfriend is seriously interested in linguistics. When she discovered I learned to code when I was a kid, and given the speed at which I learn (and forget!) foreign languages during my journeys in Europe, she ended up with the idea that learning a programming language is a similar procedure to that of a speaking language.

She also thinks that my ability to learn languages has been maintained strong by the fact I learnt a new __programming __language every year.

Also, the fact that bilingual kids are able to retain for a longer time the ability to learn languages made her think that as a kid I could be considered as bilingual.

I had the chance to confirm this looking at a colleague of mine who started to code as a kid too. Looks like we have some common abilities in language learning.

</comment><comment><USER8> I had to work on some dutch COBOL once, was actualy realy easy if you take the standard approach and that is to ignore ALL comments as the code is the only documentation you can truely trust.

But the only truely universal language for programming has to be bainfuck :).

</comment><comment><USER9> It's funny, but I've actually used the opposite idea: approaching the grammar of foreign languages like a programming language. Most programming languages are just historic collections of arbitrary rules, most of which make some amount of sense. Most natural languages are the same thing on a much grander scale. Thinking about grammar like this helped me learn it properly.

</comment><comment><USER10> There are some interesting research efforts that explore the link between early-age bilingual learners and positive cognitive effects [1].

What is striking to me is that hackers have long made statements that learning different programming language paradigms makes us better hackers. While we can make anecdotal arguments that being a good functional and a good imperative programmer makes us better hackers in general, the above research is exploring the idea that multi-language proficiency has positive cognitive effects that _cross domains._

[1] Carlson S. M. & Meltzoff A. N. (2008). Bilingual experience and executive functioning in young children. Developmental Science, 11, 282–298. Availabe at [http://ilabs.washington.edu/meltzoff/pdf/08Carlson_Meltzoff_...](http://ilabs.washington.edu/meltzoff/pdf/08Carlson_Meltzoff_Bilingualism.pdf)

</comment><comment><USER11> Interesting comparison, but I find it hard to relate to at all simply due to the sheer amount of words you need to learn in a real foreign language to achieve fluency. Seems much more daunting to me...but I guess my glasses are tinted.

<comment><USER12> Try <http://memrise.com/> for vocab learning - it's really helped expand my French vocabulary quite lot and seems to make it stick well. Doesn't really help with spoken fluency though, for that you just need lots of practice!

<comment><USER13> Thank you so much for bringing this site to my attention. It seems to have become a nice replacement for smart.fm which I used a few years ago.</comment></comment></comment>


Facebook Tries Letting You Pay To Guarantee Message Delivery - <AUTHOR> http://techcrunch.com/2012/12/20/facebook-tries-letting-you-pay-to-guarantee-message-delivery-changes-messaging-privacy-settings/

<comment><USER2> So... I know some victims of domestic abuse, for example. Whose abusers would gladly pay to ensure that their message gets through to their erstwhile (and now, once again?) victims.

FB, do you really not have a clue, or do you just not care?</comment>


Eleven nations – but not U.S. – to sign Trans-Pacific trade deal - <AUTHOR> https://www.msn.com/en-us/news/world/eleven-nations-%E2%80%94-but-not-us-%E2%80%94-to-sign-trans-pacific-trade-deal/ar-BBK1fHn?OCID=ansmsnnews11 <comment><USER2> It made a big difference not having the US in the deal. All of a sudden a lot of the conditions that were in the deal and were causing a lot of heated debate got dropped. In NZ our Ministry of Foreign affairs has given a difference between the new CPTTP deal vs the TPP deal

[https://www.mfat.govt.nz/en/trade/free-trade- agreements/free...](https://www.mfat.govt.nz/en/trade/free-trade- agreements/free-trade-agreements-concluded-but-not-in-force/cptpp/tpp-and- cptpp-the-differences-explained/)</comment>


Is Your Face Real? - <AUTHOR> http://www.petermichaud.com/essays/is-your-face-real/ <comment><USER2> I didn't think this was particularly relevant to HN, which is why I didn't submit it, but if you're looking for an angle, this article is extremely relevant to those seeking employment. Think about what kind of first impression you make, in the moment before rational thought.

<comment><AUTHOR> I posted it because there have been comments here on HN about your eye patch. To me that made your post relevant.

<comment><USER2> Yeah, you're right. Part of the reason I wrote this is that I've been getting a lot of those questions lately and it seemed convenient to have a link ready at hand 8)</comment></comment></comment>


Why We Do Code Review - <AUTHOR> https://www.culturefoundry.com/cultivate/technology/why-we-do-code-review/ <comment><USER2> > The reviewer should download the branch to their local environment, follow > the deploy instructions (if present) and make sure the code "does what it > says on the tin"

This looks like a very onerous requirement for reviewers (although tools like gitlab or getdockup.com automate it), I wonder if a) is really useful in catching issues, b) the process they used to convince the team to spend this extra time when reviewing. In my experience is that devs don't really want to spend more than a few minutes skimming through a PR.</comment>


Singing/Dancing Stuffed Animals-- Stripped to Reveal the Robots Beneath - <AUTHOR> http://s3.amazonaws.com/s3.mattkirkland.com/ursum.html <comment><USER2> The fascinating thing for me -- that reminded me of several projects I've worked on -- is that many of the detailed parts of the exteriors have no corresponding parts on the inside. It's also amazing to see how much 'dressing up' simple machineries need, in order to be attractive. Taken in the opposite light, however, this also shows that to build something good, you often don't need very sophisticated mechanisms inside.</comment>


Ask HN: Validate my startup idea? - <AUTHOR>

Imagine there was a microphone in your house that could detect events using complex machine learning techniques and can be applied to home automation?<p>e.g. if it hears the jingle of your keys (and not your flatmates) after 6pm and it hasn't heard you turn the shower on in the morning, it turns on a warm bath for you. <comment><USER2> Sounds interesting, but the use cases get me excited. It sounds like you are brewing an interesting platform/solution for an unknown problem. My advice is to stop right now, find a clear problem people have that they are willing to pay to solve.

All-encompassing platform plays are hard to do without a solid use case to build from. However niche, you need to absolutely smash someone's problem with this platform, and once you find success there, you can go horizontal.

</comment><comment><USER3> What a lot of other posts are getting at is that this sounds like a cool challenge, but that it doesn't exactly solve a pain point for anyone. No one is thinking, Man do I need a robot to pour a bath for me when I come home, but only if I haven't showered that morning!

There is nothing wrong with solutions looking for problems: there are a zillion of them on store shelves right now. But this means that 1) you'll have to overcome an education barrier to convince people why they NEED your product, and 2) you'll be able to get much less insight about whether anyone will buy your idea.

These sorts of ideas could be bad for a new startup. It's easier to build and sell something like this if you have a larger R&D budget, a large marketing budget, or a large reach so that you can get it in front of potential buyers.

</comment><comment><USER4> You're asking us to find out about product-market-fit to alleviate your product-market-fit risk, but the real risk with ideas like this is invention risk. Google still only gets about 7/10 of my voice translations right when I directly speak to my phone.

Similarly, you don't have to ask Hacker News if a pill that cures diabetes would be popular, the risk there isn't product-market-fit it is invention- risk.

</comment><comment><USER5> I think you're on the wrong forum. Try this one: [http://highdeas.com/](http://highdeas.com/)

</comment><comment><USER6> It might just be a matter of picking out a better use case.

How about if it listens for someone screaming for help and calls an ambulance? Assuming you can limit false positives, you could probably sell it for use with the elderly and the infirm.

</comment><comment><AUTHOR> the aim would be to build an api for event detection that broadcasts events that can be hooked up to actions.

<comment><USER2> Whose problem does that solve?

</comment></comment><comment><USER7> Plausibly valuable, or even inevitable someday. But would require much greater adoption of home-automation, and a few more-compelling must-have benefits, to seem a good business.

Considering your example, are there any automatable bathtubs yet? Or is making such a thing part of your idea? (The cost of retrofitting such installed plumbing, and the electrocution liability, suggest to me it'd be a long, slow quest.)</comment>


Strange Loop 2011 conference schedule published - <AUTHOR> https://thestrangeloop.com/schedule

<comment><USER2> Really pushing hard to try and get to this great conference.</comment>


Drupal tm policy: now requires you to get a license just to use the word drupal - <AUTHOR> http://drupal.com/trademark

<comment><USER2> Note that you are _automatically_ granted a license for various always-okay uses, including fair use, use in source code identifiers, and various other pre-approved uses.

There's a general problem for high-profile open source projects with valuable trademarks. Registering a trademark can be a useful measure to defend against lawsuits from other people who claim ownership of a mark, or to prevent abuses like companies misleading users by selling (fake) "official" Drupal support or licenses.

But trademark law is very different from copyright law, and it's a lot harder to make trademark licenses compatible with software freedom. For example, you can let anyone modify and sell your copyrighted work while still retaining your own rights. But if you license your trademarks indiscriminately then you risk invalidating them. (This is because trademark law was _originally_ intended to protect consumers by allowing them to trust that a mark was associated with a particular manufacturer.)

Some projects like Mozilla and Drupal have dealt with this by creating legal licenses that attempt to balance community freedoms with the requirements of trademark law. When I worked on the Audacity project we registered our trademarks (mainly as a defensive measure after a company with a similar product started using the same name) but did not craft any explicit license. Most open source projects don't bother registering their marks at all, which is probably the best approach in most cases.

</comment><comment><USER3> A rather misleading title, IMO. Actually it's completely wrong.

<comment><AUTHOR> yeah reddit sensationalism > me</comment></comment>


Amazon EKS Price Reduction - <AUTHOR> https://aws.amazon.com/blogs/aws/eks-price-reduction/ <comment><USER2> This matters and is long overdue. Google, Microsoft, Digital Ocean, and other providers have managed Kubernetes services and don't charge for the control plane. I honestly expected AWS to drop the charge altogether, but cutting it in half is certainly a move in the right direction.</comment>