Square's Growth Framework for Engineers and Engineering Managers - <AUTHOR> https://developer.squareup.com/blog/squares-growth-framework-for-engineers-and-engineering-managers/ <comment><USER2> _> Two Tracks / Becoming a manager is not a promotion_

To any young engineers who may be reading this, this is social fiction. If someone can fire you but you can't fire them, you better believe you're not on parallel tracks.

EDIT: it's not just firing. Managers know your compensation, but you don't know theirs. They'll all work to split their team into multiple teams so they can say they ran a group; the good ones will soon become managers of managers, and then managers of managers of managers. They'll sit in on senior meetings, work on strategic decisions, and you'll watch them rise further and futher in the company hierarchy while you're enjoying your "parallel track".

<comment><USER3> I think a lot of responses to you are missing the point. Managers have more _autonomy_ and more _control_.

You may get paid the same while working as an individual contributor, at least for a while, but over time your power in the org suffers a slow decay.

For example, you may not be able to direct your work towards that which generates the most revenue, people you passed on in the interview process suddenly appear next to you and you're required to mentor them, projects that seem dumb get assigned to you from on high without your input, compute resources appear or disappear from under your control, etc.

It's like death by a thousand cuts. And in my experience, the only way to actually get paid on this track over the long term is to create something so valuable that you are the single point of failure.

At least for programming, the only real 'Individual Contributor' career track that makes any sense to consider is one where you're a lead developer, that is you still code, but you manage a team of other people as well.

<comment><USER4> An IC can definitely have and maintain more control than managers, they just need to produce visible value to the company.

I've seen multiple ICs with more power than their direct manager. They are basically immune to being fired because higher-level leaders understand their value and respect their track record.

<comment><USER5> Were those higher-level leaders managers? I mean I agree with your _exact_ statement, but in my direct experience, the higher-level leaders tend to be less and less _engineers_.

Unless there is some power of the purse-strings, the two-tracks thing is always going to be biased in favor of the managers.

<comment><USER6> That's the beauty of working in a tech company compared to a non-tech company. Every single manager in my org is a former engineer. Every single manager above them is a former engineer. Heavy majority of directors in the org are former engineers.

<comment><USER7> I feel like that's really gonna depend on what part of the tech sector you're working in.

In the world of early and middle stage startups, you're probably right. However, in massive tech companies or more legacy-type tech companies, it seems like you're a lot more likely to find managers who are trained as managers and not engineers (MBAs, etc.)

<comment><USER6> I agree with you in general. However, my specific experience with this comes from working at one of those massive tech companies. And there are definitely orgs within the company that fall under the "MBA-types-running-the-show" umbrella, but, I think, I just got very lucky with my org.

<comment><USER7> Yeah, the experience probably varies a ton across the industry and even within orgs. I'm sure most people around here who've been in tech for a while can tell stories about both types of company.

</comment></comment></comment></comment></comment></comment></comment></comment><comment><USER8> In my 10 years of engineering, I've seen a whole bunch of such guides. Promotions are hard to make them fair. Promotions are also biased because it's fuzzy and there are humans involved in it.

The only true way to get to know your worth is to test the market. It's a supply and demand game. Most raises I've got is by switching companies. Most companies usually overpay their newer hires and underpay their older hires. Why should they pay more if they're not asking for it ?

Managers will always get paid more as you move up. They make bigger decisions and have bigger impact. Their impact is their management tree's summation.

As an IC you plateau out. You either start your own thing or work in more lucrative industries.

Promotions never land on your feet, you gotta fight for them.

<comment><USER9> This is so true, and often overlooked by a lot of people. There's no new incentive for companies to reward older employees (you're at the company already). But to get new employees to sign their offer, they're prepared to stretch their resources. Additionally the market has likely advanced and salary bands are probably higher. By staying at a company over a long period of time, you lose out on being compensated at the periphery / edges of the market. And you probably lose out on your read of the market too.

<comment><USER10> > There's no new incentive for companies to reward older employees

Is this just plainly not true. But the reason you may be led to believe that is simply a matter of inefficiencies. If a company loses a senior team member, they'll need to replace them with one of equivalent competency. Putting aside the fact that recruiting and on boarding costs quite a lot of money, there's a few basic economic reasons why they may have lost that team member to begin with.

1\. They hadn't adequately assessed the competency of that team member, and that person was able to find a better paying job elsewhere.

2\. The company might not hire people for that position very often. If they don't know how much money it takes to secure a candidate of that caliber, then they might not know they're not paying them adequately.

Both of those scenarios involve employers not understanding the market properly, and if that results in them losing skilled staff, then that is very costly for them. So there's a very clear incentive for companies to invest in retaining staff. However the truth is that the world is full of poorly run companies, this is a problem that's actually quite hard to get right, and there's a very slow feedback loop when it comes to measuring how good you're doing at it. For those reasons, that's why it's often better for people to seek new opportunities rather than seeking raises or promotions, because that's the best way to make sure you're getting a fair market rate, and will allow you to bypass any inefficiencies in the way your current employer may be doing things.

There's also another possibility, where the people on the payroll are simply developing their skills faster than the company can provider more advanced opportunities for them. In this case the employer doesn't need to replace you with somebody of equal competency, they just need to find a replacement for a role that you outgrew.

</comment></comment></comment><comment><USER11> A lot of what's in here reminds me of what we do at PlanGrid, and as someone who has been around to see our system evolve (and add my own feedback to it), I generally agree with the principles that it lays out.

There is one thing that bothers me here though.

> Promotions don't unlock new responsibilities; the new responsibilities and > increased scope come first and then we recognize it with a promotion.

To some extent I agree with this. The part where I don't agree with this principle is that it's essentially saying "do the job of position XYZ, and after you prove to us that you can do it, we will recognize and pay you for it."

If people are taking on additional responsibility and having a bigger impact, pay them for it. Maybe Square does, but normally comp+promotion go together and so I'd be interested to hear evidence to the contrary. If someone has been operating at a level for 6 months, then why not retropay them for 6 months at their new salary? :)

<comment><USER12> That's unfortunate. As a small business owner, I can understand the business side of things — it can be hard to maintain profitability when someone gets promoted if they aren't yet able to do all the skills of the job but get paid for it. I wonder if that was part of the strategy — not all the responsibility was added at once to transition?

We handle this a different way - when someone is promoted, they are given a chart of new responsibilities, expectation timeline and a small raise. Once they completely train into the role, which can be anywhere from 3 weeks to 3 months, we give a larger raise. Again this is because we cannot actually afford to pay for work not being done. It seems people like the acknowledgement, clear expectations and reward when they've made the cut.

<comment><USER13> What's great about this is you're hedging your bets (prudent) while acknowledging the worker's needs as well. The problem without some kind of middle ground like this is that it reeeeeally feels like getting screwed when you're working above your pay grade for an extended period of time without explicit acknowledgement. I've had a lot of unhappy friends in roles like that where they were doing team lead / manager stuff and getting jerked around by HR and their boss for months or years. Now they're hypersensitive to anything that smells remotely like that experience. I like how you're making a public commitment but making sure to manage your risk. It's a great compromise.

<comment><USER12> I hope your friends have had better outcomes since. A good thing to protect yourself as a worker is to get a timeline or plan of action or growth as soon as new responsibilities creep in — if the company had the intention to promote, they will appreciate a clear path, of not, they will push back.

</comment></comment></comment></comment><comment><USER14> This exemplifies why I quit my last job.

But primarily,

"We promote engineers and managers when they have demonstrated that they are consistently performing at the next level. Promotions don't unlock new responsibilities; the new responsibilities and increased scope come first and then we recognize it with a promotion."

Every discussion I had with my manager, my promotion was six months away, despite IC contributions exceeding the sum total of the rest of my team when measured per-project via our number one KPI: annualized cost savings (larger org was seen as a cost center).

Unfortunately these "principles" are becoming harder and harder to avoid.

<comment><USER15> Another way to say that is it's lame to give a someone a promotion to a role they are already doing, because it's the easiest way to communicate that they've been underpaid for way too long.

<comment><USER16> This is only true if the only way to get compensation increases is via a promotion. Granted at many companies this is true, but it probably isn't true at Square.

If you get promoted to Senior level, the assumption is that you're getting paid what a senior should be. If then, you take on additional responsibility above and beyond what a Senior is doing, you get merit increases up to some higher point. That is, a Senior who is getting 5/5 performance ratings probably makes more than a Senior with a 3/5\. Eventually, that 5/5 person might get promoted (I say might because an exceptional Senior might not make a great Staff engineer. Someone really good at implementing things quickly, but with terrible design sense won't make it to Senior in a lot of orgs, for example, even though they might be a very strongly performing Engineer) to the next level, at which point the process starts again.

</comment></comment></comment><comment><USER17> Looks like it's easier to resign from Square and accept a salary bump and title bump elsewhere than navigate Square's "Growth Framework".

If we wanted this much structure, we would have worked for the government or military.

<comment><USER18> Exactly. When a company hires me, they understand that they're going to have to pay me while I get up to speed and master all the responsibilities of the job. Why should promotions be any different? Are they back-paying these people after they take on extra responsibilities for 6 months?

These seems like a somewhat immoral way to squeeze extra productivity out of your workers without paying more. "Just keep piling extra work onto your plate and in six months we'll talk about the possibility of a promotion! It's gonna be great!"

<comment><USER19> How is this different from, say, Google, except that Square is open and honest about it? After I got my offer from Google last year (that I rejected), my prospective manager told me that in order to get promoted (from L4 to L5 in this case) I would need to be operating at that next level for some period of time before my promo packet was even submitted, let alone the promotion come through.

<comment><USER20> As someone who worked at Square ~3 years ago and now works at Google, the rate-limiting and "promotion by apology" (their term, not mine) feel fairly similar - Google's is a bit more impersonal. However Google pays much, much better. Square (I felt) offered more opportunities to build a good engineering reputation and to gain actualization in other ways, and to make a significant impact.

<comment><USER19> > However Google pays much, much better.

Eh, at least in my case the offer they extended me was shit (speaking from my privileged and fortunate position). Combined with that promotion process it was easy to reject (and a month later I got an offer elsewhere for one level and $75k/year more).

<comment><USER20> So my data may very well be out of date. Google's salaries are indeed somewhat low, but their total comp was way higher than what Square targeted at the time. Since Square is now public and doing so well I'm not surprised to hear they're paying better than they did.

</comment></comment></comment></comment></comment></comment><comment><USER21> This is basically a page to help with recruiting. A lot of companies do this, long detailed posts about career ladders. Then sometimes a short bullet point list about "good work-life balance", "competitive compensation", basically the things that really matter, but no real details.

As a senior engineer with a good job who has been in this industry for a while, here's what really matters for me when I look at new jobs:

\- Does your company allow WFH schedules to everyone? There's debate about full remote vs full onsite, I'm not talking about that. I'm just talking about working within driving distance, but having a 3-2 or 4-1 weekly schedule of onsite vs WFH. There's really no excuse to not offer this at this point in communication technology. Some excuse like "more collaboration" doesn't apply if everyone is still going to the office regularly. And this has to be allowed for everyone, not just special exceptions for a few. It must be a culture where everyone is WFH regularly.

\- Will I have to write code on a whiteboard? If yes, then goodbye. Even a chromebook with a html page that has nothing but a <textarea> element is better than a whiteboard for writing code. Will I have to study leetcode for weeks to pass your interviews? Then also goodbye, I don't have time for that and it's a stupid way to do interviews. A fizz buzz type question to just see if someone can write any code, that's ok. Leetcode stuff is not ok. Take-home test? It better be really short or also goodbye.

\- Compensation, it's actually ok if you're not at FANG level, few companies can be. The other things like offering regular WFH schedules are very valuable and this is how non-FANG companies should compete. But it needs to be still a reasonable amount of compensation. Guidelines are really difficult here due to so many factors such as cost of living. But I know a bad offer when I see one so don't try to trick me.

\- Well defined on-call process that doesn't burn out engineers.

\- Minimum of 4 weeks vacation. If it's less to start, you better have a schedule that increases that quickly (not some 1 week for every 5 years bullshit). And if you're doing no vacation (AKA "unlimited vacation"), you better have something really convincing that shows people do take long vacations in your office. Like an official policy to encourage this.

I don't care about free lunch, video game consoles in the office, all that silicon valley "perks". Tell me about the stuff that really matters.

<comment><USER22> When recruiters mention unlimited vacation, I translate that in my head to ~3 weeks, which is less than what I currently get, so definitely not a selling point. I think we'll see unlimited disappear over time because everybody knows it's BS, and we'll see lawsuits from employees who were reprimanded for taking 10 weeks or whatever.

<comment><USER23> I don't know. I agree that everyone thinks, or ought to think, it's BS. But everyone thinks, or ought to think, that open offices are BS and companies are all going for those because it saves on costs. "Unlimited" vacation is the same: in practice, I think people are going to take less vacation because there are no clear rules about what's appropriate, and "unlimited" vacation isn't accrued so it's not a payable benefit like set vacation hours. That second one is a big one. If I get fired or quit or roll over more vacation hours than my policy allows, CA requires that those hours be paid out. There's no such cost associated with "unlimited" hours.

<comment><USER22> But isn't CA eventually going to go after companies who are obviously just trying to circumvent payouts when it becomes clear that they block employees from taking off as much time as they want?

There are bound to be employees who want 3 months vacation and will get fired for taking it.

<comment><USER23> Maybe, but given that there's no requirement that companies offer any vacation time, it seems unlikely to me (a non-lawyer) that a case like that would succeed. "Unlimited" is probably couched in "subject to your team and manager's requirements" language in employment contracts.

</comment></comment></comment></comment></comment><comment><USER24> > We promote engineers and managers when they have demonstrated that they are > consistently performing at the next level.

This while very common has always seemed... a little off to me, and also seems to encourage people not sticking around if they feel like they could do well at the next level, but maybe have some inconsistencies or areas of learning to work on. This leads to an awkward situation especially as thinking starts to become more "meta" as you go to a higher level. At what point is it okay to start touching code less and doing more design and organization-level work, when your current level is more about individual production and output? It can make a top performer trying to get to the next level look like their output has gone down, because the measures to those outputs look differently at different levels.

Additionally, if someone's making that big impact, you're saying they have to wait a year. If they can prove that impact, they can go somewhere else and get that pay bump /now/. Don't make it easier for someone to leave than to stay, especially if they're on a promotion track.

<comment><USER25> Whenever I explain this model to my friends outside the tech industry they think it sounds insane. The idea that you're not only allowed but expected and essentially required to do work outside your official duties seems arbitrary, almost like a trap.

</comment></comment><comment><USER26> The audience and purpose of this article is unclear.

Is this meant for engineers who don't work at Square, for the purpose of recruiting? Or for engineers who already work at Square, for the purpose of justification, i.e. an internal newsletter?

The headline matches the former, but the content seems bent towards the latter.

For one, the overly-long preamble reads like a page in a corporate handbook, laced with insecurity.

> Our levels don't line up perfectly with those of other companies ... You > likely can't accurately compare level numbers or titles directly between > companies, even with tools that provide mappings between companies.

Similarly, most of the "principles" presented are either common-place:

> Levels build on each other: Each level implicitly includes all criteria and > responsibilities of prior levels.

Or deceptive:

> Promotions don't unlock new responsibilities; the new responsibilities and > increased scope come first and then we recognize it with a promotion.

Or are a lot of nice-sounding words, too incoherent to disagree with.

As someone who's had to suffer through career frameworks championed by company-lifers straight from college, I don't think this article won't quite attract the best and brightest.

<comment><USER27> The level thing reads like some kind of horrible gaslighting, but to be fair it's mostly true across all companies. Being a senior engineer in a company that has QA and/or project managers is a very different role than in one that doesn't, for example, and this is a dimension orthogonal to "level".

The bit on promotions is fairly commonplace and sound, about the only real solution I've seen to the Peter Principle.

This doesn't strike me as a particularly exciting document but it doesn't seem terrible either. I like that there are a variety of ways companies handle this sort of thing, and that companies are being more open about it. It really helps potential employees sort out where they want to be.

<comment><USER26> You're right this document neither terrible nor exciting, but I'd disagree that this document is "open".

As others commented on the falsehood inherent to "Two Tracks / Becoming a manager is not a promotion", a blurred transparency tends to be more deceiving than an obvious opacity. Or rather, being bull-shitted can be worse than being told nothing at all.

</comment></comment></comment><comment><USER9> We built [http://levels.fyi](http://levels.fyi) to help bring some transparency into hierarchies at companies (though mainly from a transitional and compensation perspective). It's really cool to see companies opening up on their specific roles and responsibilities for candidates to see before they enter the track

</comment><comment><USER28> > Promotions don't unlock new responsibilities; the new responsibilities and > increased scope come first and then we recognize it with a promotion.

Most companies follow this model now. You should already be performing at the level to deserve a promotion which sounds reasonable but not very much in practice. People will look for a job elsewhere if they get frustrated.

</comment><comment><USER29> > You likely can't accurately compare level numbers or titles directly between > companies, even with tools that provide mappings between companies.

Companies may claim this but it can't really be true in practice, otherwise it would be impossible to level incoming candidates

<comment><USER30> You should level incoming candidates based on their performance during interviews, their prior projects and portfolios, not what level they were labelled as at a previous company. I don't care if you were a Level 38 at Foo Inc., I'm going to interview you and compare what I think you're capable of to the levels at my company and make a recommendation based on that.

<comment><USER29> Most interview processes I've been a part of had a target level for the candidate, with the possibility of leveling them up or down based on the outcome of that interview.

There are other reasons why leveling can be important - the expectations of the candidate around title and comp, for instance.

This is why sites like levels.fyi exist. They are surely not perfect and I don't know much about how the data is collected or verified, but it's a question that people want answered.

</comment></comment></comment><comment><USER31> As a self taught engineer, very unhappy to see the role of a 'relevant Bachelor's degree' stated in every single level.

If I have performed above expectations in an engineering role for 5 years, do not have a bachelor's degree, and I am judged against it, it is not a company I would be working for.

Most companies acknowledge this and allow the substitution of a 'relevant Bachelor's degree' by experience.

<comment><USER32> It should say "or equivalent working experience" which is what the job descriptions on the careers page say. [https://www.smartrecruiters.com/Square/743999692210768](https://www.smartrecruiters.com/Square/743999692210768)

I work for Square, we just hired a self-taught engineer on our team. There's no concrete degree requirement. :)

</comment></comment><comment><USER33> In general, I believe in this kind of system. I've long been a fan of public engineering ladders (see: Rent the Runway) and create a real career path for non-managers.

But... I see a lot of companies hiring at the senior end of the management track (Directors, VPs, etc.) and not so much on the non-management side (usually tops out a senior engineer).

I think that if companies are serious about non-managerial tracks, they need to hire for such roles more seriously.

</comment><comment><USER34> Ooh boy, great to see this discussion. If anyone's interested, [https://progression.fyi](https://progression.fyi) has a bunch more of these to compare and contrast (I'm adding Square's to it right now...)

As you'll see, lots of companies reinvent the wheel with this stuff. It feels fundamentally misaligned to me while these rubrics are solely owned by each employer – personally, I care about my growth across a career, not just a single job.

If this was measurable and transferrable in the same way as – say – learning a language (e.g, I can take my Go experience somewhere else) it would be far more compelling to me as an employee.

The question is: how many companies are willing to 'share' levels and expectations to facilitate this?

<comment><USER17> I definitely don't want some kind of career progression standard that is shared across companies.

Please work for the government if you crave this much structure.

</comment></comment><comment><USER35> Seems to coincide with the thread a few days back about engineers valuing "growth" when considering jobs:

[https://news.ycombinator.com/item?id=20508465](https://news.ycombinator.com/item?id=20508465)

</comment><comment><USER36> I've been developing a growth framework at my org for the primary reason to ensure that we can have consistent conversations across teams and ultimately, provide a reference point for development teams as they work with their managers on growth.

I think they make a lot of sense.

One question .. has anyone planned out how Staff Developers work within most organizations? I feel the Venn diagram for them between an engineering manager has plenty of overlap, where the people aspect (all things HR, hiring, growth) are on the engineering/people manager.

Can a Staff Developer make decisions? Or are they more of an influencer?

<comment><USER37> It's influence all the way down. You want to keep things pretty abstract and let people come up with the solution on their own. 90% of the time it will be what you want, 5% they ask you for help and it's what you expect and 5% maybe they amaze you..

</comment></comment><comment><USER38> What is the point of sharing it? How is it different from any other company?

I also find Title misleading, Ladder is not a framework - it is a way to retain talent. And I find it to be quite silly (as an eng who held highest level position at famous eng company) way. Working in a title-less org now and it is much better as we can concentrate on the mission and not the the freaking ladder

</comment><comment><USER39> I'm curious how the numbers work out at the higher levels. As I understand it, and I may be wrong so please correct me if you know better, at places such as Google there are separate manager and IC tracks however at top levels there's more managers then ICs. So it's essentially easier to become a top level manager than a top level IC.

<comment><USER40> It also doesn't help that they don't have definitions of what top ICs _do_. Managers are in charge of a definite budget and a definite number of employees.

What does the IC have responsibility and authority over? Do they absolutely get to make certain technical choices? How? Which ones? What happens when the right technical decision has impact on how budget and employees are allocated?

</comment></comment><comment><USER41> The question should be whether the ladder to help you climb, or just to let you climb at the company's pace?

Not sure who the quote is from, but something like "instead of climbing the corporate ladder, why not own the ladder?"

</comment><comment><USER42> The problem with having to perform at the next job level to get a promotion is that you're still expected to perform at your current job level. So going from say staff where you're expected to spend 60% of your time coding and 40% on making a broader impact to principal where you're expected to spend 40% of your time coding and 60% of your time making a broader impact in essence means you're going to have to work 120%. That's great if you don't have a life outside work, otherwise, you're better off hiring into the level you want.

</comment><comment><USER43> This is funny coming from Square, L5 and above is all politics there.

</comment><comment><USER44> this seems directly copied from the model used at Amazon (even down to numbers assigned to levels)

</comment><comment><USER45> I don't get how L5 EM and L5 IC are seen as the same level. Based on the number of responsibility items, L5 EM has like 16 items. L5 IC has 8 items.

L6 IC and L5 EM seem more comparable.

</comment><comment><USER46> This is similiar to a framework my company utilizes for promotions. The key difference is that everyone at the same level makes the pay. This is great for transparency - everyone knows how someone should be performing and how they are being compensated (all the way up throughout the company).

For those interested, the company is Pariveda Solutions.

</comment><comment><USER47> Cool to see this published. At GitLab we have a page that serves a similar purpose on [https://about.gitlab.com/handbook/engineering/career- develop...](https://about.gitlab.com/handbook/engineering/career-development/)

</comment><comment><USER48> The devil is in the details--so much depends on how the levels and their descriptions are operationalized.

Would be interesting to hear more about calibration work, how they are defining output parity across the engineering disciplines/business units/products/teams.

</comment><comment><USER49> I'm surprised there's no guidance for IC9. Are there just not any of those yet? At my last company (300-500 engineers) level guidance only went up to IC7, and there was a single IC8 who was sort of "pioneering" new level direction.

</comment><comment><USER50> Does it matter who your manager is? Does this fit to all types of engineering work? Does the idea of responsibility unlocking promotions after the fact work for assigning levels to new hires? Do people who don't self promote get promoted by their managers? Do managers benefit from promoting their engineers?

Someone else already asked, but who was this written for other than the people who wrote it?

</comment><comment><USER51> You need permission to view the document! what!?

<comment><USER52> Somebody is working on fixing the permissions. Sorry.

<comment><USER53> You can do better. :D

</comment></comment></comment><comment><USER54> The actual career ladder document linked from the post is in a private Google document so we can't read it.

<comment><USER52> Fixed.

<comment><USER55> Still locked down unfortunately.

<comment><USER52> We're looking ... Something about Gsuite permissions isn't working as expected :)

<comment><USER49> Works for me

</comment></comment></comment></comment></comment><comment><USER56> google sheet is locked.

</comment><comment><USER57> This is relatively O/T, but I've always been skeptical of the general concept of growing yourself in top tech companies. Seems like the overall idea is just to grow yourself professionally, do more and work harder for your customers, instead of growing yourself _for yourself_ and your own personal happiness, being a better person, etc. On top of that, some companies expect you to give your all in that regard. "Practice your customer obsession" and that sort of thing so you can be a better worker. I hope in the future we'll still have very successful tech companies that don't do this -- ie. they have a more relaxed atmosphere and don't pride themselves on bending over backwards for every customer. Believe it or not, there is more to life than your career.</comment>


Health care veteran tries to upend the system and bring drug prices down - <AUTHOR> https://www.statnews.com/2020/01/12/venture-capitalist-alexis-borisy-bring-down-drug-prices/ <comment><USER2> It is unclear what differentiates EQRx from other pharma companies. As the article points out:

>The idea of creating a "fast-follower" — a new drug that is much like an existing one — is anything but new.

All I see is the claim to come in with competitive pricing - something else that is anything but new.

>But fast-followers do not compete on price, because lowering price has not historically resulted in selling more units of a drug.

<comment><USER3> It is also weird that he talks about the reason he raised prices at companies in the past was because it was fiduciary to shareholders. Then he starts a new company with VC money. I might be more believing if the company was a B corp.

We need a different approach to drug development that is not this. The big issue with drugs is that there are only legal protections for generics and other countries have far different bargaining power. We need some kind of shared risk and reimbursement model with every country having access to the ability to buy or create generics based on them paying to support the research. This is something that could be easily spearheaded by a Gates foundation or its ilk.

<comment><USER2> What do you mean by there are only legal protections for generics? As I understand it, original developers have the protections, not generics .

</comment></comment></comment><comment><USER4> 'Robert Nelsen, of ARCH Venture Partners, said he invested because "I think it's really cool and I like throwing rocks into the pond."'

Definitely a bold approach that will require lots of cash up front, which explains the $200M round. It will be interesting to see how many VCs let their FOMO get the best of them and fund the first credible copy-cat pitch they hear. After all, there is only one Alexis Borisy out there.

It will also be interesting to see what kind of tech or novel operational methodology is developed to deliver on their promise to lower costs. I'll also be interested to see whether they demonstrate substantive progress by the time they raise their next round.

</comment><comment><USER5> > "I've spent the last 25 years creating breakthrough new medicines," Borisy > said in an interview. "We've ratcheted up the prices on them ever higher, > frankly, because we can," he said. "And the reality is we can create a lot > of these great new medicines, and turn that into a viable business charging > a lot less for them. This is not fantasy world. This is something that can > be done."

Is he saying that he has participated in price increases because he could?

<comment><USER6> yea i suppose so, but then something in his world changed. And he thought maybe we can change this for the better. I would be interested to see what it was. Maybe he can just make more money doing this.

<comment><USER7> It sounds like the idea is to spend less money on research by seeking to make equivalents (but not generics? and not infringing patents) and then sell them for less. And, presumably, encourage pharmacies to fill with theirs instead of the original.

If the development costs really are lower, and the targeted drugs have enough of a market, it seems totally reasonable to reduce end user costs and still make a bunch of money.

<comment><USER2> I'm still skeptical. It seems like this approach would be harder than making a biosimilar drug, which still takes >$100 million and 5-8 years.

Pharmacies are not allowed to fill will biologic generics without FDA bio- interchangeable approval, which no biologic in the US has obtained. They would not be able to substitute an alternative drug either.

</comment></comment></comment></comment><comment><USER8> >Gleevec, the Novartis cancer drug, was approved in 2001 and turned a form of leukemia into a chronic disease. It initially cost $24,000 a year, but the list price reached $100,000 a year by the time it went generic in 2016

Novartis increased the price by about $5k a year. That's insane, I don't think this should be legal. How come there are no laws that keep the prices of medication under control?

<comment><USER9> In the best functioning capitalist systems it's the role of government to prevent monopolies. This is why I enjoy living in Europe, where we have reasonable costs for things like medicine, mobile plans, internet costs, insurance and the like. It sounds like the US has devolved to a crony capitalist system and it's not working for the benefit of the citizens.

<comment><USER10> The common trope is that European price controls hinder innovation in pharmaceuticals [1]. I've read that American pharmaceutical firms spend 4x in R&D per capita than European counterparts.

Counterpoints?

[1] [https://www.nature.com/articles/nrd2293](https://www.nature.com/articles/nrd2293)

<comment><USER11> The 4x spend on research per capita seems to be true. I looked into that before and found that the absolute spend is about $59 billion in the US and probably about $30 billion in the EU [0,1]. With the US population at 327 million and the EU population at 514 million the 1:4 factor seems about right.

But there is another factor which IMO is important: The domestic spend on pharmaceuticals. The US healthcare system seems to be more pharmaceutical friendly. Yearly US spend on pharmaceuticals alone is about $330 billion [2], while yearly EU spend on "medical goods", which I assume to include pharmaceuticals, seems to be only €120 billion [3].

So pharmaceutical spend per capita is _also_ about 1:4 when compared to the EU, not just R&D.

As production usually follows consumption (which is why e.g. every US and EU car producer has domestic factories in China), one could also argue that the high US health care costs are the reason why so many US pharma companies exist.

[0] [https://www.healthaffairs.org/do/10.1377/hblog20170307.05903...](https://www.healthaffairs.org/do/10.1377/hblog20170307.059036/full/#one)

[1] [https://www.vfa.de/download/the-pharmaceutical-industry- in-g...](https://www.vfa.de/download/the-pharmaceutical-industry-in- germany.pdf)

[2] [https://www.actuary.org/content/prescription-drug- spending-u...](https://www.actuary.org/content/prescription-drug-spending-us- health-care-system)

[3] [https://ec.europa.eu/eurostat/statistics- explained/index.php...](https://ec.europa.eu/eurostat/statistics- explained/index.php/Healthcare_expenditure_statistics)

<comment><USER12> OECD tracks pharmaceutical spend directly. It's about $1,200 per capita in the US, versus about half that in say Spain or France. [https://data.oecd.org/chart/5OrU](https://data.oecd.org/chart/5OrU). Richer countries like Australia, Canada, and Switzerland spend $850+.

<comment><USER11> Thanks! Way better source.

</comment></comment></comment></comment></comment></comment><comment><USER13> EQRx could potentially lower drug prices, but only because of competition and not out of the goodness of their hearts. Few companies deliberately lower profits without exogenous pressure. If a company has a 20 year monopoly on a drug and a similar drug shows up before the patent expires, competition will somewhat bring down prices. I say somewhat because healthcare is a convoluted opaque mess.

</comment><comment><USER14> >Drug companies like to quote statistics that prescription drugs represent only a tenth of medical costs.

How true is the claim that drug costs are ~10-15% of the final cost to patients? Apparently, It's the caregivers (hospitals, doctors, nurses etc.) who make up the vast majority of the final bill.</comment>


Perfect example of unintuitive Microsoft workflow - <AUTHOR> http://dendory.net/blog.php?id=4e5d9c30

<comment><USER2> 1\. There is no magical flag that allows all mail sent by anyone at Microsoft to hit your Hotmail inbox: that could easily be abused, and for many users it would defy their expectations. I've sent your post along to the people responsible for spam filtering in Hotmail, though, to take a look at.

2\. It looks like you were using Chrome to try to open a document in the OneNote client? For better or worse, you have to have a plugin to do that today; there's no browser-native way for a website to launch a program on your PC.

The plugin we have gets installed with Office 2010 and works on IE and Firefox. It does not currently work in Chrome, which is probably why you saw the issue.

That's not to say that the metapoint is lost on all of the folks who work on these features, and we try very hard to make flows work with no edge cases. Even for a company like Microsoft shipping often requires tough compromises, though, and unfortunately you hit some of the rough edges in our experience we haven't yet been able to address.

(Disclaimer: If it wasn't clear already, I work on some things related to this flow.)

<comment><AUTHOR> 1- I've never seen a company's automatic emails be flagged by its own mail system, that's pretty impressive.

2- I'm actually using Firefox, and I had 2 instances of that plugin installed for some reason, and it still gave me the error. Although my main point is that the error dialog doesn't tell you the real problem.

And btw I like OneNote, it just seems I won't be using the desktop version.

<comment><USER3> _1- I've never seen a company's automatic emails be flagged by its own mail system, that's pretty impressive._

It's as fair and annoying as having downloads blocked from Microsoft's website on a freshly installed server. (On one hand, at least they treat their site the same but on the other hand, it's putting yet another needless hurdle in the way of what I want to do. If I didn't trust Microsoft why would I be running Windows at all?)

</comment></comment></comment><comment><USER4> Looks like "Microsoft doesn't understand user experience" articles are becoming popular again on Hacker News.

I would like to remind people that even a design focused company like Apple do softwares with crappy user experience (think iTunes).

User experience is hard to get right and what you would consider a good user experience might not be a good one for your neighbour: some prefer clicking on buttons, some prefer clutter-free UIs with keyboard shortcuts.

<comment><USER5> Are you crazy? iTunes has one of the best user experiences of any app on the market today. Period.

<comment><USER6> Yes I agree, with two exceptions - first when clicking somewhere opens a new itunes sub-window (who wants that?) and second they changed the reveal-in- finder command from apple-R to shift-apple-R

Other than that it rocks (literally)

</comment></comment></comment><comment><USER7> Pretty much any time I try to download anything from Microsoft's websites they are a mess and make almost no sense. Everyone who is a web designer and responsible for web design at Microsoft should be fired immediately.

One example of Microsoft's ridiculousness is their hosted Exchange services -- <http://www.microsoft.com/online/products.aspx> I signed up for it some time last year. I couldn't get it to work with Outlook 2010. Their answer, call tech support. I don't sit on the phone to talk to just anyone, and I damn well don't sit on the phone to make hosted web services work with a companies desktop platform.

Another example is Microsoft Adcenter. Adwords was more advanced in 2005. A 6 year time gap for technology is unheard of. Virtually impossible for them to play catch up at this point. Their desktop software, Adcenter Desktop, running on their own operating system, Windows, crashes all the time. Ooops. The usability of the Adcenter web interface compared to Google's is like receiving a massage verse walking on glass. Its painful.

The sad thing is, Microsoft sits on a pile of very very valuable assets. Even with slightly above average management their business could stand to benefit enormously from Google's antitrust problems. The adcenter platform should be free money raining from the sky for them. But it won't be. And that is too bad because we need real Google alternatives.

</comment><comment><USER8> well good points but what does it have to do with intuition? wrong word perhaps?

</comment><comment><USER9> This reminds me of Bill Gates 2003 complaints about trying to get Microsoft Movie Maker installed.

<http://boingboing.net/2008/06/25/bill-gates-2003-flam.html></comment>


What Lies in the Path of the Revolution [pdf] - <AUTHOR> https://github.com/amb26/papers/blob/master/ppig-2018/What_Lies_in_the_Path_of_the_Revolution.pdf <comment><USER2> The abstract for those who don't want to download the file:

> Increasingly, the rights and capabilities to own technological artefacts, > where they exist at all, are re- served to corporations and not to citizens. > There are historical, economic, metaphysical, ideological and cognitive > reasons for this situation, in addition to purely technological factors, > which we will trace by following the fate of various concrete examples, > analysed into five categories of ownable elements. These categories are > those of ownable function, ownable expression, ownable data, ownable > installa- tions and ownable economies. In this paper, we attempt to align > these goals of ownability into a research and activism program by describing > a set of revolutionary goals in each category, and tracing ways we could > reach them.

</comment><comment><USER3> This is extremely interesting, and I was pleasantly surprised to see Marx and Engels in a footnote, even though the content of the paper wasn't quite what I was expecting. There has been significant debate about gradualist approaches to reforming society, even assuming common goals.</comment>


Show HN: A simple drawing and animation framework in Carp - <AUTHOR> https://github.com/hellerve/anima <comment><AUTHOR> If you're interested in how this works or would like to try it out you can also checkout the blog post I wrote about the system:

[https://blog.veitheller.de/Introducing_anima.html](https://blog.veitheller.de/Introducing_anima.html)

Please note that you have to have Carp and SDL2 (including SDL_image) installed.</comment>


Ask HN: What is with all the anti-veganism on the BBC lately? - <AUTHOR> This week the BBC bashes new plant-based protein:
https://www.bbc.com/future/article/20200129-why-vegan-junk-food-may-be-even-worse-for-your-health?utm_source=pocket-newtab
Last week the BBC claimed veganism makes you less intelligent, and their nutritional claims were soundly debunked by HN.
https://news.ycombinator.com/item?id=22174445
What's going on over there, UK friends? <comment><USER2> It's comforting to see this post, I noticed that as well.</comment>


Ask HN: How to get interviews? - <AUTHOR> I am currently about 3 years in my career as a developer. I started as a Frontend Developer but now I mostly do Backend. I use Node.js but I have worked with variety of other languages, including Java. I have worked with 2 startups so far and am currently looking for a change, mostly because the current workplace is not very aligned with my career goals. I am from India without a CS degree. I would prefer moving to US or Canada.
I have been scouring tons of job listing websites like whoishiring, AngelList, stackoverflow jobs and applying actively but cannot get interviews lined up. Mostly I would never get a reply for the position and sometimes automated (or possibly not) rejection mails.
All this is degrading my work and I feel like I am not good enough while I know friends who are in the US who have similar work experience as I do getting interviews easily.
Are there things that I can do better? I have started a few projects of my own and will possible make them open source soon but I have heard from friends that it hardly matters. I am absolutely confused. <comment><USER2> > I am from India without a CS degree. I would prefer moving to US or Canada.

> I know friends who are in the US [...] getting interviews easily

Are you applying for full-time positions in the US and Canada from outside those countries without a visa? Companies take a risk in the visa process, e.g. they have to proof to the government (DHS, can't remember) they found nobody equally qualified in their own country, it's a time-consuming and expensive process. A candidate already in the country has a huge advantage.

<comment><AUTHOR> Sadly, yes. What options do I have in this case?

<comment><USER2> You can try getting a position at a subsidiary of a US/Canadian company inside India. Then hope to transfer using an L1 visa (company internal transfer) after a couple of years. No company will promise that in writing to new hires of course.</comment></comment></comment>


Happy Birthday Steve Wozniak - <AUTHOR>

Today is Woz's birthday. Co-founder of Apple and inventor of  the first personal computer.
Woz first showed off his home-built computer, the Apple I, at Silicon Valley's Homebrew Computer Club in 1976. He brought out the first low-cost personal computers to have color, sound, hi-res graphics, and floppy disks.
I think all of us here have someway or another been touched by Woz's early work. So cheers to you Woz, Happy Birthday! You are truly an inspiration. <comment><USER2> Happy Birthday Steve, his inovation and mentality is inspiring and his Segway's are awesome. (Gotta love Segways).</comment>


It is clear that Apple is straying from its mission - <AUTHOR> Since the 90s, I've viewed Apple as the computer company that was breaking down the boundaries between humans and computers with capable technology. From the GUI to a Unix OS to the touchscreen; Apple was taking major steps to breaking down this boundary while the competition, Microsoft, was coming up with minor gimmicks with less reliable tech. Now it's the other way around!
The signs were showing, but I could not accept it until a worthy competitor appeared. The watershed moment came with COURAGE, but I reached the final stage of acceptance when Microsoft revealed the Surface Studio (and a host of other enticing gadgets) before Apple's pricey, dongle-loving, touch strip gimmick that is the MBP.
I'm feeling more and more confident in Satya's Microsoft. They're sticking through with the productizing of their research in multi-touch and 3D Augmented Reality. Their opening up of developer products (.NET is on GitHub and Edge was one of the first to fully support ES6) is bearing fruit. Now that Windows runs Ubuntu's userland, I don't have the lack of a Unix terminal to dissuade me from their OS.
Apple used to delight me with the smallest detail, but now they piss me off over the smallest fuck ups. Their gadgets now feel more like boundaries while Microsoft has been starting to break them down. <comment><USER2> Couldn't agree more. It's arguable now that windows 10 with hyper-v and bash is now the ultimate dev platform. They keep adding while macos does does the opposite</comment>


Want to unlock your door with your face? Windows 10 for IoT Core - <AUTHOR> http://www.pcworld.com/article/2962330/internet-of-things/want-to-unlock-your-door-with-your-face-windows-10-for-iot-core-promises-to-do-just-that.html <comment><USER2> > Want to unlock your door with your face?

That sounds like something a mugger would ask as he threatens you on your front porch.

(No sexism intended - statistically, most muggers are male.)</comment>


By November, Russian hackers could target voting machines - <AUTHOR> https://www.washingtonpost.com/posteverything/wp/2016/07/27/by-november-russian-hackers-could-target-voting-machines/ <comment><USER2> Part of me doesn't like the whole red scare deal, but also, electronic voting machines should be banned outright. You think that after the OPM scandal that those things have the highest security??

<comment><USER3> This Red Scare is getting ridiculous now.

One of the best things to come out of this election is how much political strategy has been exposed to the general public, inc. media complicity. I have so little faith in the political parties and media.

<comment><USER2> As do I. My stomach has been churning in disgust seeing the collusion big money media and politicians. This latest attempt to drum up the Russian boogeyman is incredibly laughable, especially since they laughed it off four years ago. Yes Russia is corrupt, shitty regime, but they can't be culpable for every U.S. politician's failures and dirty deeds (where the inverse happens in the Russia), we've become so much like our so called "enemies", obeying and apologizing to top party members, publishing stories that only favor our dear leaders, etc.

<comment><USER4> Russia is no more of a "corrupt shitty regime" than your own. So maybe, just maybe, you should focus on constructive criticism of your own country instead of offending others, hmmm? After more than 60 years of anti-Russian propaganda I am not quite surprised that most Americans take every opportunity to jab at or feel somehow morally or otherwise superior to Russia and Russians, yet you would be surprised just how opposite and rather humane most Russians think of Americans and the States... and not because of some God given right of American righteousness and superiority, or some Russian inferiority complex.

So cut it with the anti-Russian nonsense.

<comment><USER2> America has better bread and circuses

<comment><USER5> only for those with not expensive tastes

</comment></comment></comment></comment></comment></comment><comment><USER6> The whole concept of electronic voting machines seems ridiculous to me. Not only are they open to attacks at every level, there really isn't a significant benefit to implementing them, beyond possibly saving some trees.

The peace of mind in trusting election results seems well worth the hassle of paper.

The other big component that I think is missing from the debate over electronic voting is the cost to communities. I loved voting (on paper) in the 2012 election, then staying up a few hours to tally the ballots with the community. Voters came together to count the ballots together and deliver a final count, which has the dual benefit of getting as many eyes on the process as possible (less chance of fraud) and putting 'by the people' back into the election process.

I've seen complaints that that isn't scalable, but I don't buy it. Any community can be broken up into sub-groups that can cast their ballots locally and work together to tally them, even if a group just covers a city block.

<comment><USER7> One could argue that a properly implemented method of voting that involves computers in some capacity would decrease the chance of fraud, and allow voters to verify that their vote was counted. It could also allow more votes to be counted because it could immediately tell them their vote was invalid (how many hanging chads in Florida 2000?).

The problem is that we have companies like Diebold make this stuff. Companies that want to make a buck and cover their asses rather than make something secure and good for the people.

</comment></comment><comment><USER8> " Even so, we have to accept that someone is attacking our nation's computer systems in an apparent attempt to influence a presidential election. "

This is so comical. If the DNC was actually acting in an ethical, responsible and legal way, the email release would not influence the election one wit. Instead it makes it even more clear that their apparatus had already chosen the winner - and it wasn't Sanders.

On the other: Why even have computerized ballots? Back to paper. Not perfect, but hacking problem solved.

</comment><comment><USER9> If voting machines are that insecure, then you should be worried about Chinese hackers, and Arab hackers and even American Republican hackers. Let's be frank, voting machines that are insecure can be exploited by ANYBODY who cares to influence an election and those with the most to gain/lose are members/supporters of the parties whose candidates are standing for election. In other words USA hackers should be your main worry. If you can secure the voting systems against them, then you have a chance at a fair election. Otherwise all bets are off and all the foreign hackers will be battling over who controls the vote.

</comment><comment><USER10> We'll have 5 different organizations trying to rig the same election to further their own agendas. Which group is the smartest? Whose manipulations will carry the day? Find out next time on Who Wants to Screw a -19 Trillionaire!

</comment><comment><USER11> That's preposterous, don't you know the cold war is over?

[https://www.youtube.com/watch?v=T1409sXBleg](https://www.youtube.com/watch?v=T1409sXBleg)

<comment><USER2> Smug, condescending, and dismissive are recurring traits in politicians who eventually eat their words

<comment><USER11> And remember when Hillary "pushed the reset button" with Russia - because (I guess) relations with them were so bad under Bush - but her team screwed up the translation? That was a classic:

[http://news.bbc.co.uk/2/hi/7930047.stm](http://news.bbc.co.uk/2/hi/7930047.stm)

</comment></comment></comment><comment><USER12> When does this sort of thing become an act of war? It seems like "cyber" stuff is taken much less seriously in that respect. Maybe it just gets more attention.

</comment><comment><USER13> Not a problem. Let's use paper. (And dear Supreme Court, kindly stick to the constitution this time.)

</comment><comment><USER14> Obligatory XKCD reference: [https://xkcd.com/463/](https://xkcd.com/463/)

</comment><comment><USER5> They do not need to, since siloviki silently support Trump anyway.</comment>


Ever back out of a accepted Job? - <AUTHOR>

For the first time in my life I accepted a position and then had to back out at the last minute because another better oportunity presented itself at the last minute.   I have to say I feel like an ass but I just felt I had to do it.<p>How many of you has this happened to?   How did you go about managing the situation. <comment><USER2> I've backed out of going to graduate school after accepting to go, which is similar to a job offering since they were offering a fellowship. The only thing I felt bad about was I didn't realize soon enough that I shouldn't go, but it just wasn't the right move for me. Some people may disagree, but I think you should back out of it. There are very few companies who would have qualms about firing you if they could find a better employee to replace you or would make more money by firing you. You shouldn't feel guilty about holding them to the same standard (i.e. you found a better employer or one who offers a better deal).

</comment><comment><USER3> I've been on the other side of that several times - having someone who we tried to hire back out at the last minute. You look like an ass from that side too. I'm not trying to pass judgement (you gotta do what you gotta do), but it's likely you've burnt your bridges permanently with that company.

</comment><comment><USER4> It's just like any other business transaction. If the company wanted you not to bail out at the last minute, they would have signed a contract that stops you from doing that. Did they do that? If not, you don't have any obligations and a reason to feel like an ass.

</comment><comment><USER5> Ask if they would pay you better, let you do what you are going to do in that new position and give you a career path like the other offer would. If no, say you have something better.

</comment><comment><USER6> Did you sign an offer letter? If so, you've probably burned that bridge and several adjacent.

If not, it's touchy and tender, but not necessarily a mark against you.

</comment><comment><USER7> Ha, my team in Seattle was going to get a new developer last week until he backed out at the last minute. That wasn't you, was it?

</comment><comment><USER8> Yup did that this year. But I decided to stay at my current position.</comment>


Interview with Tom Wheeler – Obama FCC Chairman – On the Future of US Internet - <AUTHOR> https://backchannel.com/obamas-fcc-head-is-worried-about-our-online-future-628b8f63efc5#.t7sxd4dnv <comment><AUTHOR> I find the argument about municipal broadband the most compelling. I see no competitive reason that municipalities should be prevented from building out their own internet infrastructure if their constituents demand it.</comment>


Chromium DNS performance - <AUTHOR> https://plus.google.com/103382935642834907366/posts/FKot8mghkok

<comment><USER3> I hate articles I can't read on my mobile iOS device due to endless redirects. Esp when they sound like interesting articles. Wtf google + team?

<comment><USER4> It seems to work fine here (iPhone 4, iOS 5.1), and I've not had a problem with g+ on 4.3.x or 5.0.[01], AT&T and various Wi-Fi networks...

Is it really endlessly redirecting or perhaps timing out and trying to reload? I had that happen once when latency was extreme, on one occasion.

EDIT: here is the text for those who are having trouble.

Host resolution accounts for a significant portion of the networking component of page load time. As such, Chromium developers are always looking to optimize it. Here I present the current considerations of host resolution in Chromium, and look forward to the IPv4+IPv6 dual stack world and what that entails for browsers (and what Chromium is doing about it). I finish off with a presentation of some of the latest data we're operating on.

Currently, Chromium uses getaddrinfo() to ask the OS to resolve a host. This is a cross-platform, blocking API that abstracts the complicated host resolution. There are a number of advantages to using this API: * Correctness - it handles all the complicated rules of hostname lookup correctly. It understands /etc/hosts, non-DNS namespaces like NetBIOS/WINS, etc. Re- implementing this behavior would be difficult. * OS Caching - we get to share the OS host cache with other applications. Note that this doesn't exist by default on Linux systems. * Less code - Having to maintain code sucks. It leads to code/binary bloat and endless bugs for corner cases and OS-specific issues. And it takes engineering time.

There are many disadvantages to using this API: * Blocking - we need to use unjoined worker threads so we don't block critical threads * Performance - <USER2> can't optimize the host resolution process, since it's behind the getaddrinfo() call. There are lots of optimization opportunities we miss. * Application caching - we can't tell how long to cache a DNS record for in the application, since we don't get TTLs from getaddrinfo(). We try to be safe by only caching for a minute.

As noted previously, host resolution is a substantial portion of networking time, so we try to start it as soon as possible. Our network predictor subsystem learns to predict network requests and may initiate host resolution prefetches. While this provides substantial performance benefits, it also can lead to horrible user experience when it overloads upstream resolvers/devices. For example, check out <https://code.google.com/p/chromium/issues/detail?id=3041> and <https://code.google.com/p/chromium/issues/detail?id=12754>, where users report "internet loss" that seems correlated to DNS prefetching. The problem is that Chromium is issuing too many DNS queries in a short period of time, which may overload upstream devices (typically cheap NAT routers) which then enter a anti-DoS mode and ignore DNS queries for a period of time. Due to excessively high getaddrinfo() failure timeouts varying by platform, this results in Chromium appearing to "hang" while displaying "Resolving host…" in the status bar. Our first step to combat this was setting limits on the number of outstanding getaddrinfo() calls. Currently, the limit is 8. However, it's still possible to overload upstream devices. Worse, when that happens, now we have this limit of 8 jobs, so even if we cancel the navigations and wait a short while for the NAT router to exit its anti-DoS mode and then try to browse to other pages, we can't issue anymore host resolutions until one of the 8 getaddrinfo() calls timeout. We identified this problem in <http://code.google.com/p/chromium/issues/detail?id=73327> and implemented back off behavior to recover from this situation. The important thing to note though is that we are somewhat limited by existing network devices in the degree to which we can aggressively do DNS lookups for performance reasons.

Prior to last year's World IPv6 Day, we implemented Happy Eyeballs in Chromium (<http://codereview.chromium.org/7029049>) to mitigate the problem of broken dual stack implementations on the web. In particular, when a hostname had both IPv6 and IPv4 addresses and the first address is IPv6, we will connect() to the IPv6 address, but start a 300ms timer to fallback quickly to connect() to IPv4. We use a fast fallback system rather than simply racing both IPv6 and IPv4 in order to give an edge to IPv6, to compensate for the likelihood that the IPv6 path is initially not as fast as IPv4, since we want to encourage people to switch to IPv6. Note that this only addresses the actual TCP connect() to a IPv6 address where the IPv6 pathway is broken (be it an intermediary or the origin server).

The getaddrinfo() call itself is often slower (<http://www.belshe.com/2011/06/15/ipv6-dns-lookup-times/>) in the dual stack IPv4+IPv6 world due to serialization of the DNS queries (<http://www.potaroo.net/ispcol/2011-12/esotropia.html>). Usually the OS's getaddrinfo() implementation will issue the AAAA query first, and only once it completes, issue the A query. OS X Lion (when using CFSocketStream) changes this because it issues the A and AAAA queries in parallel, which is generally superior in terms of performance. However, remember that this doubles the number of DNS queries in a period of time, so we are more likely to hit the upstream device DNS overload situation again, since the Chromium host resolution limit is per getaddrinfo() call and not per DNS query (that's an OS specific implementation detail) and we don't pace. Also, it strictly prefers whatever is fastest, so it doesn't help incentivize IPv6 adoption. Note that Chromium will disable dual stack support if it is unnecessary (no IPv6 interfaces). But if you have both IPv4 and IPv6 interfaces, your slower getaddrinfo() calls are probably slowing down your browsing experience.

So what is Chromium doing about the various issues? After a long time of being unwilling to sink our time into it, we're going to implement our own DNS stub resolver. It sucks that we have to write this code and get it to work on all platforms and lose OS caching, but the browser is an increasingly important application, almost its own OS (especially with ChromiumOS), so it makes sense to do it. We've got an experimental DNS stub resolver implemented and are working on implementing features and flushing out bugs with it right now (--enable-async-dns).

Once we have a functionally correct DNS stub resolver, we can begin playing with optimizations. One of the obvious things we'll look at is parallelizing A and AAAA queries. Also, this obviously gives us the option of beginning a TCP connect() once we have a DNS response, be it A or AAAA. There are some subtleties here, because there's the performance vs IPv6 incentive tradeoff. On the one hand, the most performant choice would be to do a TCP connect() immediately upon receipt of the A or AAAA response. On the other hand, we want to incentivize folks to adopt IPv6, so even if the IPv4 pathway may be faster, we debatably ought to give IPv6 a slight handicap (a timer) here to help with adoption. When striking this balance, we also need to be careful not to issue too many DNS queries at the same time and overload cheap NAT routers. Clearly we'll have to do some experimentation and watch our bug tracker to make sure things work reasonably. We should switch our job accounting to be on a per DNS query basis, not a per hostname basis, to better account for parallel A/AAAA queries. Also, now that we can detect these NAT router overload situations, we should remove the fixed concurrency limit of 8 queries to be dynamically determined, so users aren't limited by engineering for the least common denominator hardware.

There are a bunch of other ideas we're kicking around, although we're undecided if/when we want to pursue them. We obviously want to experiment with revisiting DNS retransmission timers, failovers to different servers / name suffixes, etc. Also, we'd like to experiment with connecting to different IPs in the DNS response(s), in order to figure out which ones are faster (lower RTTs), so we can prefer connections to those IP addresses. This may cause some compatibility issues, so we need to be careful, but often times RTT may vary greatly among IPs in the DNS response(s), especially between A and AAAA records. Also, now that we have TTLs, if we find cached DNS entries that are about to expire, but are likely to be used again, then we can reissue the DNS query to prevent expiration.

OK, now that we've explained some of the tradeoffs we're looking at, we can delve into some numbers to assist in our understanding of how important these issues are. First off, how long does a getaddrinfo() call take? Well, that's complicated. Chromium issues a number of speculative host resolutions for non- existent hosts in order to detect domain hijacking. We also speculatively resolve tokens typed into the omnibox in case they're real hosts. Also, there will be a number of considerations with platform and IPv4/IPv6. Note that all data I present here are based on Chrome 17 on a single day in March (sorry, we have some internal issues with our metrics analysis dashboard, otherwise I'd get a longer sample). So take it with many grains of salt (especially Linux, which has a far smaller population), but it's still hopefully largely relevant. I'll update again later with way more samples.

getaddrinfo() invocations for non-speculative requests, with successful resolution Win: mean - 644ms. 10% in <= 1ms, 25% in <= 12ms, 50% in <= 43ms, 75% in <= 119ms, 90% in <= 372ms. Note, there's an upward blip of 1.45% of samples completing in around 1s (95.90 percentile), due to the Windows DNS retransmission timer. Mac: mean - 230ms. 10% in <= 0ms, 25% in <= 5ms, 50% in <= 28ms, 75% in <= 67ms, 90% in <= 279ms. Note, there's an upward blip of 2.11% of samples completing in around 300ms (91.51 percentile), and another of 1.07% at 1s (97.36 percentile), indicating retransmission timers around these intervals. Linux: mean - 293ms. 10% in <= 2ms, 25% in <= 12ms, 50% in <= 37ms, 75% in <= 89ms, 90% in <= 279ms. Note, there's an upward blip of 1.81% of samples completing in around 4250-4900ms (99.26 percentile).

I'm quite disappointed in Linux's retransmission timer here, as a 4s hang in navigation on 1.81% of URL fetches is a pretty horrible user experience if you ask me. Dude, almost 3/4 of the remaining samples in the distribution are covered by the retransmission, retransmit earlier already! Also, it's not immediately clear from the percentiles I presented, but Windows and Mac beat Linux on low latency responses, most likely due to DNS caching being off by default in Linux distributions. Remember that this sample of getaddrinfo() calls ignores failed resolutions and also only applies to getaddrinfo() calls that either weren't predicted by our network predictor, or the network predictor's DNS prefetch did not complete early enough to completely hide the DNS latency.

getaddrinfo() invocations with successful resolution, AF_INET (IPv4 only) Win: mean - 443ms. 10% in <= 0ms, 25% in <= 5ms, 50% in <= 37ms, 75% in <= 103ms, 90% in <= 322ms. 1.36% samples completing around 1s (96.65 percentile). Mac: mean - 181ms. 10% in <= 0ms, 25% in <= 3ms, 50% in <= 24ms, 75% in <= 58ms, 90% in <= 182ms. 0.89% samples completing around 1s (97.97 percentile). Linux: mean - 243ms. 10% in <= 2ms, 25% in <= 12ms, 50% in <= 32ms, 75% in <= 89ms, 90% in <= 242ms. 1.50% samples completing around 4250-4900ms (99.47 percentile) getaddrinfo() invocations with successful resolution, AF_UNSPEC (IPv4+IPv6*) Win: mean - 363ms. 10% in <= 1ms, 25% in <= 8ms, 50% in <= 37ms, 75% in <= 103ms, 90% in <= 279ms. 1.06% samples completing around 1s (96.97 percentile). Mac: mean - 266ms. 10% in <= 0ms, 25% in <= 6ms, 50% in <= 37ms, 75% in <= 279ms, 90% in <= 372ms. 9.77!!!!!!!% samples completing around 300s (83.79 percentile), 5.74!!!!!!% within 322-372ms (89.53%). Linux: mean - 359ms. 10% in <= 1ms, 25% in <= 12ms, 50% in <= 50ms, 75% in <= 119ms, 90% in <= 322ms. 1.31% samples completing around 4250-4900ms (98.79 percentile) AF_INET:AF_UNSPEC ratios (W/M/L): 1.09/7.05/26.6

There's a lot of stuff going on here. First, we have to study how Chromium chooses whether to use AF_INET or AF_UNSPEC. For details, please refer to [http://code.google.com/p/chromium/source/search?q=IPv6Suppor...](http://code.google.com/p/chromium/source/search?q=IPv6Supported&origq=IPv6Supported&btnG=Search+Trunk). Basically, we run a series of basic local tests (creating sockets, availability of a IPv6-enabled interface, etc) to see if we can support IPv6. It's pretty fascinating to see that, despite our efforts on Windows to identify this, basically 50% of the population still seems to support IPv6, while the proportion of Mac and Linux users is way way lower. The local tests are designed to be a bit conservative, so I think it's more likely that the Windows IPv6 probing is overly conservative. So there's a lot more conflation in the AF_INET vs AF_UNSPEC results for Windows. It's indeed unclear to me at all whether Chromium's dual stack implementation is slower at all for Windows, although I suspect that's mostly due to our insufficient IPv6 support probing. On Mac and Linux, dual stack support is markedly slower than IPv4 only support. It's quite incredible seeing the 300?ms retransmission timer in Mac having such a HUGE effect. In dual stack Mac, this retransmission seems to help for 10-15% of the time!!! TODO(willchan): run manual tests on Mac versions to study the retransmission…is the OS doing the AAAA query first and then falling back to A in 300ms?

Roughly speaking, the times of resolution successes exhibit a roughly gaussian distribution, other than at the low-end (due to caching in the DNS hierarchy) and the retransmission timeouts. Resolution failures on the other hand are distinctly not gaussian. On Windows, we see 27.66% finish within 3ms and slowly decrease through 9ms (35.74 percentile), spike up again between 10-24ms (49.80 percentile), spike again around 300ms (58.8 percentile), and a huge spike in the 1798-2394ms buckets (24% in these buckets, at 84.5 percentile), and then one last spike between 10048-13386ms (8.6% in these buckets, 98.76 percentile). Mac and Linux distributions differ greatly from the Windows distribution, to such a degree that I don't believe can be explained by differences in population, but most likely are due to OS differences. Their distributions are start very high in the low latency areas, decreasing into some very long tails, with much more muted spikes at retransmission timers. On Mac, 70% of resolution failures complete in <=2ms. There's a small (1%) spike in the 762-1013ms bucket (93.4 percentile), a larger (2.63%) spike in the 27425~ms bucket (98.6 percentile), and a last 1% spike at the 56188~ms bucket (99.87 percentile). Linux is basically the same as Mac here, although a lower percentile (61.6) finishes in <=2ms and different failure/retransmission timeouts. #chromium #dns #ipv6

<comment><USER3> Try logging out and then reading. I'm on iPhone 4 / iOS 5.1. Endless redirects for me, all the way down..

</comment></comment></comment><comment><USER5> It's unbelievable how kitchen-sink distros like Ubuntu and their derivatives don't install dnsmasq by default.

</comment><comment><USER6> >But if you have both IPv4 and IPv6 interfaces, your slower getaddrinfo() calls are probably slowing down your browsing experience.

 _Disables IPv6 interfaces._

-IPv6 adoption is too low!!

</comment><comment><USER7> Interesting topic, but a great example of why Google+ makes a terrible blogging platform - the formatting and link-wrapping make it impossible to read in places

</comment><comment><USER8> So much smoke. I like how they mask very crystal-clear strategy with barely- relevant DNS statistics and data in an attempt to obscure what's happening.

My prediction from 2 years ago + and again a couple weeks ago rings true now: [http://www.forbes.com/sites/eliseackerman/2012/02/25/a-close...](http://www.forbes.com/sites/eliseackerman/2012/02/25/a-closer- look-at-google-public-dns/)

 _"I'll reiterate my view that I think Google controlling search, the browser, and the network or DNS layer is a dangerous trifecta that the consumer will probably be best served avoiding. I'm sure we'll find out soon enough."_

<comment><USER9> It seems that you don't know what a stub resolver is.

A stub resolver is a piece of software which uses the DNS (and other) servers listed in the system configuration to resolve names to addresses. The stub resolver may use a variety of strategies when deciding which of the configured servers to use, how to time the queries, and what to cache; but stub resolvers respect the system admin's wishes WRT what name resolution servers to talk to.

See: <https://tools.ietf.org/html/rfc1123#page-74> and: <https://tools.ietf.org/html/rfc4033#section-7>

<comment><USER8> There is no connection between a stub resolver as Chrome is implementing it, and the system-indicated DNS server.

This, as an aside, is what's wrong with HN. People cite RFCs or other datapoint as if it validates their point, when in fact, it does no such thing. You can run 10 stub resolvers on your system, each talking to a different DNS server. A stub simply indicates that it lacks the fortitude for full-blown root-down DNS resolution and validation. It might even still have a cache of sorts. That's it.

Why my post got marked down, I'll never know.

<comment><USER9> Firstly I'm _not_ making a point, I'm giving you information that your vague hand-wavy rant seems to indicate that you don't have. Secondly, I point to the RFCs to indicate that my description of stub resolvers isn't just my opinion, but is somewhat in line with the community definition.

I never said that one could not have multiple stub resolvers installed on a single system. I also never intended to imply this. There's no reason to believe that the Chromium stub resolver will do anything but intelligent caching and fetching. Certainly, there's nothing in the linked Google Plus post to make me believe otherwise. Mr. Chan mentions that writing a browser- specific stub resolver is really the wrong thing to do, but it's something that they really need to do to get their target DNS query performance and to detect and work around certain kinds of really shitty breakage.

Why did you get downvoted? Perhaps because you said

 _I like how they mask very crystal-clear strategy..._

when the strategy is anything but clear to anyone but you. By way of explanation, you point to a multi-page article about you and OpenDNS in which -among many other things- you say "[Google has] a separate privacy policy for Google DNS, and I'm sure they are hypersensitive about privacy concerns, so I wouldn't be too paranoid [about the possibility of DNS query logging and data mining].". I'm not sure what strategy it is that you're worked up about, but it would be really nice if you'd come out and say it, rather than being all oblique and mysterious.

Additionally, it would seem that you're the CEO or President or something of OpenDNS? It's exceedingly poor form to say vague FUDdy smelling things about companies that compete with your core business. If you're going to say something, man up and say it; don't make others feel around to maybe discover a hint of what your point was.</comment></comment></comment></comment>


Google's Python class - <AUTHOR> http://code.google.com/edu/languages/google-python-class/index.html <comment><USER2> The Stanford online CS101 class will be taught by the same teacher as these Google lectures. [1] I enjoyed him very much when I was just starting with Python.

[1]: <http://www.cs101-class.org/></comment>