Surprisingly popular - <AUTHOR> https://en.wikipedia.org/wiki/Surprisingly_popular <comment><USER2> The capital is not Philadelphia, it's Harrisburg. So the idea must be that most people think they are right and others will agree with them, but there are people who get it right and know that others get it wrong, and they are responsible for the difference in scores.

<comment><USER3> There are four groups of people:

A - "Philadelphia is the capital, and others will agree."

B - "Philadelphia is the capital, but most others won't know that".

C - "Harrisburg is the capital, and others will agree."

D - "Harrisburg is the capital, but most others won't know that."

This technique eliminates groups A and C from consideration, and measures the difference in size between groups B and D.

Both groups B and D think they know something other people don't, but B is wrong and D is right.

In cases where people feel like they have "inside" knowledge, generally speaking it's because they are correct and knowledgeable (group D), not because they are misled (group B).

You often arrive in group D by virtue of having been in group A and then learning the actual truth of the matter.

You can arrive in group B by falling prey to a conspiracy theory, which makes this technique perhaps invalid in such cases? I wouldn't be surprised if the question "Do vaccines cause autism?" had a surprisingly popular answer of "Yes".

<comment><USER4> This is a great summary / reduction of the article. I appreciated how you broke it down into groups and made the example more concrete.

<comment><USER5> It's not a summary/reduction, it's a a better article with more information.

<comment><USER3> Wow, thank you!! :)

</comment></comment></comment></comment></comment><comment><USER6> The way I understand it it's effectively it's a good way to identify trick questions and the people who correctly identify the "trick" (by saying that other people would fall for it and reply differently) are more likely to be right overall and it's reflected in the percentage difference.

Wouldn't you get the same result simply by saying "let's just consider the answers from people who said others would respond differently because they're more likely to be right"? That's more intuitive to me and I believe it's effectively the same operation.

<comment><USER7> Yeah, it's a bit like "second order" wisdom of the crowds.

First order: "How many jelly beans are in this jar?"

Second order: "What percentage of people will get the wrong answer to this question about Pennsylvania's capital?"

There's also the interplay with the Keynesian beauty contest: [https://en.wikipedia.org/wiki/Keynesian_beauty_contest](https://en.wikipedia.org/wiki/Keynesian_beauty_contest)

In some sense the stock market is an infinite regress of these "what will other people think about value..." questions.

<comment><USER8> Shares do actually have a fundamental value, based on their lifetime dividends, buybacks and liquidation. So unlike, say, Bitcoin or the Dollar, there is something to be gained from knowing the zeroth-order reality.

<comment><USER9> Disagree. Think about what makes a share liquid: lots of potential buyers (i.e. people perceiving it as a safe investment).

<comment><USER8> If you were offering a long period of healthy dividends for a good price, I wouldn't care if I could sell it.

<comment><USER9> Sure but where does the amount of dividends you're being given come from? All i'm saying is there's no objective "baseline", it's all perceived value no matter how you look at it.

</comment></comment></comment></comment></comment></comment><comment><USER10> It's an interesting heuristic, but without an exposition it's hard to know if there is any reason to believe it. Has this technique been assessed across various domains? Does it work better for political facts than for scientific facts? Where does it fail? It's a provocative article and a cool thought, but it seems more like a hypothesis.

(In particular, I'm referring to the assertion at the end of the article: " _Because of the relatively high margin of 10%, there can be high confidence that the correct answer is No._ ")

<comment><USER11> One of the wiki's sources might have been a better linked article.

The MIT summary [1] notes _" The researchers first derived their result mathematically, then assessed how it works in practice, through surveys spanning a range of subjects, including U.S. state capitols, general knowledge, medical diagnoses by dermatologists, and art auction estimates."_ Across all those areas, this technique had error rates about 20% lower than other competing techniques. Those techniques included simple majority vote to two different kinds of confidence-weighted scoring.

The paper earned a prestigious publication in _Nature_.

1: [http://news.mit.edu/2017/algorithm-better-wisdom- crowds-0125](http://news.mit.edu/2017/algorithm-better-wisdom-crowds-0125)

<comment><USER10> Awesome, thank you. And 'nacc found the link to the Nature article as well: [https://news.ycombinator.com/item?id=20547787](https://news.ycombinator.com/item?id=20547787)

<comment><USER12> One interesting thing about the paper is that it seems that the Wikipedia article incorrectly describes the procedure: respondents were not asked to guess whether the majority would agree with their position. They were asked to guess what _per cent_ of other respondents would agree. I think that's a pretty severe difference in the method.

</comment></comment></comment></comment><comment><USER13> Digging down a bit this method is published in Nature! [https://www.nature.com/articles/nature21054](https://www.nature.com/articles/nature21054)

</comment><comment><USER14> Looks like an idea for a semi-supervised ensemble method for machine learning:

Prepare two equally sized ensembles of classifiers, let's call them A and B.

1\. Train each classifier in ensemble A on labelled data to predict does a picture contains a cat.

2\. Take some other unlabelled dataset and collect answers from classifiers from A for each picture from this dataset.

3\. Train each classifier in ensemble B to predict average answer of classifiers from A for each picture from the unlabelled dataset.

Then for a picture from the test dataset it would be possible to get answers from ensemble A and from ensemble B and calculate what would be the surprisingly popular answer.

<comment><USER15> Please do this.

</comment></comment><comment><USER16> Let's look at another toy example, using made-up numbers:

 _Q: Is the earth flat?_

        Yes: 10%
        No:  90%

_Q: What do you think most people will respond to that question?_

        Yes:  3%
        No:  97%

_Analysis:_

        Yes: 10% -  3% =  7%
        No:  90% - 97% = -7%

_Conclusion:_

The earth is flat, I guess ;)

<comment><USER17> A more realistic example of that would be:

Q. Is the earth flat?

        Yes: 0.1%
        No: 99.9%

Q: What do you think most people will respond to that question?

        Yes: 10%
        No: 90%

Analysis: Yes: 0.1% - 10% = -9.9% No: 99.9% - 90% = 9.9%

Conclusion: the earth is not flat.

Not sure where you're getting the "10% believe the earth is flat" from, but if that's just your guess then it's pretty reasonable for people to say they think 10% of people will respond yes to the question.

<comment><USER16> If there are only 0.1% of flat earthers, why would 10% of people think it to be the most popular answer? That does not seem realistic at all.

Or did you interpret the question as suggested by bena in his sibling answer, ie as an estimate for the answer distribution (that would then have to be averaged) instead of a choice of the most popular answer (as was done in the wiki article)?

<comment><USER18> >That does not seem realistic at all.

That does seem realistic considering how often the topic pf flat-earthers comes up in discussions on social media, despite it being an extremely niche view.

<comment><USER16> OP suggested that 10% of people think there are more than 50% of flat-earthers when the real number is 0.1%. Phrased another way, 10% of people expect half the people they meet will be flat-earthers when it's 1 in 1000.

Doesn't seem realistic to me.

</comment></comment></comment></comment></comment><comment><USER19> I'm not sure what to think of a Wikipedia article without any sections on Uses, and the only Example being unreferenced.

</comment><comment><USER20> Notice the relationships between Dempster–Shafer theory -- AKA the theory of belief functions / evidence theory -- for reasoning with uncertainty...

[https://en.wikipedia.org/wiki/Dempster- Shafer_theory](https://en.wikipedia.org/wiki/Dempster-Shafer_theory)

</comment><comment><USER21> How does this play out in the stock market?

<comment><USER22> In the cases of Brexit and the 2016 US Presidential election, betting markets reported a majority of money on Remain and Clinton with a majority of bets on Leave and Trump.

Perhaps the number of bets is analogous to what people believe, while total money bet is analogous to what people think others believe. If so, the _surprisingly popular_ answer is the one with the smallest average bet.

The obvious analog in the stock market is average order size, but this isn't a generally-available statistic, and it's not clear to me whether it would be meaningful.

Volume (shares traded per time unit) is closely-watched, however, and several technical analysis "indicators" compare price changes to volume.[1][2][3][4]

High liquidity is a state of relatively small price changes per unit of trading volume. It generally means traders are not paying large premiums/discounts to previous trades. In a way, this corresponds to a smaller bets on future returns, and thus, to _surprisingly popular_ returns...

[1] [https://en.wikipedia.org/wiki/Ease_of_movement](https://en.wikipedia.org/wiki/Ease_of_movement)

[2] [https://en.wikipedia.org/wiki/Force_index](https://en.wikipedia.org/wiki/Force_index)

[3] [https://en.wikipedia.org/wiki/Volume- price_trend](https://en.wikipedia.org/wiki/Volume-price_trend)

[4] There are similar concepts in modern portfolio theorgy, such as Amihud illiquidity.

</comment></comment><comment><USER23> I see people mentioning whenever the answer is right or not, but that's actually irrelevant. The heuristic only measures how surprising the popular answer is to the population.

<comment><USER24> It's not presented as irrelevant in the article. The article presents it as "a wisdom of the crowd _technique_ ", and ends with "there can be high confidence that the correct answer is No".

So it is assumed that people using this technique could be inferring the correct answer by finding the surprisingly popular answer. Not just a observation about what the population thinks.

</comment></comment><comment><USER25> Using this on subjective questions (e.g. "Is Cymbeline one of Shakespeare's best plays?") would help you find... what? Underrated things?

<comment><USER26> I'm not sure the algorithm would emit any answer in the case where everyone has an equal level of objective foundation for their subjective belief.

But it would probably help in cases where popular opinion is entirely misinformed about the subjective question, not having _any_ basis other than (already misinformed) hearsay on which to form their own subjective opinion.

So, for example, if there was a musician who had an _absolutely terrible_ song that somehow became the song they were best known for (being a "one-hit wonder" whose song wasn't really a "hit"), the public might believe that that song is their best song, since it's the only song of theirs the public has ever heard of. Experts (i.e. people who have heard more than the one song of theirs), on the other hand, would tend to agree that it's certainly not their best song.

(Given that example, I'm inclined to suggest that you could use this algorithm to determine when people are being judged overly-harshly for things, e.g. whether to ban someone from a website just because they've received a lot of reports about that person's behavior.)

</comment></comment><comment><USER27> Have any political pollsters ever done this sort of question format?

<comment><USER24> You mean like this:

1\. Who will you vote for?

2\. Who do you think will win?

I don't know if we can learn anything from the surprisingly popular answer here, but now I wonder if question 2 would be a better predictor of the election result. But it will be tainted by previous polls and news.

</comment></comment><comment><USER28> Forgetting one thing, surpassingly popular _to who_? You need to answer the "to who" part.

</comment><comment><USER29> The wisdom of crowds is a truly fascinating thing... the fact that people don't really need to know what they're talking about as long as there's enough of them really blows my mind.

<comment><USER24> The concept in the article is pretty much a counter example of that. It says that if there is a core part of the crowd that are expert on the topic at hand, and they answer differently from the crowd wisdom, there is a way to find out about it using a meta-question about the crowd wisdom itself.

</comment></comment><comment><USER30> I wonder if this could be viewed as an application of [https://en.wikipedia.org/wiki/Dunning%E2%80%93Kruger_effect](https://en.wikipedia.org/wiki/Dunning%E2%80%93Kruger_effect)

<comment><USER31> Nope. That is overconfidence of the ignorant, and this is dealing with trusting the high confidence of the correct.

It doesn't really matter here that people answered incorrectly.

<comment><USER24> I don't think so. It's not saying anything about how highly confident the correct are. It's about how they know that most people are incorrect. It's about misconceptions.

There is this core part of the unpopular-but-correct answerers that knows that the incorrect answer is popular. The larger this core part is, the higher the "score" of the "surprisingly popular" answer is. It's about finding something that is "mistakenly popular".

<comment><USER31> Yeah but this is in the context of not knowing the correct answer.

Saying that you believe the popular option is different to yours is a display of high confidence in your own answer, and low confidence in other people's knowledge.

Dunning kruger only deals with highly confident wrong people, which isn't the interesting group in our surprisingly popular measurements.

Bringing dunning kruger in to it would make more sense if there was an "im not 100% confident" option, but instead it was an absolute dichotomy.

It doesn't really mean that all of the people answering incorrectly are completely confident and therefore are dunning krugering. Someone who is not overconfident at 60% confidence adds the same "yes" data point that someone who is 100% confident and wrong does.

</comment></comment></comment></comment><comment><USER5> In October 2016:

* Do you think Trump will win the US Presidential election?

* Do most voters think Trump will win?

</comment><comment><USER32> Philadelphia should be the capital of Pennsylvania

<comment><USER33> Pittsburg/Philadelphia should be the capital.

<comment><USER34> There is already widespread belief, in the rest of the state, that the Philadelphia area (as the largest concentration of population and wealth) has an unfairly outsized influence over Harrisburg.

The result of moving the capital to Philadelphia would surely strain the legitimacy of the state government even more.

Moving it to Pittsburgh would accomplish the same exact thing except it would be arguably even less just.

There are any number of reasons why keeping it in Harrisburg is imperfect but it's probably the least bad solution.

<comment><USER33> No, I mean really Pittsburg/Philadelphia, like Minneapolis/St. Paul, Seattle/Tacoma, or Raleigh/Durham.

Yes, the distance between respective city centers is farther than those others, but that makes it more inclusive.</comment></comment></comment></comment>


In Superstar Cities, the Rich Get Richer, and They Get Amazon - <AUTHOR> https://www.nytimes.com/2018/11/07/upshot/in-superstar-cities-the-rich-get-richer-and-they-get-amazon.html <comment><USER2> Company opens office in location with skilled workers.

What an article</comment>


Why Silicon Valley Works - <AUTHOR> http://blog.samaltman.com/why-silicon-valley-works <comment><USER3> The premise here is that Silicon Valley "works". What do you mean by "works"?

If you mean creating artificial wealth for a small group of already-rich folk, yes, it works. If you mean transforming one of the most naturally advantaged landscapes on earth into a traffic-choked sea of concrete where people do little more than work at creating new ways to buy and sell stuff, yes, it works. If you mean polluting what was once the most democratic communications vehicle available with social drivel and advertising, it works.

If you mean creating innovations that truly improve humanity, the Valley extremely inefficient. If you mean demonstrating that humans don't need a pleasant natural environment to survive, the Valley is on the right track.

The one thing the Valley works very well at is self-congratulation, especially among the wealthy.

<comment><USER4> >truly improve humanity, the Valley extremely inefficient

I think Intel, Google and Facebook have improved humanity though your tastes may vary and I enjoy Apple's shiny stuff. Those have a market cap of $1.4 tr and it's hard to think of anywhere else with similar output. Maybe like democracy it's the worse possible system apart from all the other ones?

<comment><USER5> Two of those, Google and Facebook, have made it their core business to prime humanity for a lifetime of exploitation of their privacy in exchange for so relatively very little. How the f is that "improving humanity"?

<comment><USER6> My life is quite a bit better because of both Google and Facebook, and I would say a significant portion of the amount of time I spend online touches one of those two entities. I really don't care that much about the privacy compromises for the level I use either.

My experience is at least somewhat similar for at least a half billion to billion other people, so I'd say their impact on humanity is "pretty large".

<comment><USER7> Oh come on.

It's not like we wouldn't have basically the same world without Google, Facebook, or virtually any other SV company.

People are just so full of themselves here.

</comment></comment></comment></comment></comment><comment><USER8> Two sentences early on are problematic:

"One of the biggest misconceptions about us is that you need to have pre- existing connections to get value from the network. Remarkably, you don't. "

If Sam had said "you need them in Silicon Valley much less than anywhere else" it would be a true statement. But in an article I strongly agree with, this sort of thing... I'm not even sure what to call it, misinformation? Makes the rest ring hollow.

The networks here have vastly lower costs of entry and way fewer secret handshakes. But they still have them, and for many groups of people those costs are much higher than the median. It would have been easy to acknowledge that without diminishing the core point.

<comment><USER9> I dunno - he explains his reasoning in the same paragraph: _" It's standard practice to ask people you've just met for help – and as long as you aren't annoying about it, they usually don't mind."_

<comment><USER10> The problem, I imagine, this creates is thousands of requests for help from innocently ignorant founders to a single valuable connection in SV. Each founder may only make one naive request, but important individuals are bombarded by enough to exhaust them. I admire the patience of those who continue to offer thoughtful advice and feedback, but I imagine it's quite tiring. I think resources like Sama's free startup class at Standford are a great solution.

</comment></comment></comment><comment><USER11> I love the way people in the Startup community talk about how New York is oriented around finance. It is totally true, from the perspective of a certain group of people. New York also has a history and cultural identity that puts monetary success and business above other concerns that can arguably be traced back to its founding by Dutch traders as New Amsterdam.

That said, New York is only oriented culturally around finance from the perspective of a tiny subgroup of self-important people. The overwhelming majority of people living, working, and creating in NYC do not feel that "finance" dominates the culture, the connections, or the emphasis.

New York is the dominant hub (or close to it) for several genres of music, for television, for fashion, for theater, for many counter-culture political movements, textiles ... the list goes on and on and it will be boring if I continue to explain.

Basically, it's the kind of thing you can say if you really know nothing about New York.

<comment><USER12> Thank you for this. I absolutely can't stand the comparison between the two environments. According to NYC's labor statistics [1], Finance as a whole (including all of commercial and investment banking, insurance, and real estate) makes up about 12.5% of the city's private sector jobs, and employs about as many people as the NYC local government does. For comparison, the Education and Health Services sector employs almost twice as many people (~24%), and Leisure and Hospitality about the same (11.4%).

Its much harder to find good statistics on the Bay Area because its comprised of so many different municipalities, but I would bet the Tech industry employs way more than 12.5% of private sector employees there. There's also no real "runner-up" in the region; Finance is _one_ of the industries here, Tech is _the_ industry in the Bay Area.

[1] [http://labor.ny.gov/stats/nyc/index.shtm](http://labor.ny.gov/stats/nyc/index.shtm)

</comment></comment><comment><USER13> As a software developer not-in-SV, life is great. I don't make a shit-ton (~90) but my spouse also works (~50), (and we are both young) we have no debt, our insurance is fantastic, our commute is about 15 minutes each, our house is 1/5th the cost of SV, and we are in a very good neighborhood (meaning public school district is better than any private school unless you're going for ivy prep), and shopping for practically anything is about 5 minutes away. Oh and neither of us work more than 40 hours a week.

So... yeah. There is actual life outside of SV.

</comment><comment><USER14> From

[http://www.avc.com/a_vc/2013/02/venture-capital- returns.html...](http://www.avc.com/a_vc/2013/02/venture-capital- returns.html#disqus_thread)

at least on average, there is some serious question if Silicon Valley "works" at all well.

For

> One of the biggest misconceptions about us is that you need to have pre- > existing connections to get value from the network. Remarkably, you don't.

Well, as I recall, Sam, from one of your lectures in your current course at Stanford, both Marc Andreesseen and Ron Conway made it quite clear that they no longer take proposals from "over the transom" and, instead, learn about new investments only from their "network". Sounds quite _closed_ and not very "open" and, instead, like an _echo chamber_ from the past with not a very good view of the future.

Or, NSF, NIH, and DARPA will take proposals from strangers, but not Sand Hill Road.

<comment><USER15> I'd be interested to see a comparison of something like "dollars of revenue at ten years of age per dollar of initial investment" between Silicon Valley startups and newly formed companies of other kinds. That seems like a reasonable measure of the actual worth of a company to me, and so a decent test of whether Silicon Valley works or not.

Looking at valuations or even market caps is much less useful, because that tells you more about what's going on in investors' minds than in reality.

<comment><USER14> You have a point, but in this context necessarily _works_ has to mean making money for the limited partners (LPs) and, there, more money than the alternatives, e.g., an index fund of the NYSE. Well, follow the link I gave to AVC.com and Fred Wilson's table of data there and have to conclude that on average information technology venture capital, and, thus, nearly necessarily also Silicon Valley, doesn't _work_ very well.

I did mention NSF, etc.: There, Ike wanted pictures. The U-2 was not getting it done. So, one day Lockheed's Kelly Johnson walked in with a pile of papers and proposed a new plane, fly at 80,000+ feet, at Mach 3+, for 2000+ miles without refueling, and essentially impossible to shoot down. The project was approved; the plane flew as planned; and we got the pictures.

So, Kelly had a very ambitious _start-up_ , and delivered as planned, with the project approved from work essentially just on paper.

Such work can be done. Other cases include the F-117, GPS, passive sonar, Keyhole (essentially a Hubble but before Hubble and aimed at earth instead of space).

It really is possible to do accurate evaluations of ambitious projects just on paper and, then, go forward with a quite high _batting average_. And this fact does partly respond to your point.

Well, NSF, NIH, DARPA, CIA, DoE, etc. can work this way, but Silicon Valley just flatly refuses to. Instead, they want another SnapChat with _traction_ and f'get about anything more, and that means that they can't do the really valuable projects. And that's why, on average, they don't make the big bucks.

</comment></comment></comment><comment><USER16> My biggest concern with Silicon Valley is that it will become so expensive that many potential startups are prevented from coming here. Sam also stated that housing costs were the biggest weakness.

I wonder what YC could do about this problem for it's companies. I think it would be interesting for YC to build a few high-rise live-work spaces for smaller YC companies to foster and grow the tightly-knit community. Maybe like a high-tech college campus?

<comment><USER17> > build a few high-rise

The dominant political view in the area is that building high-rise anything anywhere is something that should be stopped at all costs

<comment><USER18> True in much of Silicon Valley (especially places like Palo Alto, which want to retain a suburban layout), but there's some moderate densification going on in San Jose. It lulled for a bit during the financial crisis because a bunch of projects' financing fell through—some apartments under construction had work stopped for ~2 years. But construction seems to be back in swing again now.

</comment></comment></comment><comment><USER19> > One of the biggest misconceptions about us is that you need to have pre- > existing connections to get value from the network. Remarkably, you don't. > Silicon Valley is a community of outsiders that have come together. If you > build something good, people will help you. It's standard practice to ask > people you've just met for help – and as long as you aren't annoying about > it, they usually don't mind.

There's some truth to this, but it's not entirely accurate. The spectrum of who, what, when, where, why and how is very broad here. The top of the food chain in Silicon Valley is just as insular as Wall Street, Hollywood, etc.

> Yes, the approximately 700 YC companies are all totally independent legal > entities – but the connection is so strong that alumni companies get > significant benefits from each other.

Not often discussed: some of those benefits are benefits until they're not. A lot of accelerator-backed startups count other startups from the same accelerator as their largest customer segment and a source of early sales traction. When the business cycle turns and "other startups" is no longer a viable customer segment, however, a lot of startups are going to see their sales decimated because they have little to no diversification.

> Unlike other cities where people are mostly focused on cash compensation for > this year, in Silicon Valley more people talk about equity than salaries > (assuming, of course, that they can afford the wildly-out-of-control housing > costs, which is probably the biggest weakness here right now).

This is simply not true. Most angel and venture-backed startups in the Bay Area today offer early employees miniscule amounts of equity, and engineers demand market rate compensation (where market rate compensation equals "what Google or Facebook would pay me if I could get a job there").

If you're a startup offering higher equity in lieu of salary, you will have a very difficult time hiring in the Bay Area. Most engineers are hip to the fact that this isn't likely to work out in their favor.

</comment><comment><USER20> Good article. It provoked a thought:

In an increasingly digital world, how long will physical proximity be an advantage?

The notion that a region could somehow have its own brand of patriotism centered around startup success merely due to density is somewhat difficult to rationalize. You can't deny SV's success relative to other technology hubs, but I'm wondering how far in the future physical proximity will continue to carry such a steep advantage

<comment><USER21> > how long will physical proximity be an advantage

There is a number of articles and research papers on the subject. In short, distance matters even within the same building.

</comment></comment><comment><USER22> No.

The reason why Silicon Valley works is because of the intense greed that it provokes amongst everyone that comes here. Maybe "greed" is too strong of a word, because people aren't necessarily selfish, maybe "money-focused" is probably more accurate. And because anyone can come here and become a millionaire/billionaire, it draws in an immense number of hard-working people which just generally increase the level of competence, and unfortunately competitiveness, which has a positive feedback loop which causes these smart people to want to work even harder over time.

Outside Silicon Valley, all you hear about is billionaires being made day- after-day, and smart people they look at this nonsense and think "I can do that". Once the come to Silicon Valley, they get sucked into a vortex of work and money-focus that is so strong that the only way to get ahead is by working long hours and taking risks.

Think about it this way: a person who makes 120k/yr with 2 kids and a spouse who stays at home will be considered the "working poor" here. I feel sorry for anyone who tells me they have a spouse, 2 kids and a house in any place other than NYC, and are contemplating moving to the Bay Area. Everything is priced perfectly to the point where no one can afford to live in a good neighborhood with a good commute with good schools. It's like the CAP theorum, except it's more like "Good Job, Good Neighborhood, Good commute: pick one" (unless you're rich).

So, in order to get ahead, both spouses need to work, and they need to work very good jobs with long hours in order to "make it well". If you have Dual Income No Kids, it's a bit easier, but you still need to deal with oppressive traffic and high rent. It's so expensive, that all you do is think about money. And you fall into fallacies where if both spouses make #120k/yr, for a total of basically $250k/yr, you start thinking "why don't I feel rich?" or "why am I working so hard with nothing to show for it?" It's because you work 10 hr days, and then you're spending 2 hrs a day commuting, and you get into a zone where all you do is think about work, and how to buy the next iPhone because at least you have something to show for your money. Meanwhile, none of my friends outside of SV have iPhone 6's let along iPhone 5S's, many are still using iPhone 4's which are perfectly good but a bit slow, but you wouldn't be caught dead in SV with one of those relics.

So all of this money-focus necessarily squeezes out as much productivity from the best and smartest in Silicon Valley, which is why you have so much success here. If you compare SV to place like Japan or China, where the population is generally better educated, you don't get anywhere near the same level of productivity. In Japan you get high prices and long work, but there's no payoff like there is with Silicon Valley because of the social impedances, so you'll never have a Silicon Valley in Japan. In other places that purport to be another SV, even places like Austin, there just isn't the same level of greed/money focus like you have here, so I don't think you'll be able to get as much productivity. It's the sky-high prices, work-life imbalance, the startup-millionaire stories that create this unique pressure cooker environment that can't be duplicated.

<comment><USER23> All this is part of the reason why I've strongly been thinking of going back to school to become a doctor. What follows is mostly a rant, just to be upfront about it.

It's a really radical transition, but I really, really dislike the city- centric focus of software engineering. I hate the idea of being a contractor (I just want stable, life-long work), and I can't tell you how tired I am of crippling traffic causing me not to even want to drop by the grocery store until it's almost time for it to close. I'm not in the Bay Area, but am in one of the five largest cities in the US. I went to college not giving a shit about money or how much of it I would make, and now I find myself thinking about it most days because I have to confront the fact that I can't afford any housing near where I work. I wouldn't be so upset about it if it weren't for the traffic which makes commuting from far out an absolute nightmare.

In terms of raw numbers, it's a transition that doesn't really make sense, not for a relatively skilled developer anyway. The salaries aren't that much higher outside of surgery and a few ultra-competitive specialties, and the time and debt involved in the training are immense. Depending on any number of factors, it could be better or worse than a life in technology. The two things that are for sure are that you're not going to have financial difficulties and you're not going to be subjected to ageism (quite the opposite). If you want to be _rich_ , medicine's probably a pretty bad choice; but if you want stable, well-paid, life-long, rewarding employment wherever you want in the country, it's among the very best options.

The most amazing part of it is that you get to directly help people every day (so you don't have to wonder if your crazy ass startup is actually "changing the world" or not), there's _more_ demand for doctors in rural areas (not less), and most non-surgical specialties can easily work comfortably into their mid-60s (or even later, health permitting). You're also surrounded by people who, at least in the beginning, were motivated out of passion and not money- and status-seeking. (Of course, some people will change as they age and grow more successful, but not all.)

<comment><USER24> Uh - remote? I've spent most of my professional life working from home.

With some extra effort, you can earn an SV salary but live in an area that has a very un-SV cost of living.

Traffic isn't an issue either.

You can network remotely. You can build companies remotely. You can even find investors remotely.

A few people are doing this already. More and more will in future.

(Are we _really_ supposed to believe it's always better to build Internet companies in meatspace? Hmmm.)

I know there are downsides to remote working. Guess what? There are epic, catastrophic downsides to working in SV meatspace too.

I'm looking forward to the day a few decades from now when what's left of SV gets turned into a museum, while the rest of the planet gets on with doing cool fun stuff... elsewhere.

<comment><USER25> Honest question: how does one get started working remotely? I understand that it's easy to keep finding remote work once you've already done a few remote gigs. But how do you break in?

<comment><USER26> Either you apply to a remote job - there are plenty here on the monthly "who's hiring" threads. Or you get your current company to allow it as part of negotiations. An annual review makes a good setting - ask for a big raise, then offer working from home as an alternative (it'll save me money on my commute, so I won't need the increase).

</comment></comment></comment></comment></comment><comment><USER27> This is only tangentially related to "Why Silicon Valley Works", but am I the only person who is frustrated by "famous" VC's saying that "founders should stop worrying about their valuation"?

I can't get over this. This is like a used car salesman telling you not to worry about the cost of the car, just to remember that there are free weather mats included!

I feel that there is a connection here as to "why Silicon Valley" works from the perspective of VC's in a position to influence founders. However I'm not feeling particularly eloquent today and haven't fully formed these ideas.

<comment><USER28> I thought about the valuation problem for a little bit when my company finished raising its Series B, and I came to the conclusion of not worrying about the valuation independent of hearing the quotes, as an engineer. Here's the important point - the valuation means little until an exit. Any valuation beforehand is only a step in that direction. All work needs to be done to maximize the value of the company in preparation for that day - anything along the way is a distraction from that end goal.

Granted, it is a necessary distraction for many of these companies, since they require VC funding in order to expand operations.

</comment></comment><comment><USER29> >> The focus in Silicon Valley on long-term compensation is also important. Nearly everyone wants to get rich but they're willing to wait to do so. Conspicuous consumption isn't that cool; not too many people drive Ferraris or talk about their vacation homes.

Instead, they talk about how healthy they are, their diets, or the climbing gym.

<comment><USER30> I'm no fitness master myself, but, far from begrudging the paleo-crossfit- organicos out there on their latest PR (am I using the lingo right?), I'd like to say, from those of us nodding and smiling:

Being healthy is great! Keep it up! :)

<comment><USER31> I think that the point is that it is the new status-symbol.

In my experience (not Silicon Valley), status symbols in the form of material possessions have taken a back seat to status symbols in the form of fitness and "experiential materialism" (frequent travelling, exotic and varied experiences).

<comment><USER30> I don't disagree with the thrust of what you are saying, and I am particularly amused that in the age of almost infinite variation deliverable through the internet there is a kind of odd counter-signalling effect to avoiding that (I also like travel and have been doing it constantly for more than a year, though, so what do I know).

Describing physical fitness as "the new status-symbol" seems pretty bizarre to me, though. As a status symbol it probably predates humanity.

<comment><USER29> Physical fitness itself isn't a status symbol, per se. How people conduct their personal physical fitness is what has elevated it to a status symbol. Things like when, where, how often, how public, what activities/exercises, and even what they wear to work... that's the new status symbol.

A ridiculous recent example is the Trulia co-founder rowing to Hawaii with his wife for the cause of...

wait for it....

the fight against sugar.

<comment><USER32> Thanks for mentioning this I hadn't heard of it! Sorry, I think it's awesome, and not because I care what other people eat.

[http://www.fatchancerow.org/blog/2014/8/4/zero-miles-to- go-d...](http://www.fatchancerow.org/blog/2014/8/4/zero-miles-to-go-divorce- papers-still-untouched)

</comment></comment></comment></comment></comment></comment><comment><USER33> Sam misses some hugely critical pieces, for example:

1\. Good IT infrastructure

2\. Plurality of high tech companies' headquarters

3\. Proximity to major technology focused academic institutions

4\. Legal infrastructure that enables innovation

All discussed previously here:

[http://www.enterpriseirregulars.com/25271/silicon- valley%E2%...](http://www.enterpriseirregulars.com/25271/silicon- valley%E2%80%99s-defining-characteristics/)

and here:

[http://papers.ssrn.com/sol3/papers.cfm?abstract_id=124508](http://papers.ssrn.com/sol3/papers.cfm?abstract_id=124508)

</comment><comment><USER34> Another reason I think the valley works so well is the audience its news outlets have. If you have a great product but can't get news coverage on it, its going to be hard to get growth without a large marketing budget. However, if you're friends with a friend of a Techcrunch write, its much easier to get covered and start to get a core group of users who love your product - and are tech savvy enough to understand that features that don't exist yet will be coming, and they should be patient.

If you try to bootstrap a startup without using SV's media outlets (as would be the case for people who don't have connections to these folks), you're going to have a much harder time.

<comment><USER35> Is asking tech journalists you've just met to write about your startup considered annoying?

<comment><USER36> All depends on the "how". If you make the effort to show that you understand their coverage niche, and make sure that you actually fit into it, and also have (or can package) your story as something newsworthy then go for it.

</comment></comment></comment><comment><USER37> I think the physical location is kind of a fallacy. For example, there are so many super smart and talented developers in DC working for government contractors because they may not be exposed to the startup option. Most people now have to move to Silicon Valley to focus on long term compensation, but often there is a stronger force of their local network (friend, family & significant others) who keep them stationed near by. Having quality startups in new geographical areas is probably the key ingredient to bringing in talent and capital, not the other way around.

<comment><USER38> > so many super smart and talented developers in DC

As one of those developers working in DC I can tell without a doubt that the majority of us do it because we get paid a large amount of money.

I make more here than I ever could in Silicon Valley. Plus the cost of living is around half. Last I checked a one bedroom in SF was around $4k per month. I pay just over $2.5k here in DC (with a parking spot!).

</comment></comment><comment><USER39> Interesting to hear YC classes framed as "we help each other." I thought there was a fair bit of internal fighting to end up as a "featured startup" from the batch. Conversely, my startup just went through the Hackers and Founders accelerator, where each startup is given equity in the other startups in the class. This is a great incentive to help the other founders in your incubator.

</comment><comment><USER40> Throw-away account for obvious reasons:

> "Silicon Valley works because there is such a high density of people working > on start-ups and they are inclined to help each other"

Bull shit. With all due respect to Sam--who no doubt knows what he's doing and saying and has become very successful--he is undoubtedly living in a bubble of his own. While I've found people generally open to the idea of chatting over a coffee, I've run across _very_ few people that go out of their way to help. In fact, I more often find people who will either offer to help, and don't, or are so self-absorbed that they can't stop talking about how great/impressive they or their startup is without bothering to ask about you or what you're doing. I've even asked for a meeting with and gotten the following response: "I'm very busy. Let me know how you think you can help me and I'll think about it."

I've been trying to tell myself that it's just my luck: That it's not socio- economic or pedigree based (race or school) or that some people here don't have basic people skills and don't know how to carry on a conversation. That maybe it's just the luck of the draw that I run into more people that I'd classify as "ass holes" than not, but I'm coming--very late--to a sad reality: it _is_ about socio-economic status--even more than that, it _is_ about pedigree (yes, both race and the school you went to). It _is_ about manners-- most people don't have them. AND, it's about gender as well! I'm not a woman, but I'm willing to bet that if I was a woman, things would be a shit ton harder for me than they already are. I'm not a 20-something white male that went to Stanford and it's impossible for me to imagine how hard things would be if I had a vagina [0].

These are things we don't want to talk about. Thank god a conversation about gender is taking place, but there's another conversation about elitism and race that hasn't even started yet.

> "One of the biggest misconceptions about us is that you need to have pre- > existing connections to get value from the network."

Complete and utter bull. Maybe in Sam's world, sure! White? 20-something? No- kids? Harvard/Stanford? B2C? Let's chat!

Early-40's/Late-30's? Not-white? Married w/ kids? Not-Harvard/Stanford? B2B? Nope. (Don't bother to apply to YC either).

The reason Silicon Valley works is because there's a new-boys _network_ that helps out those that check most of the right boxes. But here's the good news: There are other networks. Networks of people in their 40's. Networks of people that are from a specific ethnic group. Networks of people that went to the same (non-Harvard/Stanford) business school as you that _are_ willing to help out. Are most of these networks as big and powerful as the circles Sam roles in? Absolutely not. But are they out there? Yes. One of the marvels of Silicon Valley is that because there are so many people out here, there's a semi- decent chance you might find someone in one of these networks that will meet you for a coffee, offer to help, and try to make something happen for you. Is it likely? No, but possible, yes? And that—the possibility of something working out despite the odds—is what entrepreneurship is about. [0] - <USER2> It's Like Raising Money As A Woman In Silicon Valley ([http://www.forbes.com/sites/jeffbercovici/2014/08/07/what- it...](http://www.forbes.com/sites/jeffbercovici/2014/08/07/what-its-like- raising-money-as-a-woman-in-tech/))

<comment><USER41> > Complete and utter bull. Maybe in Sam's world, sure! White? 20-something? > No-kids? Harvard/Stanford? B2C? Let's chat! > Early-40's/Late-30's? Not- > white? Married w/ kids? Not-Harvard/Stanford? B2B? Nope. (Don't bother to > apply to YC either).

YC does not discriminate in this way. We fund lots of people working on B2B and enterprise ideas, who are married with kids, who are in their 30's, 40's and yes older too. This claim is just plain false.

<comment><USER38> While that may be true to a certain extent I'd be willing to bet a sizable amount of money that the majority of founders that YC funds are in fact white, male, hailing from a wealthy family, graduated from a prestigious college, and are under 30 years old.

And while you or other YC partners may defend that fact with the fallacy that it's just a numbers game, as in those are the majority of applicants, I can't help but remind you that age discrimination and class warfare is rampant in the valley.

</comment></comment></comment><comment><USER42> "not too many people drive Ferraris or talk about their vacation homes" unless you're in Palo Alto, Atherton, Woodside, Saratoga, Los Gatos, Los Altos, etc.

That's probably why there's been a geographic shift to SF.

</comment><comment><USER43> It does work if you are a VC. Tons of talented folks living in dorm like conditions (some living out their car), working long hours, in the hopes of striking it rich one day. Most fail. Some succeed proped by blogs owned by VCs. It is a nice machinery. The world mostly gets another music discovery app or another ad targeting company.

</comment><comment><USER44> This is a pretty thoughtful and to the point article. You might summarize it by saying that SV success is defined by Community and Capital. It makes a lot of sense that rich networks backed by money make an area successful. It would be interesting to explore the effect of capital from within the community vs external capital inflow.

</comment><comment><USER45> > Conspicuous consumption isn't that cool; not too many people drive Ferraris

99.9% sure I saw Sam pulling out of De Anza College at SUS 2014 in a Lotus Elise.

</comment><comment><USER46> "I wrote this for the FT, but it's behind a paywall. Since I wrote it, I feel like it's probably ok for me to post it here."

Well legally it's either ok and the contract (written or otherwise) allows you to post a copy or it doesn't.

</comment><comment><USER47> 'This seems like the future to me – large groups of independent companies, loosely tied together.'

This could also work against YC (Or many startups in the valley) very easily. I mean all these companies are being created in a time of easy money, what happens when the money dries out? Will they all fail? Since they are so correlated with each other's success.

<comment><USER48> _Will they all fail?_

Not all of them, no, but a lot will.

</comment></comment><comment><USER49> "Nearly everyone wants to get rich but they're willing to wait to do so.

... A focus on making a lot of money in the long term at the expense of short- term opportunities is essential to building companies that have a huge impact – they take a long time."

Sounds like a good advice for CEOs and politicians in DC. Focusing on the long term goals...

</comment><comment><USER50> The Silicon Valley works because of the toxic culture of "work until you die". [1]

[1] I grew up there. Never going back.

</comment><comment><USER51> I would like to ask a.radical question. Is Silicon Valley actually working? Because if you throw a lot of money at something, even if it is bad, will make it better. Given the amount of money banks an investors are throwing at the valkey, is it successful given the output?

</comment><comment><USER52> It strikes me, as do a few other posts and comments from YC that this is an incredibly American centric view of the world. I've heard PG say "just consider moving here" but without a visa as a startup that's very difficult.

</comment><comment><USER53> For anyone interested, I have compiled many of Sam's interviews at [http://newslines.org/sam-altman/](http://newslines.org/sam-altman/)

</comment><comment><USER54> Another weakness of the valley is the lack of a visa. We would have atleast 5 times the number of startups, if there was an easy way to remain in the united states and start a company for immigrants.

</comment><comment><USER55> 'I wrote this for the FT, but it's behind a paywall. Since I wrote it, I feel like it's probably ok for me to post it here.'

If the FT paid you to write it, I feel like it's probably not ok to post it where you'd like.

<comment><USER56> they didn't pay me.

<comment><USER57> get paid</comment></comment></comment>


The 100-Year-Old Man Who Lives in the Future - <AUTHOR> http://www.bloomberg.com/news/articles/2016-06-10/the-100-year-old-man-who-lives-in-the-future <comment><USER2> I'd like to see more details regarding what exactly his vision consists of.</comment>


The selfish scientist's guide to preprint posting - <AUTHOR> https://thewinnower.com/papers/3674-the-selfish-scientist-s-guide-to-preprint-posting <comment><USER2> The key issue is getting funding. The funding agencies like the NIH need to set the rules.

Proposal: Funding agencies should require publishing in open access pre-print servers and ban publishing with for-profit publishers like Elsevier.

If I have to post to pre-print, worry about others scooping me with what I post there while I'm waiting for the regular journal to publish it, why bother? We need the funding agencies to accept the pre-print servers as the endgame, and not just another thing we have to do. Otherwise it's a non- starter.</comment>


FCC Chair Genachowski declares himself Top Cop of the Internet - <AUTHOR> http://www.reason.com/news/show/136278.html

<comment><USER2> A Washington power grab at its finest. The FCC has announced that it, and it alone, is ruler of the internet, although of course the announcement was couched in terms of "openness". It will be interesting to see how this plays out in the courts against First Amendment issues, when they finally collide, as they will.</comment>


Screaming Fast Galois Field Arithmetic Using Intel SIMD Instructions (2013) - <AUTHOR> http://web.eecs.utk.edu/~plank/plank/papers/FAST-2013-GF.html <comment><USER2> At a glance, this seems like a clear explanation of using standard SIMD instructions to solve the problem, but I think the landscape has changed since this was written such that there are now better approaches.

In 2010, Intel released processors with the a dedicated instruction for "packed carry-less multiplication": [https://software.intel.com/en- us/articles/intel-carry-less-m...](https://software.intel.com/en- us/articles/intel-carry-less-multiplication-instruction-and-its-usage-for- computing-the-gcm-mode). Unfortunately, the early implementations (through Sandy Bridge) were slow, and could be beaten by combining other SIMD operations as shown in this paper.

With the Haswell generation released in 2013, though, PCLMULQDQ got much faster. Instead of being able to complete one instruction every 8 cycles, it became possible to finish one every 2 cycles (inverse throughput went from 8 to 2). This paper 2015 paper "Faster 64-bit universal hashing using carry-less multiplications" shows the difference this makes: [https://arxiv.org/pdf/1503.03465.pdf](https://arxiv.org/pdf/1503.03465.pdf)

If you are looking for an explanation of how the problem could be solved with the basic building blocks of SIMD, the 2013 Plank, Greenan, Miller paper might be a good resource. But if you are hoping to implement high performance solution for modern processors, the 2015 Lemire and Kaser paper is probably a better starting point.

(This is with the caveat that I don't actually understand the theory or terminology of Galois fields, and maybe there is something about applying it to Erasure Coding that makes the faster PCLMULQDQ approach inapplicable.)

<comment><USER3> Erasure codes normally deal with fairly small fields (e.g. F(2^8)); which might reduce the usefulness of the carryless multiply instruction.

<comment><USER4> This sounds potentially useful for things like GCM too, would it be more helpful there?

<comment><USER3> AFAICT, the carryless multiply instruction was pretty much added for GCM's benefit.

</comment></comment></comment></comment><comment><USER5> Klaus Post did an implementation of this in Go using with the relevant bits in SSE3 assembler: [https://github.com/klauspost/reedsolomon](https://github.com/klauspost/reedsolomon)

He references the paper in the amd64 code blob: [https://github.com/klauspost/reedsolomon/blob/master/galois_...](https://github.com/klauspost/reedsolomon/blob/master/galois_amd64.s)

</comment><comment><USER6> Intel's ISA-L ended up implementing this method. Their implementation is interesting because they took this further and took advantage of knowledge of the instruction latency to pipeline multiple iterations of this method to achieve some really amazing throughput.

For reference, see [https://01.org/intel%C2%AE-storage-acceleration-library- open...](https://01.org/intel%C2%AE-storage-acceleration-library-open-source- version) and the source code at [https://github.com/01org/isa-l](https://github.com/01org/isa-l) (see the erasure code folder for details).

In general, I've found Prof. Plank's other papers and presentations very interesting, innovative, and accessible.

</comment><comment><USER7> Looks like this technique is covered by a patent claim by a 3rd party? (See the link on that page to download their software.)

<comment><USER8> [https://www.techdirt.com/articles/20141115/07113529155/paten...](https://www.techdirt.com/articles/20141115/07113529155/patent- troll-kills-open-source-project-speeding-up-computation-erasure-codes.shtml)

<comment><USER9> Time to file an ex-parte review/challenge.

<comment><USER10> The founder, president, and CEO of StreamScale, Michael H. Anderson, seems to have 'over a dozen granted patents in the storage field', some listed here:

[http://www.streamscale.com/cgi- bin/complex2/showPage.plx?pid...](http://www.streamscale.com/cgi- bin/complex2/showPage.plx?pid=34)

</comment></comment></comment></comment><comment><USER11> Is there a TL;DR of how much faster Reed Solomon codes are with this?

</comment><comment><USER12> Needs a (2013) tag.

</comment><comment><USER13> What is a Galois Field used for?

<comment><USER14> Reading the abstract will answer your question.

<comment><USER15> So:

"Galois Field arithmetic forms the basis of Reed-Solomon and other erasure coding techniques to protect storage systems from failures. Most implementations of Galois Field arithmetic rely on multiplication tables or discrete logarithms to perform this operation. However, the advent of 128-bit instructions, such as Intel's Streaming SIMD Extensions, allows us to perform Galois Field arithmetic much faster. This short paper details how to leverage these instructions for various field sizes, and demonstrates the significant performance improvements on commodity microprocessors. The techniques that we describe are available as open source software."</comment></comment></comment>


As a student, my startup achieved great feats with ZERO experience. Here is how - <AUTHOR> At Drexel University, I co-founded Boost Linguistics, an NLP startup. We achieved great feats for students with ZERO experience. Among them, making it into Inc. Magazine's coolest college startups, raising over $50k non-dilutive capital, and the opportunity to pitch Dorm Room Fund (FirstRound Capital)& Rough Draft Ventures (General Catalyst).
I look to share my knowledge with the students searching for answers. Over the next 15 weeks I'll be publishing posts on the topics below. I hope the information shared helps student founders that are balancing class, social life, and the dream of creating a successful company.
LIST:
1. How to raise non-dilutive capital with $0 and no experience. (LINKED) https://medium.com/@Jefnwk/how-to-raise-startup-capital-as-a-student-80dd26b79deb
2. How to successfully balance student and founder life. (LINKED) https://medium.com/@Jefnwk/a-guide-to-becoming-a-successful-student-founder-b01dbb88474d
3. How to know if being a founder is right for you. (LINKED) https://medium.com/@Jefnwk/the-startup-founder-illusion-1fc9a9e69934
4. Getting free legal and accounting aide.
5. How to conduct validation research for an idea while in class.
6. How to find other founders.
7. How to operate a business while in school.
8. How to get free office space.
9. How to get advisors for your business.
10. How to get a personal mentor.
11. Using your .edu email domain to contact anyone.
12. The misconception about high tech startups.
13. Does major matter?
14. Advice from current founders.
15. Advice from older founders.
16. What early investors are looking for.
17. Questions I would ask myself as a student.
Posts will be on Medium & LinkedIn.
https://medium.com/@Jefnwk/
I have Drexel University to thank for guiding me through these experiences. <comment><USER2> I don't know if these are actually great feats. Are you sure you've achieved enough success to be giving others advice?</comment>