$1,790 a month and 'no cooking' allowed: Welcome to renting in the Bay Area - <AUTHOR> https://www.sfgate.com/expensive-san-francisco/article/rent-room-no-cooking-kitchen-law-habitability-13691981.php <comment><USER2> My first 18 months in the Bay Area were in a $1600 spare room with no cooking allowed. My mental and physical health both tanked. Things are better now, but when the time comes to leave my current job, I want out. The cons severely outweigh the pros if (like me) you are starting from next-to-nothing, have no family to keep you here, and you don't find your personal identity through your job. I would recommend working literally anywhere else.

<comment><USER3> It definitely takes it's toll, I had a one bedroom near "Oracle Park" and while I can't complain about it's quality or location, you knowingly paid for it. 8 Years ago It was around $3500 and I was frustrated and decided to get out, I looked up the price now, around $6k. It's just crazy.

Then there was the process of getting the place, Print out credit score in advance, show up on the spot (hoping to get in on the early rental showing, prior to the publicized showing) and be prepared to put down 1st and last months rent and 1 month deposit then and there.

It's simply not feasible for most americans to do that. Even the tech-oriented jobs aren't all paying the crazy rates you read about. The fact that the Department of Housing and Urban Development now states that $82k a year puts a single person in "Low Income" is just preposterous. As ahead of the curve as the city likes to project itself on social issues, it has a lot of do as I say not as I do attitudes among the NIMBY crowd.

I think it's turned it into a situation of in your early 20's, go to SF/Bay and experience it, then in your thirties start looking elsewhere for quality of life.

</comment></comment><comment><USER4> When I was in the military this is how our barracks were. They closed the on base kitchen and gave us a food stipend instead. Our rooms were the size of a large bedroom, shared with 2 people and we had a mini fridge and a microwave in the room.

Still managed to get pretty big working out and eating a diet that consisted of meats you could microwave (a lot of precooked sausage and can chili haha), we definitely didn't make SF money so there wasn't a lot of eating out.

</comment><comment><USER5> Some of these are AirBnb superhosts (multiple renters sharing). They want to avoid low ratings due to:

\- strong food smells

\- dirty pots and pans, spills

\- poor sharing behavior

There may also be safety/insurance/lawsuit concerns. And less profit due to appliance/plumbing repairs.

</comment><comment><USER6> Man, Silicon Valley really is the sweat shop of the IT world. Why anyone would strive to go live and work there, is beyond me.

<comment><USER7> ...you don't _have_ to rent a $1800 room.

<comment><USER8> Seriously - the Bay Area is expensive, but you can still rent an entire 3br house in the south bay for $2500 or less (within a few miles of Apple HQ, even). With a couple roommates, that's under $1000/each, and you can cook all you want.

Not everyone lives in SF.

<comment><USER7> I spend $2.7k a month and I don't even live in SF!

</comment></comment></comment></comment><comment><USER9> Toronto feels ya...

[https://www.blogto.com/real-estate- toronto/2018/10/what-2200...](https://www.blogto.com/real-estate- toronto/2018/10/what-2200-apartment-looks-toronto-right-now-2018/)

</comment><comment><USER10> When I first moved to the area with a friend I rented a fifth wheel trailer in the back of some guy's yard in Morgan Hill and my friend rented the shed outside (but he had a futon, electricity, a little heater, his computer...)

That was what we could afford at the time, until stuff took off.

I am so glad I did it though. The area is a lot better fit for me than Western PA although you give up a lot of freedoms. Definitely can't walk outside and shoot guns in your back yard :)

Pretty much I pay to be around other smart people.

I'm not sure any location in the world with a high concentration of a valuable talent/skill will have cheap living expenses.

</comment><comment><AUTHOR> It's incredibly sad.

When I first moved to the Bay Area a decade ago, there were many houses made up of artists, musicians, anarchists, and punks that held poetry readings, music shows, fed people (via Food Not Bombs or similar efforts, or just by simply cooking food on regular days and offering it to anyone who happened to swing by), supported people who were struggling (Needle Exchange, mental health support groups, survivor groups), or even provided a temporary place to sleep to strangers in return for abiding by a code of conduct and doing some chores or paying some (truly) nominal fee.

Those places, some of which had been around since the dotcom days and had effectively become local institutions, quickly started disappearing as landlords started aggressively serving eviction notices. One notable example that comes to mind was Hellarity [0] -- Hellarity was somewhat exceptional as it was a squat, where most of the houses I'm thinking of weren't. But it was a pretty special place nonetheless.

I don't want to glorify these houses too much since there were issues as there inevitably will be with any large, mostly unstructured group of people. There was plenty of shady and unsavory stuff going on.

What's sad to me is the loss of diversity of human experiences, and the loss of the resources offered in spaces like this. The loss of these inordinately impacts lower income people. People escaping abusive situations. People who aren't on tech salaries or benefits. People who don't have health insurance at all.

Also: a friend of mine who rents space in an art studio in Berkeley that dozens of artists work out of was telling me that the space was just sold, and it's becoming increasingly clear the new owners aren't interested in keeping it as an art studio.

The Bay Area kind of bums me out, to say the least.

[0]: [https://www.indybay.org/newsitems/2012/03/07/18708901.php](https://www.indybay.org/newsitems/2012/03/07/18708901.php)

</comment><comment><USER11> 'No cooking' is a phrase with a lot of ins and outs. 'Heating' is only 'cooking' if the food isn't pre-cooked.

</comment><comment><USER12> This has just created a premium category for places that allows you to cook.

Things have just got crazier in rental market in Silicon Valley.

</comment><comment><USER13> [https://www.redfin.com/blog/february-2019-housing-market- tra...](https://www.redfin.com/blog/february-2019-housing-market-tracker)

Looks like home prices are down 8% since last year in SF. Maybe things will get better soon

<comment><USER14> Hopefully but probably not.. Uber, Lyft, Airbnb, Pinterest, Slack, and maybe palantir and Robinhood down the peninsula are going to have IPOs in the next year.. that's a lot of millionaires.

<comment><USER15> Sweet, you mean there's going to be a ton of studios and 1 bedroom apartments coming onto the market then?

<comment><USER16> No, now that they are millionaires, they will be able to afford a studio or one bedroom apartment. So that supply will shrink.

</comment></comment></comment></comment><comment><USER17> What the heck how could you even have a decent diet without cooking...

<comment><USER18> When I was in my early 20s living on my own in the 'big city' for the first time, eating a healthy diet didn't exactly make the top 10 list of my concerns.

</comment></comment><comment><USER19> This should be illegal.

<comment><USER20> Why? Someone probably _really_ wants to live there, if they're willing to pay that much to do it.

<comment><USER21> Someone also wants to rent out their kids as child labor, and someone will want to hire that kid. Doesn't mean it should be allowed.

<comment><USER22> Kids don't have the same level of rights as adults to enter contracts to work. We do it because we accept that child labor damages education potential.

I'm an adult, if I never cook because I like to eat out, shouldn't that be my choice? Who am I hurting by choosing not to cook?

</comment></comment></comment></comment><comment><USER20> Then don't move there.

</comment><comment><USER23> I have an anegdotal explanation of "no cooking" but don't lynch me. The is a major representation of South Asians (Pakistan, India, Bangladesh) among IT folks. They're infamous for poor sanitation and hygiene standards. Heavy use of spices, the smell of them biting into every fabric in the apartment, yellow stains of curcuma all over the place. Then cherry on the top - eating with hands while not washing them neither before, nor after the meal. I shared personally kitchen space with them during university times and that cured me for good from South Asian cuisine.

<comment><USER24> I don't see a reason the landlord should care about eating with your hands, it doesn't affect the state of the apartment.

<comment><USER25> What do you think happens to door handles that are used with unwashed hands after eating?</comment></comment></comment>


New, more effective optical fibers have semiconductive core - <AUTHOR> http://www.gizmag.com/fiber-optics-with-zinc-selenide-core/18020/

<comment><USER2> Feel free to downvote but that is an _annoying_ site.</comment>


The Most Intolerant Wins: The Dictatorship of the Small Minority - <AUTHOR> https://medium.com/@nntaled/the-most-intolerant-wins-the-dictatorship-of-the-small-minority-3f1f83ce4e15#.9169rn420 <comment><USER2> Fantastic article. Great examples, some of which I benefit from (English as a lingua franca for example) and some that are pretty horrifying.

The realm that I most notice this happening every day is "political correctness". My Facebook feed is dominated by people tripping over themselves to attack racists, sexists, religious bigots, etc. Great. But many times they go way too far IMO and crucify innocents. However, there's no way I care enough to stake my reputation on defending innocent strangers. So they continue to rack up positive feedback on their campaigns.

For example, this article is popular right now: [http://www.huffingtonpost.com/entry/top-10-most-sexist- thing...](http://www.huffingtonpost.com/entry/top-10-most-sexist-things-to- occur-at-the-2016-rio_us_57a88bfce4b0c94bd3c9fb86)

Most of those remarks are totally fine in my book and are only sexist by willful misinterpretation. For example, how is it not newsworthy that a recent mom is competing? Even the article calls it "an incredible feat!" Then Ledecky, who "swims like a man" because she is strong enough to use the traditionally male stroke, unlike all of the other women. Girls "standing around as if they're at the mall" was just an old commentator picking the most casual place he could think of. And when he tried to defend himself by pointing out the Cherry-Picking Fallacy (he has covered women's sports for many years, so they picked one sentence online that _might_ be _misconstrued_ as sexist to crucify him online when his body of work is totally fine) they actually cropped his defense and cherry-picked from _it_ as well.

Anyways. Things like that annoy me because I can't say them with my name attached or I'll end up cast aside. Nobody is interested in rational discussion, the Most Intolerant does always win.

<comment><USER3> That's a really terrible article. Sure, certain complaints are kind-of legitimate (e.g. mentioning someone is someone's wife - although I'm sure same is also often said of men married to famous women), but most just seem to be petty denials of the fact that (the best) men are almost universally better at a given sport than (the best) women.

<comment><USER2> For sure, and even the follow-up on that issue has the IMO totally reasonable explanation + apology for how something that ridiculous came out:

[http://www.wbur.org/onlyagame/2016/08/12/cogdell-unrein- twit...](http://www.wbur.org/onlyagame/2016/08/12/cogdell-unrein-twitter- olympics-trapshooting)

The Chicago-based newspaper was trying to emphasize her connection to Chicago (she's from Alaska) and overdid it. Then apologized. Woo, makes sense, no harm no foul.

<comment><USER4> I don't see any foul either.

When the captain of the Australian cricket team was going out with a famous Australian model, that was news. Even though she was overall a bit less less famous than him.

Had she being going out with some otherwise unknown olympian, that also would have been news -- especially if the dude won a medal.

</comment></comment></comment></comment><comment><USER3> I doubt the whole of Europe will ever eat Halal. Or, at least, I hope not. Personally, I find the method of slaughter cruel (slicing the throat and waiting for the blood to drain - granted, the animal can be stunned, but there were plenty of footages of animals being tortured instead), and discriminatory (has to be performed by a Muslim, Jew or Christian). I actively avoid eating Halal meat.

<comment><USER5> I actively avoid anything Islam-related (I'm in France). It may sound racist (and it may be) but France seems to be actively denying my identity as a Christian. As a side note, there are about 55% Christians and 7% Muslims in France.

We've been removing any reference to Christianity in state-related documents - which is fine but let's see some examples: Christmas holidays are renamed "End-of-year Holidays", just like Easter holidays are now "Spring holidays", and we're removing Christian days off here and there. Street names are renamed, churches aren't always maintained by Christian-adverse mayors.

On the other side, every vegetable mix is now a "tajin", often halal, the state is funding Mosques, and our streets fill up with women with the foulard (that thing to hide hair), our beaches with women in burkini, and males have Arabic dresses and beards. When there is some sand on the floor, you'd believe you were on the other side of the Mediterranean. Upon football matches, they burn the French flag.

Most leftish people insist that we should encourage this because "tolerance" (Fraternité). At the same time when they're introduced to a Christian they go on and on about priests raping kids and the Spanish inquisition. Ok, I get it, as a Christian I'm a murderer, and television headline news enjoy a lot conveying arguments against Christian representants.

I have the feeling that tolerance, mutual understanding and equality of chances only goes towards the 7% Muslims.

Same goes for women. Someone else in the thread pointed to a link which "10 sexist things", saying they didn't see them as sexist. I don't know, if there are fewer women in IT, it could also be that programming is hard and few people can live the life of a good programmer. Meanwhile all girls of my Engineering school were promoted way before men, and get a better salary than men I know, while having usually less technical knowledge. It doesn't match the usual message that feminists tell, but stats aren't built to find such results. Once again equality only strikes one way and women don't have to work as hard as men to succeed.

So how can we stop harassing the white, the Christian, the male, the hard workers? Do I really have to vote for that extremist party, or could women and Muslims have a bit of tolerance and check we still have an equality of work/results ratio even though we're white?

 _Goes back to running my own company, because I was never promoted while employed_

<comment><USER6> T'inquiète, t'es pas le seul à être frustré...

For non-French readers, it should be mentioned that the degree of political- correctness in France often exceeds that of the united states, especially in the capital.

It's quite hard to imagine for those who haven't encountered it.

Regarding feminism, I think the problem is that internet-feminism (and indeed, most forms of feminism you see in the media) represent a very specific and strange brand thereof. Namely, it's a very vocal sliver of 3rd-wave American feminism, which appart from it's mind-boggling ethnocentricity, has it's roots in a very misandrous circle of American lesbian thinkers (or rabble-rousers, depending on your perspective). I should, of course, mention that I have no quarrel with the bulk of lesbians, American or otherwise.

In particular, this group is characterized by the ascetic rejection of biology as an explanatory factor for ... well... _anything_ related to observed sex differences. Otherwise stated, they assume that all observed differences between men and women _must_ stem from social injustice -- biology can have no role whatsoever, here. To claim otherwise is quickly met with accusations of essentialism, or worse.

As an interesting aside, I'm "surprised" they haven't taken up the cause of getting more men into nursing...

Lest this be interpreted as a rant, my point is twofold:

\- The climate of political correctness in France is increasingly suffocating to the majority, and this in a climate of repeated bloodshed perpetrated by "a certain group that must not be named". The tension is palpable, and the majority (it seems to me) feels increasingly alienated, if not outright _invaded_. I'm not saying it's a fair assessment, but rather that the phenomenon exists, and will likely inform some very concrete changes in policy.

\- Feminism is not fundamentally insane and I think sexism does exist. I also think its effects are vastly overstated by the profoundly anti-intellectual demagogues that have hijacked the feminist movement. By and large, I find the intellectual and moral substance of mainstream American feminism to be appallingly weak, and frighteningly zealous.

<comment><USER7> > Regarding feminism, I think the problem is that internet-feminism (and > indeed, most forms of feminism you see in the media) represent a very > specific and strange brand thereof.

"Internet feminism" represents pretty much the entire range of feminism; if you disproportionately perceive some particular segment, its probably because you have a disproportionate reaction to it.

> As an interesting aside, I'm "surprised" they haven't taken up the cause of > getting more men into nursing...

I'm surprised you think that they _haven 't_ taken up the issue of men in nursing , since its been a fairly common theme in feminist circles (including on the internet) for quite a number of years, including especially the idea that the lack of men in the field is directly tied to the same gender stereotyping that leads to disrespect for the field as "women's work". (And, frequently, this discussion also treats it as a failing of early waves of feminism that this issue was neglected in the past.)

See, among others:

[http://feminismnursing.blogspot.com/](http://feminismnursing.blogspot.com/)

[http://minoritynurse.com/rethinking-gender-stereotypes-in- nu...](http://minoritynurse.com/rethinking-gender-stereotypes-in-nursing/)

[http://www.ncbi.nlm.nih.gov/pubmed/15238127](http://www.ncbi.nlm.nih.gov/pubmed/15238127)

<comment><USER6> >"Internet feminism" represents pretty much the entire range of feminism; if you disproportionately perceive some particular segment, its probably because you have a disproportionate reaction to it.

Please stop playing with words, and please stop the thinly-veiled accusations.

Yes, the full range of feminism is represented on the internet, but far from proportionally. Some groups are distinctly louder, more zealous, and receive more "air-time" than others.

>I'm surprised you think that they haven't taken up the issue of men in nursing , since its been a fairly common theme in feminist circles (including on the internet) for quite a number of years

Indeed, though not in third-wave revolutionary circles that are the object of my criticism.

Please understand you're conversing with a feminist, and please understand that most people _would_ be surprised by the (very valid) examples you bring up. This speaks to my original point: a very thin and radical sliver of feminism is over-represented on the internet and in the media, which in turn hurts our cause.

<comment><USER7> > Please stop playing with words, and please stop the thinly-veiled > accusations.

I'm not playing with words, and the accusation was direct and not veiled at all. Your description that internet feminism represents only the particular narrow subset that you claim is ridiculously unconnected with reality and only explainable by either a complete ignorance of the domain (which you deny, and I believe you), outright dishonesty (which I don't suspect, and, as the least- charitable interpretation, I wouldn't suggest even if I had suspicions in that direction), or the common human trait of magnifying the significance of things to which one has a particularly strong negative reaction.

</comment></comment></comment></comment></comment></comment><comment><USER8> > _Should a society that has elected to be tolerant be intolerant about > intolerance? (...) Yes, an intolerant minority can control and destroy > democracy. (...) So, we need to be more than intolerant with some intolerant > minorities._

This has been tried during the French Revolution, one of its slogans being "pas de liberté pour les ennemis de la liberté" (no freedom for the enemies of freedom).

It led to Terror and mass executions, so I'm really not sure it's the right path.

<comment><USER6> Provocative counter-argument: the revolution undoubtedly benefited future French generations immensely, and led to (arguably) one of the most successful and tolerant nations in the West. Maybe it was necessary to purge the enemies of freedom?

French revolution aside (the Terror was, indeed, a dark period I would not like to see repeated), there is an argument to be made for:

\- the lesser of two evils

\- the necessity of cracking eggs to make an omelet

</comment></comment><comment><USER9> Fascinating article, although probably not for the reasons Taleb would like.

He's signaling something, but I am too much of a plebe to know what:

" _a glass of that type of yellow sugared water with citric acid people sometimes call lemonade_ "

This reverses his majority/minority argument:

" _Let us apply the rule to domains where it can get entertaining:_

" _An honest person will never commit criminal acts but a criminal will readily engage in legal acts._ "

This argument is pretty nonsensical; if the majority is flexible, then in what sense does it make sense to say "submit"?

" _...then the (flexible) majority will have to submit to the minority rule._ "

Taleb seems to need to work on his cut-n-paste skills:

" _Note that these slaughter rules_ [for halal/kosher meat] _are skin-in-the- game driven, inherited from the ancient Eastern Mediterranean [discussed in Chapter] Greek and Semitic practice to only worship the gods if one has skin in the game, sacrifice meat to the divinity, and eat what's left._ "

I'll just note this in passing:

" _In promoting genetically modified food via all manner of lobbying, purchasing of congressmen, and overt scientific propaganda (with smear campaigns against such persons as yours truly), the big agricultural companies foolishly believed that all they needed was to win the majority. No, you idiots. As I said, your snap "scientific" judgment is too naive in these type of decisions...It is strange, once again, to see Big Ag who spent hundreds of millions of dollars on research cum smear campaigns, with hundreds of these scientists who think of themselves as more intelligent than the rest of the population, miss such an elementary point about asymmetric choices._ "

Could someone run through this with the math? This is not a use of renormalization I am familiar with.

" _The method of analysis employed here is called renormalization group, a powerful apparatus in mathematical physics that allows us to see how things scale up (or down). Let us examine it next –without mathematics._ "

Signaling again: " _flâneur_ ".

Are we talking about the same French language?

" _...the prestige of France or the efforts of their civil servants in promoting their more or less beautiful Latinized and logically spelled language over the orthographically confusing one of trans-Channel meat-pie eaters._ "

<comment><USER10> I actually found the article interesting, but you are right that it would improve much if he worked a bit more on these parts. I stopped at exactly the same points.

As to the last point French is a bit more logically spelled than English - even if it is still horrendous.

</comment></comment><comment><USER11> Why aren't the LGBTQXYZ movements given as a recent example? A tiny group of people (that is disproportionately wealthy and powerful) managed to slide the West from something unthinkable (gay marriage, same sex attraction) into legalization and acceptance of the phenomenon as normal. Granted, we should distinguish between those with persistent same sex attraction and gays i.e. those who embrace the so-called "gay lifestyle" and express support for its acceptance and legal recognition. The left has a history of such manipulation. Saul Alinsky is famous for it. Soros is a contemporary symbol of it.

<comment><USER12> Regarding same sex attraction, the fact that you never thought of it before doesn't make the concept "unthinkable". History begs to differ with that label.

Also, you shouldn't use "i.e." to equate those who _are_ LGBT to those who _support_ their being accepted into society and legally recognized _despite_ not being LGBT themselves.

</comment></comment><comment><USER13> [https://en.wikipedia.org/wiki/Nassim_Nicholas_Taleb](https://en.wikipedia.org/wiki/Nassim_Nicholas_Taleb) is freak'n hilarious. If you have a chance to see him in person, I highly recommend it. [0]

Preprints [https://arxiv.org/a/taleb_n_1.html](https://arxiv.org/a/taleb_n_1.html)

[0] [http://www.fooledbyrandomness.com/scheduledseminars.htm](http://www.fooledbyrandomness.com/scheduledseminars.htm)

<comment><USER14> He is one of my favorite authors, who does not feed you pop-psychology or some feel good narrative. I understand people here are turned off by his seeming arrogance, but I like one quote (probably by someone else), "Character trumps reputation." and Reputation is fragile. This one quote gives me more courage than any to speak my mind.

<comment><USER9> But there is nothing _but_ pop psychology in this article. Well, and some paranoia and grammar errors.

And did you check the end? I think he decided that it didn't feel good enough.

</comment></comment></comment><comment><AUTHOR> Question to moderator. I thought HN allowed dupes as long as community found them interesting. Now the post is marked "[dupe]". Did it change?

<comment><USER15> It's always better to email the mods, because they often don't see these questions if they're posted in threads.

I'm guessing the [dupe] tag is a mistake.

</comment></comment><comment><USER16> So the article talks about intolerant minorities taking over and we have the comments section here taken over by intolerant minorities pushing their agendas.

I feel like this is an elaborate joke.

</comment><comment><USER17> His use of the intolerant-minority principle to address the challenge faced by tolerant societies is very interesting.

<comment><USER18> I'm a little confused as to how the West is committing suicide as he puts it. Allowing the sale of halal meat? What a joke compared to the old communist threat.

</comment></comment><comment> r-w > Aramaic is a Semitic language which succeeded Canaanite (that is, > Phoenician-Hebrew) in the Levant and resembles Arabic; it was the language > Jesus Christ spoke. The reason it came to dominate the Levant and Egypt > isn't because of any particular imperial Semitic power or the fact that they > have interesting noses.

Well then.

<comment><AUTHOR> The author is from Lebanon himself and is probably referring to his own interesting nose in that passage.

</comment></comment><comment><USER19> This is just a rant. Why is it here?

It's very easy to advertise what's not in food if it never had it anyway. This is just good marketing. Many foods are kosher or halal or vegan or whatever because as far as the majority is concerned it's a checkbox feature that doesn't affect you and maybe even works as a signal of quality. I'm having a hard time seeing the issue.

On the other hand it seems like nearly everything having to do with nuts is labelled as processed on machines that also process peanuts so apparently economics wins much of the time.

<comment><USER2> As far as I can tell, this article isn't even angry. It's just explaining the mechanics behind some changes. How is that a rant?

<comment><USER19> You can't really call it "explaining the mechanics" when there's no evidence he actually talked to anyone or did any research to see if his theorizing holds up.

<comment><USER20> So can nothing be published unless it's cleared by experts in the field? This isn't even in a scientific journal - it's a Medium blog post. I'm not sure why you're attempting to hold it to the rigor of a journal (which he'd have to pay for to apply to, I might add). And on the accusation of having "no research", his research is all the things he observed happening in the world in the first half of the essay.

<comment><USER19> Speculation is fine - everyone does it. But usually you admit that right up front. "I've done no research but here's my half-baked theory of how this works." You show a bit of humility, some understanding that you might not have it right the first time, and invite corrections. (Also, if it's based on what other people wrote, links help.)

</comment></comment></comment></comment></comment><comment><USER21> Previously on HN [https://news.ycombinator.com/item?id=10567630](https://news.ycombinator.com/item?id=10567630) Edit: as a pdf with a marginally different title but the same content.</comment>


Load Balancing is Impossible [video] - <AUTHOR> https://www.youtube.com/watch?v=gas2v1emubU <comment><USER4> It seems that the speaker is discussing small servers; 150 concurrent requests per server/process sounds very small to me (but maybe this is common in enterprise?); in this case, I can see why the variance introduced by the random algorithm is such a problem.

But if you have an efficient system where each server can handle 10K+ concurrent requests, then the variance introduced by the random algorithm becomes insignificant. Also, it doesn't matter that some requests take much longer than others; if you're dealing with a lot of users per server, the slow requests will be distributed evenly too and average out.

I implemented the balls-and-bins algorithm and I fond that when dealing with few buckets and a large number of balls, the variance in load between buckets became smaller. I even added randomness to the 'weight' of each request (to account for slow vs fast requests) and the distribution was still even - <USER2> more requests each server can handle, the more even the distribution is with the random algorithm.

<comment><USER5> 10K+ concurrent requests to a single server sounds like a server that is mostly shuffling bits around. I assume the 150 concurrent is more realistic for a frontend server which actually does something.

Also note the speaker is a CTO of a CDN (fast.ly), I am guessing he has experience with large concurrent requests as well :)

<comment><USER6> > 10K+ concurrent requests to a single server sounds like a server that is > mostly shuffling bits around.

Isn't that all a load balancer is supposed to do? I certainly don't want my load balancers performing computations or logic. I want it to pass that work off to another server.

<comment><USER7> If you actually intend to serve traffic back to the original requestor, you can't have too many requests handled per server because you have a limited amount of bandwidth. 150 requests per server on a 1Gbps pipe (which is what you get on, for instance, a single ELB instance) only actually gives you about 813 KB/s per request for 150 requests/server. For 10,000 requests you'd be down to 12 KB/s per request. For comparison, the _average_ web page size is now up to 2048 KB!

To be clear, you can do a lot better with a better pipe, smart caching, compression, etc. But people often have horribly unrealistic estimates about how much traffic their servers can handle because they don't take bandwidth into account, and load balancers are no exception.

<comment><USER8> The HTML for an average web page is not 2048KB

<comment><USER7> The average web page size is more like 2331KB now: [http://www.httparchive.org/interesting.php?a=All&l=Apr%201%2...](http://www.httparchive.org/interesting.php?a=All&l=Apr%201%202016). (I didn't say anything about HTML, by the way).

Of course, when you break it down by individual web request, _most_ responses are still below 800KB, but you shouldn't load plan for the average case. And clearly even the average case is well above 12KB, especially for a CDN (which is responsible for serving the image, video, and large script content). I'm also pretty confident the page I linked already includes compression (which decreases size, but can increase time quite a bit; many people expect software load balancers to be using the absolute fastest compression available, but that's often not the case in my experience).

<comment><USER8> But most of those resources are static and cacheable.

<comment><USER7> Sure, if you actually set the headers correctly and the browser actually retains them (recall that browsers are free to discard their caches at any time). And, if they can't be cached on the user's computer, who caches them? That's right, the CDNs. Like fast.ly. Who wrote the article.

<comment><USER8> Regardless, the small pipe on your web server isn't serving the assets.

</comment></comment></comment></comment></comment></comment></comment></comment></comment><comment><USER9> One of the approaches and the last Tyler, the speaker, introduced in the talk to improve the fat tail distribution of load balancing latency problem was Join-Idle-Queue algorithm[1] that came out of UIUC Extreme Computing Group, Microsoft Research and Azure in 2011.

And according to Tyler's data (fast.ly CDN data since he's the CTO there), it beats all the current load balancing algorithms including Join-Shortest-Queue. Six years later, I wonder:

a) was there any other novel / better algorithms tops JIQ in LB performance?

b) has Microsoft Azure LB uses JIQ internally in their LB offerings?

c) has any open source LB software implement JIQ algorithm on the horizon? (according to Tyler, he found none.)

Can someone with expertise share some lights on these? Thanks.

[1]: [https://www.microsoft.com/en-us/research/wp- content/uploads/...](https://www.microsoft.com/en-us/research/wp- content/uploads/2011/10/idleq.pdf)

<comment><USER10> Google Scholar can come in handy here; there have been 117 papers citing JIQ and some of them may have improved on it: [https://scholar.google.com/scholar?cites=3283470738712232937...](https://scholar.google.com/scholar?cites=3283470738712232937&as_sdt=5,44&sciodt=0,44&hl=en)

<comment><USER9> I am afraid that merely a citation reference doesn't tell the whole story. I guess someone has to dig into those papers and implement them and then use real world data to test and see, right?

<comment><USER11> If anyone _does_ want to dig in to the papers, a friend and I made a discussion platform for research. I've added all the papers in Tyler's talk as well as two he recommended on Twitter to a list here: [https://www.projectcredo.com/briankung/server-load- balancing](https://www.projectcredo.com/briankung/server-load-balancing)

I have zero background in academic computer science, so it may take me a while to understand even one of these papers.

</comment></comment></comment></comment><comment><USER12> The slides are here:

[http://www.slideshare.net/Fastly/load-balancing-is- impossibl...](http://www.slideshare.net/Fastly/load-balancing-is- impossible-62833005)

Wrap up and Q&A:

[https://www.youtube.com/watch?v=gas2v1emubU&t=30m24s](https://www.youtube.com/watch?v=gas2v1emubU&t=30m24s)

The article from the slide 23:

[http://latencytipoftheday.blogspot.co.at/2014/06/latencytipo...](http://latencytipoftheday.blogspot.co.at/2014/06/latencytipoftheday- most-page-loads.html)

"#LatencyTipOfTheDay: MOST page loads will experience the 99%'lie server response"

<comment><USER13> I thought the point from that article was a bit of a stretch, as most of the requests are going to be async ad requests, or cdn requests, and hopefully none of that blocks the user experience. I'd probably give it most sessions would experience it, which is nearly as bad. overall though it was really informative and interesting

<comment><USER14> > hopefully none of that blocks the user experience

Let's hope for the best. _cough_

</comment></comment></comment><comment><USER15> Very nice. I was expecting this to ignore all the research on things like power of two choices and similar provable techniques, but the talk is actually really thorough in its research.

It's very interesting to see what experiences people have with using all of this in practice. The title seems a bit clickbaity though.

</comment><comment><USER16> Good talk, I feel like I've tread exactly this same thinking and given a similar talk internally (i accidentally ended up as the LB SME), even used poisson distribution to model the requests and do some code demos.

He references a simple algorithm/paper on simply chosing between two servers, and the results on the smoothing out of variance was impressive (paper: [https://www.eecs.harvard.edu/~michaelm/postscripts/mythesis....](https://www.eecs.harvard.edu/~michaelm/postscripts/mythesis.pdf) ).

So, no real conclusion, it's just interesting to peel back the layers of LB.. so many people treat it as magic, when really its a statistical lie - <USER3> impossible to achieve perfect distribution (hence the title)!

</comment><comment><USER17> Round robin is somewhat better than random here, because you get more uniform distribution of requests to servers; but you do have to keep a counter and if you have multiple load balancers, with few requests, there's a chance that they'll get synchronized and you'll get bursts of traffic to one server, then the next, etc.

You still end up with uneven load if the requests are not equal in load generated (essentially always), but trying to be smart (choosing the least loaded/queued server) doesn't really work (too much work on load balancer, too much latency between request picking and load measuring, etc).

</comment><comment><USER18> Imagine a beach. A beach is made of particles of different sizes, aka length. Now physics apply to this beach, as a line of "processors" that grab one particle out of the array and release it to the future beach, taking the time of length of the particle. There can be emptiness between particles.

First property of this, is "entropy" that the beach is self-sorting interaction by length, as time taken to process if close to a different sized particle.

Second property is the "light-speed" of this physical simulation. This is the maximum particle length multiplied with the number of processors.

Third property is causality, causality is the range of the furthest out particle reached by the light-speed length again multiplied by the light- speed. Obviously causality can be reduced by reducing light-speed.

Fourth property is non-interaction, if you add traversable, unmoving particles into the scheduler queues, this warps "time" as in, large particles are forced to jump over them, while small particles take free particle-size time to traverse, and as being non-interacting, are invisible to the huge particles.

Sixth property is determinism. The whole physic-simulation is precise and from a deterministic input, we can derive a deterministic output, as long as causality remains unviolated.

Now for scheduling, the data-physics are rather simplistic, as the processor- front, or "time" is not repeatedly interacting with the same beach.We also can determinate ahead of time, which processor interacts with what particle, assuming light-speed is not altered. Also note, that big particle being processed equal reduced light-speed.

Now what is optimal scheduling? Optimized for throughput, as in shoving huge particles to dedicated processors, with huge fields of non-interacting particles? Fair scheduling, as in having all particles having average particle-size time going through? Prevent a causality buildup over lights- peed?

PS: I once wrote a little visualization of these data-physics, and showed them around algorithm class- nobody ever beside the prof cared for it. Nice to meet people who are fascinated by mind games.

PS^2: You can use similar property's to compress data, via Conway's game of life into deterministic lock-step simulations. Never saw that used in the wild, as you need either pre-computated simulations to be fast, or some sort of relational operator on some simple pre-set of simple simulations.

PS^3: Sorry, got carried away. This is fascinating. Should upload the simulation to GitHub. Excuse any typos.

<comment><USER19> > Should upload the simulation to GitHub.

Sounds interesting. I'd fork it!

</comment></comment><comment><USER20> Great talk, however it is not explained the relationship between the Poisson process and the log-normal distribution.

It actually skip that explanation.

</comment><comment><USER21> Huh. It occurs to me that your brain needs to deal with a poisson process for incoming information. How does it handle it?

<comment><USER22> Dropping packets.

<comment><USER23> Lots. You don't notice every sensation of every part of skin and of every sense. You're probably only focusing on specific part of a specific sense.

</comment></comment></comment><comment><USER24> Why not just use a PID controller to load balance?

<comment><USER25> thats afaik not done because of the performance overhead.

most requests take only a fraction of a second to answer. to actually monitor threads on that frequency you'd need not only a significant portion for the monitoring (ever looked at the CPU usage with running top?) but also for the network communication to send that over to the lb, which needs to process the data for load balancing

<comment><USER24> You wouldn't need to monitor individual threads and processes. Only average response time for each server or total CPU load.

</comment></comment></comment><comment><USER26> Interesting talk and I am wondering if the op tried load-balanced algorithm like "leastresponsetime" and "leastconnection"? Today LB are pretty advanced and sometimes are even not even used in technology like ecmp.

</comment><comment><USER27> In my experience given talks given by Fastly are always worth seeing or attending.

If have the opportunity to attend a talk given by the founder Artur Bergman don't miss it.

It will be as informative as it is funny and entertaining. There are many on youtube from conferences past.

</comment><comment><USER28> For the case of _load balancing_ as in the OP, for a statement of the problem, over time, requests for work arrive at an Internet computer server farm. The requests differ in what computing resources at the server farm are needed and for each resource how much work. E.g., a request may need 10 GB of main memory and 40 GB of virtual memory paged memory, 3 TB of I/O to rotating disk, and four processor cores, all for 50 minutes.

So, at each arrival, have to assign the work to a server. When the assignment is made, we do know about about the, say, _state_ of each of the servers, say, what work has been assigned and how busy the server is.

The goal of the work is in some sense to get the best performance from the server farm for the loads. There are various candidate definitions/measures of performance, e.g., minimize the average elapsed time for the work of the requests to be done, that is, minimize the average _response_ time as seen by the users/customers.

So, the work of the load balancing is just the assignments, one work request at a time, to the servers. This assignment is under _uncertainty_ maybe about the details of each request and, so that can _plan ahead_ , also about what the next requests will be.

E.g., if we had some idea that soon we would get a request that would need all of one server, then maybe we should think ahead and have reserved an empty server for that request. If we don't reserve such a server, then to assign this request we might have to keep waiting until have an empty server and, unless make an effort to have such an empty server, the request could go unassigned for a long time thus hurting the goal.

So, we have to make decisions over time under uncertainty to achieve a goal that is an average of the results.

Now likely we have a problem in the field of applied math called _stochastic optimal control_. In our case the _control_ is the freedom we have when we make the assignments. _Stochastic_ means varying with uncertainty over time. The _stochastic_ part is the uncertainty of the next requests that arrive. The _optimal_ part is that we want to do best as possible likely on some average measure, only _average_ because we are working with uncertainty. Sure, if we don't like average, we could use median, etc.

IIRC there was some work on this problem by some especially diligent researchers at the IBM Watson lab and based in part on work of H. Kushner at the Division of Applied Mathematics at Brown University.

IIRC the good news is that those researchers made good progress. IIRC the bad news was that the computations for their solution were a bit too large to be practical! Also, from the references I saw them using, the math prerequisites for their work were a bit much!

So, maybe good work for a solution is in the literature but too difficult to use. Maybe.

My suggestion: Be practical. First cut, assign the next request for work to the least busy server. If the server farm is so busy that often all the servers are too busy for more work, then have to leave requests in a FIFO queue waiting until some server is not too busy for the first request in the queue. Collect empirical, production data on this _control_ , the FIFO queue, the response times, any problems etc. If see no significant problems, then declare the problem solved for now. Else focus on where the problems are and look for a control that also handles those problems.

This material about stochastic optimal control was heavily from R. Bellman. Other authors include W. Fleming, D. Bertsekas, S. Shreve, E. Dynkin, R. Rockafellar, R. Wets. The field has been of interest by parts of applied math (e.g., Brown), operations research, and systems and electronic engineering.

This load balancing at a server farm contains as a special case some of the topics in queuing theory.

Computer science has long been interested in the problem of _job scheduling_ on batch computer systems which, of course, is a relatively small special case of load balancing as in the OP. IIRC the usual result of the job scheduling work was that the problem was in NP-complete and otherwise in practice more work to do than the work being scheduled! Maybe now computer science will also be interested in something stochastic optimal control for load balancing.

</comment><comment><USER29> Great presentation!

</comment><comment><USER30> Pretty sure that's a donkey.

</comment><comment><USER31> SFW cause no stupid meme slides. So.. whats the conclusion of the video i did not hear?</comment>


The Future of Peer Review - <AUTHOR> http://techcrunch.com/2012/02/05/the-future-of-peer-review/

<comment><USER2> I'm all for new publishing models in academia, but this post doesn't address the fundamental difference between the web in general and academic papers: expert curation.

The web's "curation" works (to whatever degree it does) because there are thousands, if not millions of people who can rate the quality of most articles/essays/websites. But in many scientific fields, there are often only a very small number of people who would be qualified to review a particular paper (usually well under a hundred, sometimes as few as a handful).

And make no mistake, this is a numbers game -- there's the well known 100/10/1 rule for content websites (for every 100 viewers, there are 10 who rate content, and only 1 who creates content). The exact numbers vary quite a bit from website to website, but the point is that it's _orders of magnitude_ difference. That's all fine if your initial pool of people is huge, but if you start with 20 qualified people...you see the issue.

Another related point not addressed is that peer review is fairly demanding work. You're not just saying "accept" or "reject", but you're obliged to write a page or two suggesting the strong and weak points of the paper, things to improve, etc. You need some mechanism for enforcing this (even if it's via social norms), otherwise review quality deteriorates quite severely, which in turn affects paper quality.

In lieu of this article, I'd recommend seeing lectures by (or essays by) Michael Nielsen, who's thought about this problem in much more detail and (I believe) has a much better understanding of the way things work.

<http://michaelnielsen.org/blog/essays/>

<comment><AUTHOR> Thanks for your detailed thoughts.

Peer review is already evolving. Increasingly academics discover their papers via search engines (Google, Google Scholar), and social platforms (blogs, email, arXiv, Academia.edu, Twitter). The usage of these channels as discovery channels for research shows that the metrics that drive those channels are doing a good job at expert curation - i.e. surfacing good content.

On Google Scholar, it's citation counts that drive the ranking, and therefore the discovery process. This is the "crowd review" process I talked about in the post. With social platforms, it's recommendations from friends and colleagues that drive the discovery process. This is the "social review" process I talked about in the post.

In the pre-web days, people used to walk down to their libraries to read the latest edition from a particular journal. In those days, the journals did drive a lot of the discovery process. Nowadays, pretty much all discovery of research papers happens on the web, and most of the discovery is via channels such as search, and recommendations from colleagues over social & communication platforms.

You are right to think about the incentive systems that drive different kinds of peer review. In the case of the social review process, you want to share good recommendations with your friends, not bad ones (this is why Twitter and Facebook work as discovery channels). In the case of the crowd review process, you want to cite good works that your work genuinely built on, not bad work.

<comment><USER3> My understanding is that most publications are not heavily cited, and address very specific problems. I'm curious as to how you plan on creating incentives for qualified people to review the papers in their field which focus on problems outside of their present research interests.

To give a personal example: I used to work in a research group that applied techniques of a field called geometric control theory to the problem of modeling flocks of birds. There are not that many geometric control theorists in the world, and their interests range from robotics, to circuits, to modelings schools of fish. Right now, there are journals of control theory that force their reviewers to review all submissions, regardless of their personal interests.

How do you plan on emulating that? In other words, there need to be incentives for geometric control theorists who work on modeling schools of fish to spend a few days of their time grinding through the proofs of some grad student's paper in modeling bird flocks, which he needs for his thesis.

<comment><AUTHOR> Peer review often works the worst when a journal asks someone who's not that qualified to peer review a paper. See the comment below from billswift about an unbelievable case of peer review going wrong, where the reviewers weren't in the research area of the target article (you need to follow the links to get the full picture).

Something doesn't have to be heavily cited for it to start getting traction, just as a website, or blog post, doesn't have to get heavily linked for it to get some traction. A friend might find the paper and share it via Twitter, their blog, or Academia.edu; this is how the social review process works on Twitter and Facebook. It might sound slow: you might wonder 'won't it take ages for a paper to get traction like that?'. But in reality, when you have social platforms like Facebook and Twitter, content can be surfaced, and whip around the world, at incredible speeds. The journal peer review process, by contrast, is several orders of magnitude slower.

The web is known for having a long tail of content: lots of content that appeals to certain niches of people. It is part of the magic of the web's discovery channels that this long tail content gets routed to the niches who care about it - via search, Facebook etc. The fact that academics overwhelmingly use the web now for research discovery is testament to how good a job the web's discovery channels are doing at surfacing good content in highly niche areas.

<comment><USER4> As a researcher myself, I do not use the web as a means of content discovery. I do use google scholar, pubmed, and citeseer, but these are only useful because they restrict the search to just peer reviewed journals (and optionally patents in google scholar case).

Trying to use general web search is often far too noisy. Often times I find interesting papers on page two, three, or four of google scholar. In a general web search these same papers might be on a much later page, or perhaps buried so far that they don't come up at all.

As others have said, there is also much more to peer review than discovery. Peer review is additionally intended to help authors improve work that isn't quite up to standards. Academics do peer review for free because they recognize the value of it and because they are asked directly by the editors to do it. In your model, what would be the incentive to look for new papers and give reviews? How would you handle old versions of papers with mistakes or papers that are of insufficient quality? The peer review process currently filters these intermediate stages of paper writing. In your proposed model, you would potentially have many versions of the same paper that a user would then have to filter through, in addition to a very great many papers that are of very low standard.

The quality of content on the web and its curation is a _very_ poor standard to compare academic literature to. Many users are happy enough to be able to filter out obvious spam pages let alone judge quality to the level needed in academics.

<comment><AUTHOR> It's true that the papers in the indexes of Google Scholar, Pubmed, and Citeseer are peer reviewed, but what that shows me is that what is really driving the discovery process is the ranking system in those search engines: i.e. the order in which the hundreds of results for a given search query show up.

One of the drawbacks of the existing peer review process is that academics don't get credit for their reviews, nor do the reviews see the light of day for others to benefit from. I expect that there would be significantly more discussion, and reviews, of papers in the future if there was a credit system that allowed people to get credit for reviews and comments they made of papers. I think that credit system is possible and will be built.

There is an interesting question regarding the immutability of content. Right now, once you publish a paper, you can't edit it, or delete it. It's an immutable piece of content. Before the web was established, there was a line of thinking, developed by Ted Nelson, according to which the internet should evolve like that, and that a link should always work: once some content is posted, it can never be taken down. Most people are probably glad that the web developed along the lines of Tim Berners Lee's thinking, rather than Ted Nelson's, and that they are now free to edit and delete content they have posted. I think similarly people would appreciate being able to update a paper in response to a comment they have received. The author is better off, and so are subsequent readers, as they find themselves reading a more evolved and advanced paper.

<comment><USER4> Reviewers are _intentionally_ hidden from view. The reason is to prevent social or political backlash from a poor review. In fact, it is sometimes the case that papers reviews are _double blind_ , the reviews do not know the names of the authors either. This is also to prevent bias on the part of the reviewers. This secrecy is a positive and important part of the peer review system. Science should not be politically or socially biased.

The immutability of published works is also crucial. It routine for writers to leave out details covering in prior works. This saves immense amounts of time on both writers consumers of scientific works. However, it also means that it is crucial that all cited works be preserved forever. If a document truly goes missing, then entire lines of work become incomplete. Papers can and are infrequently withdrawn, but as far as I know, the work is not erased, but merely marked as bad.

The way updates or corrections are made is via newer papers revisiting topics. But it remains that at every step some amount of decent due diligence is done to correct errors and not clutter up the records with incomplete versions.

As great as the web is for unstructured content, you cannot easily apply it to every area, and especially not to scientific publications. There are plenty of other examples of curated sources on the web that crucial. Map systems, curated databases of restaurants, directories of people like LinkedIn and Facebook, and even Wikipedia can be counted a curated system due to its system of editors.

Scientists have always and are still free to share data, white papers, and whatever else outside the peer review system. The main reason peer review is still here is that no suitable alternative has ever been proposed that addresses all the points that peer review does. There _is_ a push towards open access journals to benefit the world at large though.

I understand that you are passionate about this, but I'm not convinced by your arguments.

</comment></comment></comment></comment></comment></comment></comment><comment><USER5> Summary: Current peer review is not 100%; we need to think in terms of authors' motivations (tenure); try a hacker news model, possibly layered with badges-of-approval from editorial board.

The quality of peer review is only as good as the editorial board and reviewers make it -- often this is a lower standard than we expect. This article lists several extreme examples of poor practices by a (previously-)prominent publisher:

[http://blog.mathunion.org/journals/?no_cache=1&tx_t3blog...](http://blog.mathunion.org/journals/?no_cache=1&tx_t3blog_pi1%5BblogList%5D%5BshowUid%5D=30&tx_t3blog_pi1%5BblogList%5D%5Byear%5D=2012&tx_t3blog_pi1%5BblogList%5D%5Bmonth%5D=02&tx_t3blog_pi1%5BblogList%5D%5Bday%5D=05&cHash=a2d6424f899302a7ea3b75b9bb591802)

The key to shifting academic quality control away from the present archaic model is through author's motivation. The replacement needs to be a system that improves and filters incoming papers, and at the same time earns at least as much prestige for the authors who participate.

One idea is something like a combination hacker news and stackoverflow with academic articles in place of news articles / technical questions as the object of discussion. Papers can be changed in response to comments. High scores for either a paper or an author on such a site could replace the prestige granted by acceptance into a top-tier journal. Points are awarded for reviews, and researchers are respected for their scores (closer to tenure), so there is more positive motivation to review than in today's system (which is: it's rude not to review when asked).

I think that alone could work. Some academics might balk at a lack of editorial boards. Fine, let's allow some editors in the mix. An editorial board is an aggregator. They choose what content fits their filter, and may open a discussion with an author on improvements, possibly bringing in reviewers. In the end, readers will see a prestigious badge of the board's approval on high-quality articles, and there could be a page or rss feed to browse the articles meeting this standard. This can all exist on top of the hacker news / stackoverflow system, which itself could be built on top of an arXiv-like system.

Several of these ideas are motivated by thoughts from the blogs of Tim Gowers and Michael Nielsen.

<comment><USER3> I don't think the hacker news / stackoverflow model will work for research papers. People read hacker news articles because they want to keep up with what's going on in the tech community, even if it's not super related to their particular job. Researchers generally don't just read research papers for fun. Rather, they are working on their own research, and they are looking for relevant papers whose results they can use. Hence, it's critical that when they read a paper, they know they can trust the results.

I can skim an article posted on hacker news in 30 seconds and have a general idea of what the content was. I can also skim a research paper, but only if I know that the research was done well (e.g. all the proofs are correct, or the assumptions in the model made sense, or all correct variables were properly controlled in the experiment). If that stuff isn't guaranteed, the paper is useless because I don't know if I can use it for my research. And to figure out all that stuff myself would take a few days literally out of my research. Just because 10 people skimmed it, thought it sounded cool, and up-voted it does not mean the results in the paper are correct.

Other than this review aspect, I really like this product. If they can figure out a good incentivized review system, I think it will be a big hit.

<comment><USER5> You have some good points.

Perhaps reviewers could publicly vouch for an article to give it credence. Reviewers, who are also researchers, can have scores based on their own papers. If a reviewer with a high score publicly vouches for an article, that article would receive points in favor of being correct.

The most interesting time in writing a paper is when it's under review before being accepted. The process is extremely similar to debugging an app shortly before a release. Part of my motivation for the proposed design is to upgrade this to an ongoing process -- analogous to the move from rare-release desktop software to dynamic web apps. If the author cares about their paper and receives useful comments, then the paper can iterate on that feedback.

I agree that many researchers in practice hear about papers they want to read through word of mouth. For this use case, I like to compare an online platform to stackoverflow (SO), in that you most often visit stackoverflow to solve a focused problem you have, as opposed to browsing for fun (like hacker news). You get an external problem to a page (a math paper / an SO question), you check it out, and you learn what you wanted to learn, or you contribute your knowledge or ask a question. It's a mini-community around a particular idea (a paper or a question). I see this as the main use case. The analogy with hacker news was to imply that the platform could be built around links to arXiv papers in the way HN is built around external news stories (while SO is more self-contained). Like HN, it doesn't seem necessary for an author to submit their own paper to the site.

Finally, I would say that some researchers do browse for fun. I enjoy reading several mathematicians' and computer scientists' blogs, and occasionally visit mathoverflow. This would be a secondary use of the platform, but it would be fun to have a front page of popular papers, and per-topic subpages.

</comment></comment></comment><comment><USER6> For an example of lousy peer review, I posted this yesterday on LessWrong:

A new science journal recently published a seriously crackpot paper, this has the abstract a link for the pdf. I first heard about it from Derek Lowe, who has also written two follow-up posts. The first has a couple of links discussing how news of the paper spread, while the second includes a link to the journal making excuses for why they published it.

<blockquote> Moreover, members of the Editorial Board have objected to these papers; some have resigned, and others have questioned the scientific validity of the contributions. In response I want to first state some basic facts regarding all publications in this journal. All papers are peer-reviewed, although it is often difficult to obtain expert reviewers for some of the interdisciplinary topics covered by this journal. I feel obliged to stress that although we will strive to guarantee the scientific standard of the papers published in this journal, all the responsibility for the ideas contained in the published articles rests entirely on their authors. </blockquote>

I included the links to all of Derek Lowe's posts because they have other interesting links, including in the comments.

The permalink is here [http://lesswrong.com/lw/9p9/open_thread_february_114_2012/5t...](http://lesswrong.com/lw/9p9/open_thread_february_114_2012/5trq) , so you can check out the links I included.

<comment><AUTHOR> Fascinating example of peer review not working out. It reminds me of the Sokal hoax, except that this one doesn't look like a hoax.

</comment></comment><comment><USER7> Also see the now finished LiquidPub project by the university of Trento, Springer, and others, for more ideas and views on peer review in the digital age:

<http://liquidpub.org/>

This paper about journals:

[http://wiki.liquidpub.org/mediawiki/upload/9/9b/Liquid- journ...](http://wiki.liquidpub.org/mediawiki/upload/9/9b/Liquid-journal- proposal_v0.13.pdf) (Liquid Journals)

And this paper about peer review:

<http://ubiquity.acm.org/article.cfm?id=1226695> (Publish and Perish)

Though in a way the differences between peoples descriptions and views are details mostly, as what is clear is that peer review will change, and that it will move away from publisher-control.

Imho the biggest difficulty will be the social aspect; breaking the feedback- loop between journal-publications and perceived academic credit. Because until individual scholars can escape from the social trap of journal-publishing & handing over their copyrights, without harming their careers, they will be unlikely to do so.

(<http://en.wikipedia.org/wiki/Social_trap>, and a great article on social traps: [http://www.agls.uidaho.edu/agec467/agec467/Other%20readings/...](http://www.agls.uidaho.edu/agec467/agec467/Other%20readings/Platt.PDF))

<comment><AUTHOR> Wybo - thanks for these links. I totally agree that breaking the feedback loop you mentioned, that binds academics to the current journal system, is key. I think the central things that needs to happen are: building an alternative discovery system (already happening with the web), and building an alternative credit system (happening with citation counts, and other metrics, but is lagging behind the growth in alternative discovery systems, and needs to be accelerated).

<comment><USER8> I wonder if a distributed algorithms, as used in web based currencies (eg. bitcoin) or incentive based systems (eg. bit torrent), can be adapted to the peer review system? The currency being traded would be academic credit.

</comment></comment></comment><comment><USER9> I think it is not possible to achieve the quality of a peer review by "crowd review" or "social review". While most of the people are able to rate a funny picture or an interesting article on Internet, the review of an academic paper requires much more expertise in the field and knowledge of the related work. Therefore I don't believe that a truly objective rating can be created. Also, making a peer review is time consuming because you have to really understand the paper in detail and think about if the contributions are big enough and well explained, if the experiments are done properly and if the comparisons are fare and so on. This is not an easy thing to achieve by just throwing a lot of people on it (the users of the Internet).

Furthermore, when a paper is published in a well recognized journal or conference one has a guarantee of the quality of the paper. For example when I see something published at CVPR (Conference on Computer Vision and Pattern Recognition) I know it is certainly worth reading. I don't want to imagine how much more difficult it will be to find the valuable papers among all the bad stuff that is being submitted for review and rejected.

I don't want to say that the peer review process is perfect, but if I have to choose between "peer review" and "crowd review" I choose the former.

<comment><AUTHOR> Academics have always based reading decisions on recommendations from colleagues. This trend has accelerated since email and the web were invented, as your colleagues can communicate with you through more channels, and more recommendations go back and forth. Social review always has been a big part of research discovery, and it's become more important with the advent of the web.

Search engines play a huge role in driving research discovery, and their ranking systems are fundamentally based on crowd review - calculations using links, as in the case of Google and Bing, or citations, as in the case of Google Scholar.

One might wonder in theory whether a search engine could do a good job of ranking academic papers, or whether recommendations from one's colleagues is a viable way of discovering papers. But when you look, in practice, into how academics operate day to day, those channels are the ones that academics increasingly use to discover research.

</comment></comment><comment><USER10> It's also worth noting that academia.edu is a for-profit enterprise. The domain academia.edu was purchased prior to the strictures put in place to ensure that only qualified educational institutions could register .edu domains, and has been grandfathered ever since. Academia.edu itself launched in 2008 apparently.

The author of this piece very much has a horse in this game, and as a result I very much view his writing with a skeptical eye.

There are serious and substantial problems with social models of publishing, especially when organizations/institutions start tying promotions and/or remuneration to metrics like impact factor (which itself is a proprietary formula).

We should keep in mind to reward good science because it's good science, not just because it's popular.

<comment><AUTHOR> I think you are getting the horse and the cart in the wrong order (to continue the horse analogies :) ). I started Academia.edu, after finishing my PhD at Oxford, because I believe that science, is too closed, and too slow, and that I think that we can build a faster, cheaper, and better system for disseminating research. You're wondering whether I'm asserting the things in the post because I started the company, but really it's the other way round. I started the company because I believe the philosophy expressed in the post. One has to be the change one wants to see in the world.

<comment><USER10> I'm sad to see that someone downvoted factual information provided about you and the post you wrote.

And I'm not asserting anything about your motivations for why you started a for-profit company. I agree with you that the publishing models we have currently are broken. At the same time, I'm not entirely happy with some of the changes that have taken place in the past 5-10 years.

When you look to the changes that digital distribution have wrought in industries like the news media over a similar time span, the changes are not all healthy ones either from the perspective of publishers or consumers. The fact that HuffPo and Gawker have risen to the top of the pile as the sorts of money making enterprises is not such a great example to set.

Moreover, I'm unconvinced that adding more eyeballs to papers is going to improve quality of review. If your assertion is that we can make scientific review faster and cheaper without a subsequent loss of quality (let's just say "more scalable"), then I'd love to hear your response to a claim i like to make. I'd assert that publishing scientific papers is more similar to the trials and travails of online media than it is to publishing source code.

The thing that makes open source software work is that software behavior (broadly speaking) is verifiable. I can run your code and we can agree on the results, or you can tell me i'm an idiot for not configuring the software appropriately. I'd assert that FOSS scalability derives from this ability. More eyeballs helps because everyone is working from/over the same artifact.

The same can't be said for most academic papers, because most academes don't even publish their raw data, let alone possess have the facilities or the will to publish verifiable algorithmic processes for arriving at their data.

I'd love to see a project which encouraged better replicability and verification of results/test data. But you haven't suggested any of those things.

P.S. absolutely, please do be the change you're looking for. But I'm not convinced that you're the change _i'm_ looking for.

</comment></comment></comment><comment><USER11> _The drawbacks of the Two Person review process are that it is [...] expensive: $8 billion a year is spent on subscriptions to journals, which is money that could be spent on more research._

None of that money goes to the two reviewers. The reviewers could do the same work for open access journals, and they would have (IMHO) no motivation not to do it for free.

</comment><comment><USER12> Am I the only one that finds it ironic that TechCrunch is writing about peer review?</comment>


MDXC: A tool to create React components from Markdown - <AUTHOR> https://github.com/jamesknelson/mdxc <comment><USER2> this could be very useful for some applications.</comment>