Half of England is owned by less than 1% of the population - <AUTHOR> https://www.theguardian.com/money/2019/apr/17/who-owns-england-thousand-secret-landowners-author <comment><USER2> Whoever has will be given more, and they will have an abundance. Whoever does not have, even what they have will be taken from them

Matthew 25:29

This reminds me of the power law that Peter Thiel talks about. He says that in any given field, a few people will dominate and the rest will get nothing. In Theil's example, one or two companies will dominate a given market while everyone else scrambles after a small piece of the pie.

But this seems to apply to other areas too. For example, a small group of composers wrote most of the repertoire of classical music that's played, while the rest of the composers are forgotten and hardly played. And of those composers, it's only a small number of their works that is widely played.

It seems to be a rule of the world that distribution is unequal, a small group of people do everything and the rest do almost nothing. A small group of people get everything, the rest get almost nothing.

<comment><USER3> Pareto Principle.

</comment></comment><comment><USER4> Not really surprised. Farmers are less than 1% of the population and most land is farm land.

<comment><USER5> The subtitle of tfa:

> Research by author reveals corporations and aristocrats are the biggest > landowners

<comment><USER6> Among the corporations how many are agriculture corporations?

</comment></comment></comment><comment><USER7> Appalling really. It's almost as if those Georgists are onto something.

<comment><USER8> Could you imagine wanting to rename every male and female "George"? Those Georgists are once more on their throne, one bony hand clutching their terrible rods.

<comment><USER7> ?? Not talking about the Kings George I-VI, but [https://en.wikipedia.org/wiki/Georgism](https://en.wikipedia.org/wiki/Georgism)

<comment><USER8> I was making an absurdist joke. I double-checked the style guidelines and you are allowed to make absurdist jokes.</comment></comment></comment></comment>


I'm looking for a co-founder (UK) - <AUTHOR>

I'm a 30 year old web developer from the UK (north west).  For the past two months, I've been building "an ebay for time", similar to taskrabbit.com and zaarly.com where users post small jobs and tasks and others bid to complete them.  The bulk of the coding is done - I still need to add payment integration and geolocation.
This is my first web startup, but I've run two successful companies in the past. One wrote shareware for the (original) Palm OS platform.  The other consisted of three computer retail/repair stores here in the North West.  Both companies were eventually sold (at profit) as I wanted to try other things.
I'm looking for someone with excellent UI and design skills to join as co-founder.  Knowledge of html and css is essential.  You would be responsible for all aspects of design including logo, branding and of course the web pages.  I'm now working on this full time and I'm looking for someone who can commit at least 20 hours a week.  I would strongly prefer to work with someone from the UK but I'm happy to hear from anyone who might be interested (email in profile). <comment><USER2> @Bazookoid I've been thinking of something similar for the Japanese market as well. So I would be interested in a collaboration..

Some questions: 1\. What is the technology stack you chose to develop in? 2\. Have you integrated i18n into the code base? 3\. Will the whole or part of the code be open sourced to Github? (or similar) 4\. Does this have a mobile component? Smartphone and/or feature phones? Or are you expecting to go to html5 based web apps for mobile? Responsive design?

<comment><AUTHOR> 1) Standard LAMP stack + codeigniter

2) Yes, there is full i18n support. The beta will initially launch in the UK, Italy and Spain (see later comments for why).

3) I have no plans to open source the code at present

4) There's a very strong mobile component. I have a contractor building iphone and android apps using Rhodes (<http://rhomobile.com/products/rhodes/>). The site itself is html5 and mobile friendly, but a mobile app will be required for anyone undertaking a task

When I said it's similar to taskrabbit and zaarly I didn't mean to say it's a clone. It's similar to those sites the same way airbnb is similar to ebay or HN is similar to reddit - the underlying concept is similar but the implementation is very different.

I'm not sure which market you're thinking of targeting but my site is focused on a niche in the travel sector, hence the three-country launch. I'm trying to solve a specific problem which will make use of the mobile camera, mic and sensors while a task is being carried out, hence the native mobile apps. It's intended for people already working in the travel industry so I'm not sure how relevant the code would be to your idea but it could easily be updated to a different market (with or without the mobile component). I'd be happy to discuss specifics by email.

</comment></comment><comment><USER3> First of all, HN isn't a place to find a co-founder for your start-up. You can't seamlisly find a co-founder. Co-founders are people you may have known for a while, such as colleagues, roomates, friends, etc.

Second of all, if you end up hiring a UI designer, they won't work for free, given the reason 80% of startups fail, just because you had 2 successful start-ups from the past, doesn't mean you will have a third start-up that will succeed. So in this case, you will have to pay them enough so they can work with you, and I'm sure they'll ask for the same amount they currently get paid for.

Finally, if you know your start-up will be a clone, in this case it obviously is, you know theirs going to be an issue. You're gonna have to come up with better ways to make money as a start-up, and market your users, which is strategically challenging, because these two start-ups (I've interviewed both founders) are launching worldwide in less than two months. Zaarly will hit the grounds first, soon TaskRabbit will follow, so my best advice would be that if you could tell us one good reason why your start-up will succeed, I'll give you props, other than that, it's undoubtful that your start-up will succeed.

<comment><USER4> You say that as if launching a two-sided marketplace worldwide is a trivial thing. In the market in question you have to fight for dominance in each city individually having to attract both workers and customer.

Even if Zaarly and TaskRabbit launched in the UK tomorrow it would still take years for anyone to become established. And the fact is that both of them have only penetrated a handful of American cities at the moment, it would be crazy for them to pursue a tough international growth plan when they've barely penetrated their home market.

If you look at a history of two-sided market startups (job boards, classified listing, auctions, daily deals) what happens in most cases is that you end up with a large number of local players who end up merging or being acquired to form a global player.

<comment><USER3> Even if Zaarly takes time to establish it's grounds in the UK, a clone is a clone. You have to have a better idea than whats out in the market, you can't create a Groupon clone and expect for it to rocket (unless you have an amazing deal that offers $100 of Food for $2) but in reality, that is impossible.

Zaarly already uses Geolocation. I'm here to help, if you can provide us with mockups, or designs of what your start-up will offer to it's users, we can improve the stability, and provide harsh feedback for it to succeed, and take on those competitors.

<comment><USER4> Tell that to SoSasta (India), Groupoer (Israel), Twangoo (South Africa), Disdus (Indonesia), ClanDescuento (Chile), CityDeal (Europe), GroupMore (Malaysia), etc.

All local Groupon clones which were acquired by Groupon for multimillion dollar amounts.

<comment><USER3> I'm sure the OP doesn't want to get his start-up acquired, well atleast that is what I'm hoping, to beat the guys in the market he needs to provide some key features that can set him aside from his competitors.

Why won't people go out and create Groupon Clones and sell them to Groupon for millions of dollars? \- Because Groupon has a key acquiring strategy they use, and a lot of other start-ups use. They look at the key features that Groupon doesn't have, then they look at their market and get term sheets ready.

<comment><USER4> Acquisition is the primary exit route of startups, only a very very tiny minority of startups going the IPO route.

Despite me giving a list of Groupon clones who Groupon have acquired you seem to have missed the point. Groupon enters new markets by acquiring the leading player in the local market, it's absolutely not about features but rather about market share.</comment></comment></comment></comment></comment></comment>


Audio First: Spotify Acquires Gimlet and Anchor - <AUTHOR> https://newsroom.spotify.com/2019-02-06/audio-first/ <comment><USER2> If you haven't listened to it, check out the "Startup" podcast (specifically the first season and all subsequent episodes that focus on Gimlet). The synopsis is that the host (Alex Blumberg) quits his successful job at NPR to found a podcasting company, and simultaneously starts the podcast in order to tell his own story approximately as it happens. It's fascinating to observe the genesis of the company through the eyes of its founder, starting with him pitching the idea to VCs, then hiring its first employees, and eventually growing the company to well over 100 people (and now, getting acquired by Spotify). It's a really unique and intimate perspective that would definitely appeal to the HN audience.</comment>


Freeman Dyson: The Question of Global Warming - <AUTHOR> http://www.nybooks.com/articles/21494

<comment><USER2> Somewhere in between the raving Al Gores of the world and the equally raving "No, global warming is totally impossible!" folks is a sparsely-populated region of sensible opinion. It seems like Freeman Dyson is somewhere in this region, but very few others are -- at least, very few voices who can actually get heard in the media.

The basic problem is this: it's pretty clear that if we add extra CO2 to the atmosphere, the planet will get warmer. And it's also clear that if the planet gets warmer, then there will be bad effects. Thus we really should be able to come up with some sort of dollars-per-ton value (let's call it K) for exactly how much harm is caused by CO2 emissions and work from there, perhaps applying some sort of tax to properly take into account the externalities of CO2 emissions.

The problem is that sensible estimates for the value of K vary by many orders of magnitude, and values which are _not_ sensible but are nonetheless widely implicitly accepted vary by an additional bunch of orders of magnitude on either end. First, we have the uncertainty of exactly how much CO2 causes exactly how much warming. Secondly, we have the uncertainty of how much warming causes how much harm -- again, several orders of magnitude in the sensible estimates.

And that's without getting into the additional problems of expressing total worldwide harm in dollars (if we make Siberia into great farmland but drown Tonga is that a net loss or a net gain?) which I'm not gonna even talk about.

Now, if we could all accept that there's a genuine question to be answered about how big K is, and that there are still enormous error bars involved in determining this quantity, then we might be able to get down to having the sensible and important discussion which we as a species really need to be having at this point in history.

Unfortunately, the majority of the voices we hear nowadays are coming from the extremes -- either the Al Gores and Ted Turners of the world who want us to believe that K is huge, or the folks who want us to believe that K is zero. (Actually I'm not sure that the K = 0 folks really exist in large numbers, but they're a convenient strawman for those who believe K is huge.)

I don't know what the solution is, but as usual it would help if the media would hire a few more people who actually had some clue about the way science works.

</comment><comment><USER3> Oh, now it is clear why any other interpretation of the global warming is met with name calling and personal attacks (mostly trying to engineer some link between the opponent and the oil industry). It's because we are dealing with a religion. Beliefs are not to be questioned by definition. That's OK, but only up as long as the believers do not try to shut the scientists up. Which sadly is commonplace nowadays.

Thankfully, it seems a topic like this can at least be _discussed_ on HN. I've seen some scientists' discenting views published in mainstream British press... The resulting outcry was not pretty.

<comment><AUTHOR> It's the same in Canada. I heard a phone-in show on CBC radio where a climatologist with unorthodox views was subjected to hysterical, nasty personal attacks like nothing I'd observed before. (Among which, you're right, he was accused of taking bribes from the oil industry.) Meanwhile his arguments were never addressed. Which, of course, doesn't mean his arguments were correct. It does say something about what Dyson calls secular religion, though.

Edit: I just remembered something funny that's related. I went with my son's class on a field trip to the planetarium once. Instead of the nice star show everyone was expecting, the kids were shown a film about global warming. That wasn't so bad, except that it was the worst kind of propaganda: millions of people are going to die in flooding and other natural disasters, millions more will die of malaria, all kinds of cute animals are going to die, etc, etc... _unless you kids grow up to do something about it_. How low do you have to stoop to propagandize grade-school children by scaring the crap out of them?! Anyway, the film ended with some advice about turning off dripping faucets (I swear)! The lights were turned on and we all looked at each other in stunned silence. Then one hilarious child said, quite loudly, "Do as we say, or hurricanes will destroy your house!" The entire audience cracked up. We laughed it out of our system and moved on happily to the next event. I swear that kid has a future in show business...

<comment><USER3> I think it's a kind of natural selection for kids. Surviving the peer pressure (and authority pressure). That should be OK education for some of them, not for the majority though.

Funny, I was exposed to similar bullshit growing up in the communist USSR. All propaganda forced on people was always justified by the most of humane reasons. Do you want to support the oppressed of the world? The starving? The exploited? Of course we were!

It was only later in life you start to notice some fat smirking person sitting in the shadows and enjoying the fruits of that mass enthusiasm... and mass scarifie. Some of the kids I new have noticed the falsehood early on, but decided to play along. Most were clueless and later shocked when the whole communist system collapsed around them. And a rare few had chosen to simply stand for the truth and face the consequences.

</comment></comment></comment><comment><USER4> I thought it ironical that Freeman Dyson has been very sceptical to computer models of the atmosphere:

"The real world is muddy and messy and full of things that we do not yet understand. It is much easier for a scientist to sit in an air-conditioned building and run computer models [than to do empirical research]" -- <http://www.edge.org/3rd_culture/dysonf07/dysonf07_index.html>

but in this article seems to think that the economists have their models down cold, at least when they support his viewpoint.

<comment><AUTHOR> The same thought occurred to me, with reference to exactly the same earlier piece. Economic computer models surely suffer from most of the problems that the climate models do. I kept expecting him to repeat the same critique and was a little surprised when he didn't. I'm willing to cut him a little more slack than you are, though; it's true that he might just be changing his tune to suit a predefined viewpoint, but I can think of other interpretations as well. Hopefully he'll write more about it and we can see.

</comment></comment><comment><USER5> Freeman Dyson's brilliant, as usual. His analysis of what the wiggle in the Keeling graph means has profound implications for global warming policy.

<comment><USER3> I was thinking how you define brilliance. The problem with arguing such a complex topic is that in order to make a point you need to understand and take into your system the whole long chain of assumptions and conclusions that your opponent has made. Was your model right? Let me explain my model... Which is normally more than an average person is prepared for.

Hence, the power of the soundbite.

Freeman Dyson (and others quoted earlier on HN) however wins over the complexity not through unsubstantiated soundbite, but by arguing a simple logical point. CO2 is not a 'pollution' in itself. It's not an alien element on Earth. Plants put all of the carbon through their system 'every 12 years' on average. Plants are not alien. Forest fires and volcanic eruptions can produce carbon, and there is a ready system to 'digest' it. Plants.

That's something of a radical shift in the thinking. All that smoke from burning coal is not pollution???? Hm, taking the thesis to the extreme -- actually no. It's just taking one natural element from the soil and placing it in the atmosphere. Nature does exactly the same thing all the time (forest fires, eruptions etc.).

Obviously, even if we assume that increase of CO2 in the atmosphere is not a disaster, it may still be unpleasant because the change would affect the current status quo. Siberia will become fertile and Arkansas will turn into desert. This change should be prevented or at least managed. But not by the government austerity measures, but by gradual and natural means.

These simple ideas are powerful. They don't require complex arguing over whose weather model is better. Hopefully they will spread.

<comment><USER6> I don't get the "natural" part of the argument. If I carve a spear out of wood and put it through your heart, it will still kill you, even though it is all natural. Clearly it is what you do with it that matters...

I think earth has been terraformed by organisms several times, like there used to be not so much oxygen floating around? But I think the organisms that released the oxygen died after the terraforming. The same could happen to us... Not saying it will, just warning against some sort of Gaia theory that nature will fix everything to our benefit. Nature doesn't really care about us.

<comment><USER5> His point is that the CO2 is not pollution in the same way that plastic in the Pacific gyre or PCBs in Love Canal are pollution. The excess CO2 is like the water from a flood caused by humans.

When you want to prevent floods, you need to find technologies and techniques that work with the laws of physics, natural systems, and economic forces. If you're going to build below sea level, what is the engineering required to compensate? Does the cost/benefit pan out? Likewise, if you're going to push the climate towards warming, what is the engineering required to compensate? Does the cost/benefit pan out? This is what Freeman Dyson and the authors he is reviewing are asking.

Nature doesn't care about us. But we humans have harnessed nature. Dyson is suggesting that we could do that once again to get us out of the current predicament.

<comment><USER6> I understand the economics viewpoint, I just don't see how being natural makes a difference for CO2 compared to, say, plastic. Sooner or later there will probably be organisms that can "digest" plastics, too.

<comment><USER5> Soon, there may well be bacteria that produce it. That would blur the lines a whole lot.

The difference is in the fact that there's been a Carbon Cycle for ages, just like there's been a water cycle for ages. Hydroelectric dams use energy in the water cycle. Gravity driven irrigation canals do as well. The point is that there's already a natural system out there that's capable of capturing huge amounts of carbon.

<comment><USER6> Sure, but ultimately it just boils down to "some miraculous future technology will fix it", biotech or whatever.

I think I read recently that the plants that were turned into oil actually are from a time when there was no process in nature yet that could process the bound carbon (similar to plastics today). So it was never part of the natural cycle.

I thought more about the proposed "super-carbon-harbouring trees", but really, how should that work? Any plant that does that would have to be more efficient than existing plants, meaning it would replace the natural plants. Dangerous game to play? (Those genetically engineered plants would definitely replace existing plants, if only because humans would plant massive amounts of them).

<comment><USER5> How are "super-carbon-harbouring-trees" any more dangerous than "super-flesh- growing-bovines"?

Remember that transportation is less than 1/5th of the US's use of petroleum. Since petroleum will be getting more and more scarce, something has to become the new chemical feedstock. Plants are an obvious choice.

<comment><USER6> I think it takes a lot of energy to raise "super-flesh-growing-bovines", the trees would have to work in a different way. Not sure what you mean by chemical feedstock.

Also, bovines are not popular in the CO2 scare-opinions.

Not saying that the trees couldn't work, but so far they don't exist, so they are nothing but a miraculous cure. Thinking about it, their mentioning makes that article lose a lot of authority in my opinion.

<comment><USER5> North America was overgrown with "super-flesh-growing-bovines" until white men up and shot them all.

Miracle game-changing technologies have been endemic in the history of human civilization. In fact, our current level of population wouldn't even be possible without a bunch of these happening. (From the "Green Revolution" of the 60's back to the development of the plough.)

<comment><USER6> Sure, but to dismiss all problems with a reference to a future miracle technology??

</comment></comment></comment></comment></comment></comment></comment></comment></comment></comment></comment><comment><USER7> There are some good startup ideas in here. Carbon-eating trees would be fun to make.

<comment><USER8> I think carbon-eating trees are only part of the solution. We already _have_ carbon-eating trees; all that coal and oil didn't just come from dinosaurs. The problem is that the output of trees is so useful, we don't have to willpower to cut them down (or even just rake up all the leaves each fall), store them in a cave somewhere, and forget about them.

We really need the carbon in a form that doesn't decompose and is more useful than wood. Diamond would be great...

<comment><USER4> We also already have genetically engineered carbon eating plants. Most of our agricultural crops have been genetically engineered (using selective breeding or molecular biology) to be much better at getting CO2 out of the atmosphere to produce carbohydrates than their wild counterparts. Unfortunately it seems that taking CO2 out of the atmosphere is like taking pee out of a swimming- pool: it's possible, but takes a lot of energy. Most crops compensate the increased energy cost for carbon uptake by reducing their defences against pests and droughts, which means they need to be subsidised with pesticides and irrigation (today produced mainly using fossil fuels).

It would be great if we could design trees that catch more CO2 than they need but, being an evolutionist, I'm not holding my breath. I imagine that even a small energetical efficiency increase would be a big evolutionary advantage and that plants are pretty close to their theoretical efficiency limit.

Of course IANAPNAPP (I Am Not A Physicist, Nor A Plant Physiologist), if you are please enlighten me.

</comment></comment></comment><comment><USER9> Great read. The things is, all the planets are heating up.

[http://www.livescience.com/environment/070312_solarsys_warmi...](http://www.livescience.com/environment/070312_solarsys_warming.html)

It would be a wonderful and improbable coincidence if the net gain in heat from the sun gained by the earth where absolutely perfectly offset by the net loss in heat.

</comment><comment><USER10> Brilliant piece. Thank you for posting the link.

Likewise: Aliens cause global warming. [http://www.crichton- official.com/speech-alienscauseglobalwar...](http://www.crichton- official.com/speech-alienscauseglobalwarming.html)

</comment><comment><USER6> It sounded good at first, but "Environmentalism as a Religion"?? Personally, I care about the environment because I LIKE nature, not because I feel it is some god-given mission to be steward of the earth.

Also, nature is a common good, so if I care about someone else destroying it, it might be less about wanting to spoil his fun, and more about me not wanting him to effectively steal something from me.

</comment><comment> patrickg-zill Dyson admits: there is no accurate way to measure CO2 previous to 1958, when Keeling starts. Further, CO2 varies in important ways depending on location.

</comment><comment><USER11> See <http://www.uoguelph.ca/%7Ermckitri/research/ispm.html> for a well reasoned dissent with global warming.</comment>


Free Google+ Invites - <AUTHOR> http://makingmoneywithandroid.com/2011/06/free-google-plus-invites/

<comment><USER2> I like how I keep seeing 'free' and 'contest for invites!'.

at this point I think most people who want one could get one very easily without demeaning themselves or spamming their twitter contacts with some contest…

<comment><USER3> This one's just free - no contest.

<comment><USER2> oh, I realize that. I was just commenting about the situation in general

</comment></comment></comment><comment><USER4> My email: <http://i.imgur.com/qKZce.png>

</comment><comment><AUTHOR> Looks like they just disabled invites again.</comment>


Last Issue Is December 17, 2014 - <AUTHOR> http://the-magazine.com/closing

<comment><USER2> what went wrong? It sounded like things were good when marco sold it

[http://the-magazine.org/under-old-management](http://the-magazine.org/under- old-management)

[http://www.marco.org/2013/05/29/sold-the- magazine](http://www.marco.org/2013/05/29/sold-the-magazine)

<comment><USER3> See here: [http://www.marco.org/2014/10/08/the-magazine-shutting- down](http://www.marco.org/2014/10/08/the-magazine-shutting-down)

and here: [http://www.loopinsight.com/2014/10/09/the-magazine-shuts- dow...](http://www.loopinsight.com/2014/10/09/the-magazine-shuts-down/)

for a couple more perspectives (tl;dr - readership decline, running a magazine is a lot of work, and Apple lost interest in News Stand)</comment></comment>


Embedded "turn Pages" UX for pdfs? - <AUTHOR>

I am trying to embed an ebook into a website I am working on and I wanted to have readers the "turning the page" experience that are in ibooks etc.  Are there any hosted ebook platforms (pdfs) that allow the user to "turn the page"? <comment><USER2> Try <http://issuu.com/>

</comment><comment><AUTHOR> Any ideas?</comment>


An Update About Changes to Facebook's Services in Australia - <AUTHOR> https://about.fb.com/news/2020/08/changes-to-facebooks-services-in-australia/ <comment><USER3> Australian government and more specifically Rod sims, the chair of ACCC should get some counselling. Really good counselling. They clearly don't understand how publishing works and have a biased understanding of who is benefitting from this relationship.

However, the entire Australian media has been nothing but puppets controlled by murdoch. The one true less biased source is ABC which has been muddied by constant federal government censorship and budget cuts almost to a point where independent journalism and Australia cannot be spoken in one sentence.

I hope they move forward with this code and I hope that both Google and FB remove news from their site so that newscorp can die quickly. It a shame they are doing this. Come next election I am voting against both SCOMO and the likes.

<comment><USER4> > the entire Australian media has been nothing but puppets controlled by > murdoch

You're going a bit heavy on hyperbole there. There's a decent amount of non- murdoch news; ABC, SBS, The Age, Sydney Morning Herald, The Guardian, The Conversation, etc.

<comment><USER5> It's actually getting worse and worse. The Age and the Herald, having been sold to Nine are much lower quality than they once were. The Government is putting more and more pressure on the ABC, with constant (effective) funding cuts, and both direct editorial interference or indirect (i.e. threatening more funding cuts because of journalists publishing articles they don't like). With the ABC, this has been going on for years - in 2013, they withheld a detailed analysis of the flaws of the Liberal's NBN plan (which has been proven accurate) by Nick Ross before the election because of pushback from "the Turnbull camp", and then eventually he was made redundant because of pressure from the Government. Last year, emails were leaked of the chairman saying they needed to "get rid of Emma Alberici and Andrew Probyn because the Government hates them and they put our funding at risk" \- for Alberici because she wrote some articles that got a lot of traction while the Government was trying to push big company tax cuts, pointing out that tax cuts haven't historically correlated with increased economic growth or employment growth (this is true). Last month, Emma Alberici was made redundant...

<comment><USER6> That's not even mentioning that the chairman of 9News is former treasurer of the conservative party which Murdoch is backing.

</comment></comment></comment></comment><comment><USER7> As I've mentioned before you only need to look at the fact that public and smaller news organisations e.g. ABC, SBS are not compensated to get an idea of the intention of this policy.

It's purely to prop up News Corp which is struggling (16% loss in 2020 revenue) because of weak newspaper sales and Foxtel bleeding customers to Netflix, Disney etc.

And unfortunately just like in UK and US whenever Rupert Murdoch says jump you better say how high or else your government will go down in the next election.

<comment><USER8> News Corp is impressive I think they are the only company that can actually make Facebook look sympathetic.

</comment></comment><comment><USER9> Stratechery has a good analysis of the "News Media Bargaining Code":

[https://stratechery.com/2020/australias-news-media- bargainin...](https://stratechery.com/2020/australias-news-media-bargaining- code-breaking-down-the-code-australias-fake-news/)

<comment><USER9> Committing the faux pas of replying to myself, here's another take:

[https://www.crikey.com.au/2020/08/31/scott-morrison-v- facebo...](https://www.crikey.com.au/2020/08/31/scott-morrison-v-facebook- google/)

</comment></comment><comment><USER10> As far as I can tell Facebook makes a good point. The Aus government designed heavy-handed regulation which clearly plays favorites and is totally one- sided. This is not behavior that we should tolerate. I absolutely support Facebook standing up for themselves here.

</comment><comment><USER11> I don't like FB as a company. However I have to admit, they are correct that news organizations benefit the most from this relationship. Why doesn't google do the same thing? Don't index news sites. Googles response seemed less effective than FB's.

<comment><USER12> The legislation explicitly prevents platforms from deindexing/downranking news sites that participate in the code in favour of other sites that do not.

Which places Google in a more difficult position, because if a user searches for some news, Google is essentially forced to return results from a participating news organisation, or not show any relevant news results at all.

While Facebook can just prevent users from sharing any news links, Google can't exactly stop users from searching for news.

<comment><USER13> Google absolutely can stop users from searching for News, by removing all News tabs, News boxes, and News News-specific metadata uses (such as current events timeliness) in search results at Google.com.au. They can then legitimately claim that they are no longer indexing news, only websites in general, and if site operators don't wish to be indexed they can follow the standard practices for exclusion.

Whether they (Google) will do so or not is a timely question to consider.

</comment></comment></comment><comment><USER14> Regardless of politics, media and big tech dynamics, anyone who thinks that sharing links can or should be charged for just doesn't understand the fundamental nature of the internet.

</comment><comment><USER15> Somewhat related, from Daring Fireball's clipping of [https://themargins.substack.com/p/facebook-the-pr- firm](https://themargins.substack.com/p/facebook-the-pr-firm):

> So, in that way, I read Facebook less as a tech company, but instead a > communications one. Not a telecom communications, but more like a PR / > marketing consultancy. There's nothing original about Facebook. It's a > company that hires people to build others' ideas, and, more often than not, > it does that better and faster than them too. And when it can't do that, it > just buys them outright. There is a lot of building, but the ideas are > outsourced. But what Facebook is really good at is actually doing all this > while fighting what seems to be a never-ending, at least since 2016 or so, > PR battle while not giving an inch.

> With all the negative press around, you might think they are not doing a > good job at avoiding criticism, but consider the alternative that they've > been able to weather all this because they've been able to deflect the > criticism and avoid scrutiny and accountability. I know this all sounds > pretty unhinged right now, but, stay with me. This is a company who hires > conservative politicians to its highest ranks in multiple countries, while > maintaining a veneer of political neutrality. The same company pretends its > not the arbiter of truth while employing tens of thousands of people to do > exactly that. Ask yourselves: What has changed at Facebook?

</comment><comment> ryan-allen I'm no fan of the Murdoch media and the damage it has done to Australia in terms of our internet infrastructure, but if Facebook are to block news media from their platform it will not affect me in the slightest, I don't trust _any_ of them to provide me the correct information.

I don't trust Facebook to serve me news, and I don't go there for news, and I don't think anyone should go to Facebook (or Google) for news.

I don't think Australians have a lot to lose from any big tech company blocking all news media on their platforms, and if they do so as a result of this legislation, and if it furthers the bleeding of these conglomerates, it sounds like a win/win situation.

There are probably negative follow on effects I'm not considering, though.

<comment><USER16> I agree with your assessment.

</comment></comment><comment><USER17> Social media sharing helps accelerate the sensationalism of news. Which in turn has been creating very bad situations in our reality. Without social sharing of news, maybe it would be more tame.

The quality of discussions on those platforms would also very likely benefit.

Hopefully other laws are passed with similar responses from tech companies in other countries.

It would be a great sort of reverse-monkey-paw fate. The world could certainly use some good luck right now.

</comment><comment><USER18> It really does feel like Murdoch has reached Bond villain status with this one.

He's has made the Australian government write a law that specifically and _only_ targets two multi-billion dollar global companies (Facebook and Google) and forces them to give him more money.

It specifically isn't going after other online companies that don't have a huge net worth, and it specifically isn't even attempting to hide the intent by also giving money to the not-for-profit news outlets in Australia (ABC, SBS).

Because he controls the very vast majority of media in Australia he can obviously force the government to do his bidding - one by ensuring all his government buddies stay rich, and two by promising not to publish anything that would result in them getting voted out.

A fiction writer couldn't come up with such an outrageous idea.

As an Aussie, I can confidently say Australia has utterly lost the plot, and I applaud Facebook for standing up to them. I sincerely hope Google follow suit, and they both refuse to come back even if the legislation is scrapped. I hope News Corp. dies a rapid death without the 2.3B clicks

</comment><comment><USER19> If Facebook really wanted to fight the government it could unleash it's democracy crippling social network to hurt this toxic government we have here in Aus. Wink wink Zuck.

But seriously this is just regulatory capture from News Ltd. Our Government works for them. Corruption is becoming a real and visible problem in Australia these days.

<comment><USER20> Because that worked so well in the US?

<comment><USER19> Yeah I know, I'm just wishing bad things on our current government who really don't live up to the word 'liberal'.

</comment></comment></comment><comment><USER21> This may not kill the large newspapers, but surely it will destroy the local newspapers. When does a tech blog become news? Can you even share anything on the internet through Facebook?

<comment><USER5> News Corp (who lobbied to get this bill created) already did a lot of that this year [1] - over the last few decades, they had bought up most of the newspapers in regional Australia, and then ended up deciding they were unviable and shuttered 125 of them in May this year.

They're keeping some kind of digital version but let go basically all the local reporters, photographers, etc. and closed all the local offices, and I think just run all the remaining online versions out of just a handful of locations now.

1\. [https://www.abc.net.au/news/2020-05-28/news-corp-makes- regio...](https://www.abc.net.au/news/2020-05-28/news-corp-makes-regional- papers-digital-only/12295408)

<comment><USER22> New Zealand has seen a proliferation of local news apps, employing local journalists, payed for by advertisements from local business, come to fill this gap [https://www.rnz.co.nz/national/programmes/mediawatch/audio/2...](https://www.rnz.co.nz/national/programmes/mediawatch/audio/2018644094/local- apps-fill-news-gaps)

</comment></comment></comment><comment><USER23> Without reading the legislation in detail, I have to say that I'm inclined to just take the side of whoever News Corp is upset at.

<comment><USER24> If you change your mind, [https://news.ycombinator.com/item?id=24337359](https://news.ycombinator.com/item?id=24337359)

</comment></comment><comment><USER25> If you are like me and wondering what the legislation draft is, please see

[https://www.accc.gov.au/focus-areas/digital- platforms/news-m...](https://www.accc.gov.au/focus-areas/digital- platforms/news-media-bargaining-code)

Some of articles discussing this topic do not provide a link to the the draft legislation source.

</comment><comment><USER26> Nevermind the reasoning

If the result is no more/fewer news on FB, doesn't that mean it will be a more difficult platform for political parties and Rupert Murdoch's of the world to manipulate?

<comment><USER27> Murdoch already has his news distribution systems in place in Australia -- the less other outlets can gain visibility the better as far as he is concerned

<comment><USER28> News Corp is losing money across almost all of its Australian properties. The government keeps finding ways to prop them up under the guise of subsidies for local sports coverage because News Corp provides them with such favourable coverage. It's blatant corruption.

</comment></comment></comment><comment><USER29> I haven't followed this debate too closely but this seems pretty disingenuous.

 _> The ACCC presumes that Facebook benefits most in its relationship with publishers, when in fact the reverse is true. News represents a fraction of what people see in their News Feed and is not a significant source of revenue for us_

News may not directly contribute much ad revenue but it surely is a huge driver of engagement on the website, which in turn indirectly keeps people engaged with the facebook ecosystem at large. Political commentary involving news pieces certainly seems to be a feature on Facebook's most popular pages.

Also I haven't figured out yet what part of the law actually would imply that Facebook needs to blackout news wholesale. As I read it the law merely requires that Facebook negotiate with each publisher.

<comment><USER11> Facebook can remove news and it will minimally effect engagement. I wonder what happens to the publishers when they lose 2.3BN clicks.

<comment><USER29> Which would indirectly prove the point of the legislation right? That Facebook has immense market power in the social media / ad space with no competitor in sight, which would mean expanding bargaining power for news orgs is legitimate.

Because switching to a competitor would actually be the answer to your question in a functioning market.

<comment><USER30> No, because the argument being made is that facebook is paying the news orgs less than they're owed. But the reality is that facebook is providing a free service to the news sites.

You normally have to pay someone for advertising. Facebook (and Google) give it away for free.

<comment><USER29> This seems to miss the point because normally platforms also have to pay for content. Facebook is a platform that hosts user/business produced content. No content, no reason to go to facebook. Likewise, Facebook brings attention to content owners. All of this is 'free' nominally.

That's not the question, the question is who has more market power and squeezes who.

<comment><USER30> Are you saying that Google should pay me to include my website in it's search results?

<comment><USER31> If they're stripping your information to hand out to users via their scraping mechanics, absolutely. That's what news companies, and plenty of other sites, have been subject to for years: "Google would like to scrape your content and serve it on their own site. Agree or delist yourself from search."

[https://theoutline.com/post/1399/how-google-ate- celebritynet...](https://theoutline.com/post/1399/how-google-ate- celebritynetworth-com)

Google's monopoly position gives them complete negotiating power, because you can't survive on the Internet without them. What Australia is doing here seems extreme, but only because you aren't accounting for how extreme Google and Facebook's existing negotiating power is. If you don't understand how abusive Google and Facebook are, you'll think this deal is unfair. It's not.

<comment><USER30> Right but we're talking about news, where that _isn 't_ the case. I open Google news and I see a list of headlines. I click one one and go to the full article.

I mean, if you're arguing that news provides zero value beyond a headline, that's certainly an indictment, but not of Google and facebook.

Extending this, any aggregator would need to pay to link to things. Hackernews included. It's the link-tax all over again.

<comment><USER31> Realistically? What percentage of news do you read the in-depth article, versus the headline, picture, and two-sentence-blurb that tells you the main point? And the front page of a news site, say, cnn.com, is it monetized with ads? (It is. Even with all my blockers I see a LendingTree ad right now.)

Is Google News ripping off the monetizeable property of every news site's homepage for zero cost? Yes. Sure, it may not present the full text of every article, but you've stolen the front page of every newspaper, at minimum.

It is the link-tax "all over again"... because it's still what needs to happen. (It's not really a link tax, but branding from lobbyists paid by Google's public policy team were very successful branding it as such. It's a scraper site tax.) The problem is Google is so immense, and so powerful, it is able to obliterate any one nation's attempts to do this.

We actually need a unified front of governments stating that Google and Facebook need to pay for the content they scrape, or go out of business. And it probably won't be successful until a large enough contingent of governments can do it at once. (This is why the EU as a whole has generally been successful whereas attempts at regulation from France or Spain independently have failed.)

<comment><USER30> > Realistically? What percentage of news do you read the in-depth article, > versus the headline, picture, and two-sentence-blurb that tells you the main > point? And the front page of a news site, say, cnn.com, is it monetized with > ads? (It is. Even with all my blockers I see a LendingTree ad right now.)

For _me_ the answer is, either I get a headline that I don't care about, or I read at least some of the article.

> (It's not really a link tax, but branding from lobbyists paid by Google's > public policy team were very successful branding it as such. It's a scraper > site tax.)

It's absolutely a link tax and _not_ a scraping tax. A scraping tax would apply to anyone _scraping_ the site. But the link tax proposals don't apply to people scraping the site, they only apply to sites linking. Why? Because if I go and scrape CNN.com, they just block me. But if Google does it, they don't want to block the scraping, because the net value of the scraping is nonzero, but they want to be paid anyway.

You can't have it both ways. "Pay me for this." "No." "I'll make you pay me!" "Ok then I'll stop." "No don't stop that's even worse. Pay me and keep doing the thing!" That's the conversation happening, and it makes zero sense.

And "scraping" doesn't even apply at all to Facebook's case, where if _I_ post a link to CNN on facebook, now Facebook owes CNN money. WHAT?!? It's literally a cost on links. It's antithetical to the _inter_ net. But you're right, it's not a link tax. It's more akin to link-licensing. Which should sound even scarier.

<comment><USER31> > You can't have it both ways. "Pay me for this." "No." "I'll make you pay > me!" "Ok then I'll stop." "No don't stop that's even worse. Pay me and keep > doing the thing!" That's the conversation happening, and it makes zero > sense.

But that is the level of craziness we are under in the Google regime: Accept Google's terms or go out of business. That's the whole problem. Google has infinite negotiating power. The counter to that does indeed sound insane, because where we already are is insane.

Yes, Australia is giving news companies a loaded gun, but it's because Google already has a boot to their neck, and they're barely still conscious. Where we are right now is _really, really_ bad. Tech companies are bleeding journalism dry.

And frankly, Google and Facebook can afford this. Tech companies have so much spare cash they don't know what to do with it, while everyone else is struggling to survive. Expect laws to feel one-sided against you for a long time. Let's call it a wealth redistribution. Imagine how much quality investigative journalism could've been funded with just the money Google spent developing messaging apps.

<comment><USER30> You can dislike google and facebook, and feel that they need to be regulated, and still not support objectively bad laws.

> Imagine how much quality investigative journalism could've been funded with > just the money Google spent developing messaging apps.

If you want to tax big tech companies because you feel the end result of doing so is good, then do that. Tax them. Don't create some hairbrained scheme to allow a media conglomerate to extort them. Especially when it won't actually help investigative journalism. The law is crafted to support an entrenched tabloid, not local and investigative journalism, which is what's suffering the most. This law funnels money from tech companies to rupert murdoch. That's not a win. The fox news guy doesn't need a bailout, he's doing okay on his own.

> Accept Google's terms or go out of business.

What are these "terms"? Aa far as I can tell they're "get traffic for free". What am I missing? Your counter seems to be that Google posts the headline so that, you know, you can pick what to click, and that somehow having the headline means that I won't click on the link. I don't get it.

<comment><USER31> > What are these "terms"?

Let me highlight some history from the article above:

> In 2014, Warner got an email from Google asking if he would be interested in > giving the company access to his data in order to scrape it for Knowledge > Graph, for free.

> At the end of it, we just said 'look, we're not comfortable with this.'"

> "But then they went ahead and took the data anyway."

The sole way to opt out that Google offered was to delist the site from Google Search, which is effectively deleting your site from the Internet.

So the terms of your employer are basically "let Google do what it wants or else". And that's what they've been doing to news companies for years. There's nothing "objectively bad" about forcing a bad actor to finally pay up for it's conduct. Allowing Google or Facebook to decline to participate would be simply allowing the robbers to leave with all of the loot.

To better explain: Google and Facebook are monopolies. One common monopoly practice is called predatory pricing: Where you price so low, you are losing money, _temporarily_ , in order to kill competing businesses. For Google and Facebook, refusing to negotiate here, removing service from a given country, is similar: They're aware if they decline to participate, a given country's news organizations will suffer, and eventually relent.

Google and Facebook, as global monopolies, are large enough and profitable enough that this doesn't hurt them much, so they have no reason to negotiate fairly and honestly. Hence why Google and Facebook argue the price they should pay is $0: They currently don't bear a significant enough loss from simply refusing to play. They must be _forced_ to participate so that honest negotiation can happen. Realistically, the long term answer is to break these companies up, but in the interim, we need to use regulation to force them to pay up in areas where they would have to if they weren't monopolies.

<comment><USER30> Again: I'm talking about news. You keep bringing up a thing that isn't news. What are the terms for _news_ sites?

> There's nothing "objectively bad" about forcing a bad actor to finally pay > up for it's conduct.

Correct. That's why I said, essentially, "if you want them to pay up, tax them". I'm not saying any law that forces tech companies to pay money _must_ be objectively bad. I _am_ saying that _this law_ is. You understand the difference.

> Realistically, the long term answer is to break these companies up, but in > the interim, we need to use regulation to force them to pay up in areas > where they would have to if they weren't monopolies.

And to do this, you support a bit of regulatory capture that instead directs the funds to a media monopoly. Like I said, it's an objectively bad law.

> One common monopoly practice is called predatory pricing: Where you price so > low, you are losing money, temporarily, in order to kill competing > businesses

You're actually arguing that the price, free, which Google and Facebook charge is too _high_ , and that they need to further decrease it (by subsidizing the content they aggregate). Is the price too high, or is it predatorily low? You can't argue both sides. If you were really concerned with predatory pricing, you'd be against this law, because it (if applied generally) prevents competitors to Google or Facebook from gaining marketshare, since they can't afford to pay media companies. So as long as the entrenched actors pay the tax, upstarts _can 't_ start.

<comment><USER31> > "if you want them to pay up, tax them"

Tried that. Your employer as of yesterday has... decided to make the little guys pay for that too: [https://www.businessinsider.com/apple-amazon-google- pass-cos...](https://www.businessinsider.com/apple-amazon-google-pass-costs- digital-services-taxes-2020-9?op=1)

Fundamentally, we're trying a lot of different strategies to fix this, but Google is greatly invested in dodging any sort of responsibility for it's externalities and actions. Perhaps the solution here is to both implement a tax _and_ prohibit harmful over-trillion-dollar-monopolies from raising prices at the same time? But then, they'll say it's unfair that they can't set whatever prices they want.

I am not sure Australia's attempt will be any more successful, but it's increasingly becoming apparent we just need to ban these companies from operating, forcibly shut them down, and retire their user-hostile services.

</comment></comment></comment></comment></comment></comment></comment></comment></comment></comment></comment></comment></comment></comment></comment><comment><USER32> Australian here... Not sure who to trust on this.

</comment><comment><USER33> This is a good thing. Media content quality has primarily been garbage the past 10+ years (and getting progressively worse), mostly due to the advertisement click-bait business model forced on them in order to survive. Social media kind of naturally fell into position to reap the benefits...they shouldn't be the gatekeeper to news without paying a fair price.

</comment><comment><USER34> 2 Points: Underscoring that facebook is not now, nor was it ever a good source of news is a very good thing. Don't get your news from facebook. Don't get it via facebook. Anything on facebook that isn't actually personal is tainted and horrible to be treated with extreme suspicion. Facebook have earned that reputation and deserve it.

People going elsewhere on the internet than facebook to get news will reduce facebook engagment, use and get people in the habit of not going to facebook first.

Bring it on. Facebook are horrific. The less they do the better. Especially given they apparently won't do anything about being used for violent organisation.

About the only organisation guaranteed to be more wrong than the Australian Government about literally everyting is Facebook.

[1] [https://www.vox.com/recode/2020/8/28/21406022/facebook- banne...](https://www.vox.com/recode/2020/8/28/21406022/facebook-banned- violent-militia-groups-kenosha-protests-jacob-blake-shooting-kyle-rittenhouse)

[2] [https://www.nytimes.com/2018/11/06/technology/myanmar- facebo...](https://www.nytimes.com/2018/11/06/technology/myanmar- facebook.html)

<comment><USER35> I get lots of news from Facebook. My friends who I trust post articles that they've read. It's a perfectly fine source of news if your friends are the kind of people who actually think critically and read the articles.

If the news you get from Facebook is suspect, it's your friends that are the problem, not Facebook.

<comment><USER34> you get that news from your friends that facebook algorithms deign to show you.

They have algorithms to influence. They have said so. They were trying to manipulate people's mood with it for "research"

This isn't contraversial.

It's a garbage source of news because of that filter. Even if your friends are lovely, wise, empathetic, intelligent and critically minded and their editorial selection is beyond reproach - that isn't what you get.

</comment></comment></comment><comment><USER36> Why does the title say "An Update About Changes to Facebook's Services in Australia" rather than "Changes to Facebook's Services in Australia"? For me (2) is much more readable.

<comment><USER37> "An update about..." implies the reader is unlikely to care unless already following the story. If they wanted to reach normal users there would be A/B-tested headlines, pastel illustrations, and bolded reassurances that your Facebook isn't going away.

</comment></comment><comment><USER38> This is the right move by Facebook (and Google) in the face of naked crony capitalism.

It's an effort by the Morrison government to engage in rent-seeking on behalf of Rupert Murdoch's News (which essentially operates as the communications arm of Morrison's government), Nine (which held a fundraiser after the last national election for Morrison) and other media groups.

Yes, there are many valid criticisms of and concerns we should have with tech giants, but this proposed code doesn't deal with any of those concerns.

Australia's former Prime Minister Kevin Rudd, who has stepped up his war against Murdoch's influence on Australian Democracy in recent years, was incredibly perceptive and scathing of this proposed code recently [1].

[1] [https://www.youtube.com/watch?v=XiFTT_QgqIc&feature=youtu.be...](https://www.youtube.com/watch?v=XiFTT_QgqIc&feature=youtu.be&t=351)

</comment><comment><USER39> Why are we getting hot by this in New Zealand? Last I checked we were autonomous from Australia and our only legal connections are treaties and the crown.

<comment><USER40> Where are you reading that this affects New Zealand?

<comment><USER39> Thank you. I see what I did. I read the job title and confused things.

</comment></comment></comment><comment><USER41> How much do we have to pay Facebook to do this in the US?

</comment><comment><USER42> A week ago back some friends and I looked at the draft of the law... it's honestly terrifying.

They basically want Google/fb/? to be upfront with all changes. The goal is clearly to give aussie media outlet an unfair advantage and let them use this information to manipulate the ranking algorithm and/or bypass whatever restrictions are put in place. These info need to be "in terms that are readily comprehensible".

IANAL, but here's a few (reformatted) articles from the draft:

52M:

Google/fb/? will need to give aussie media companies access to:

\- a list and explanation of the data that the digital platform service collects (whether or not it shares the data with the registered news business) about the registered news business' users through their engagement with covered news content made available by the digital platform service;

\- a list and explanation of the data that the digital platform service currently has a practice of making available to registered news businesses;

My interpretation: these companies will be required by law to share more user data than they currently do.

52N:

\- if: changes are planned to be made to an algorithm of the digital platform service; and the changes are likely to have a significant effect on the ranking of the registered news business' covered news content made available by the digital platform service.

\- then: notice of the change is given to the registered news business corporation for the registered news business at least 28 days before the change is made

My interpretation:

The government wants aussie media companies to be able to be proactive and change their site/service so that ranking changes are minimised

52O:

\- if: the changes are specifically designed to have an effect on the ranking or display of content behind a paywall.

\- then: notice of the change is given to the registered news business corporation for the registered news business; and the notice is given at least 28 days before the change is made;

My interpretation:

media companies don't like when Google penalises paywalls. This will give them a 28 day window to change how they do paywall so that Google won't be able to detect it. Like a whack a mole where you have to announce what you are about to hit

and this keeps going:

\- 52P: 28 days notice for changes to how news items are displayed

\- 52Q: 28 days notice for changes to advertising (if it affects news)

\- 52S: media outlet must be able to moderate comments on their items (does that include comments on shared items? surely not?)

Draft: [https://www.accc.gov.au/system/files/Exposure%20Draft%20Bill...](https://www.accc.gov.au/system/files/Exposure%20Draft%20Bill%20-%20TREASURY%20LAWS%20AMENDENT%20%28NEWS%20MEDIA%20AND%20DIGITAL%20PLATFORMS%20MANDATORY%20BARGAINING%20CODE%29%20BILL%202020.pdf)

<comment><USER43> > The goal is clearly to give aussie media outlet an unfair advantage and let > them use this information to manipulate the ranking algorithm and/or bypass > whatever restrictions are put in place. These info need to be "in terms that > are readily comprehensible".

I wonder what kind of malicious compliance we could see while still being within the letter of the law.

e.g. Articles that start with "Murdoch is a cancer on society" will be ranked much higher than those that don't. Enjoy.

</comment></comment><comment><USER44> What I find funny about this reaction is that most Australians are actually either quite blase about this situation, or actually are quietly enjoying the fact that massive multinationals who will do anything to avoid paying taxes in Australia have finally discovered that their power is quite limited in Australia.

Most Australians have seen what Facebook was up to during the 2016 election. They have seen the rise of Trumpism and, despite the incredibly conservative government, don't want that sort of crazy in their nation.

<comment><USER16> I'm Australian and I see this law as confirmation of dementia in the government. This is all kinds of unrealistic.

Having to notify Murdoch of any changes to search algorithm 28 days in advance? I wonder if that is even possible - it's probably a magic ML algorithm that they tweak here and there.

I can't imagine that google/facebook would continue linking to news under these conditions. The small return is not worth the cost.

</comment></comment><comment><USER45> But not Whatsapp so no biggie.

</comment><comment><USER46> I am no fan of this right wing government, which is letting Fake News Corp write it's media policy, but Facebook does not have an inherent right to other people's content.

Rather than redistributing ad dollars away from big tech, this is likely to force people consuming news to go directly to the source. Either that or it will further dumb down our discourse (though Fake News Corp is doing a pretty good job of that anyway). Time will tell.

<comment><USER47> Facebook is not taking anyones content. News companies willingly submit their content to facebook and now demand that facebook should pay for the stuff they submitted. So facebook is taking the logical step by stopping them from submitting the content.

I can't walk up to someone and hand them the keys to my car and then demand that they now owe me the value of the car.

<comment><USER48> They're not even submitting their content; they're submitting _links_ to their content. Readers still need to go to the publishers' pages to read the articles.

<comment><USER46> This isn't even true when you consider news articles that are cached by Google's Accelerated Mobile Pages (AMP).

<comment><USER49> The publisher controls whether AMP is used. If they don't want to use it they can stop publishing it.

</comment></comment></comment></comment></comment><comment><USER50> As an Australian I have no doubt the policy is misguided, but the fact that these American companies are being so loud about it makes me wonder if the government is actually on the right track.

In general I would say whatever corporations want is not in my best interest, so whatever they don't want, might be.

<comment><USER51> That feels like a pretty bad take on how to review changes. You might get better mileage by looking at the context each time. In this case its pretty clear that the Australian goverment is acting on behalf of specific vested news companies and NOT for the consumer...

It seems dangerous to have this kind of government-on-the-right-track reaction when the corporation cheering it on is in the business of fossil fuels, factory farming, or opiod manufacturing just because a different corporation yells loudly about it.

<comment><USER50> Fair enough, I simply don't trust any corporation to act in my interest - <USER2> there was a trusted third party lobbying for FB/Google's position I may have another look

Of course I would like to see the government go further and upset News Corp as well, but upsetting these two seems like a good first step

<comment><USER52> FB doesn't have to have your best interests in mind for their actions to make sense. You also don't need to trust them to agree or disagree with them.</comment></comment></comment></comment>


Marijuana Contains "Alien DNA" from Outside of Our Solar System, NASA Confirms - <AUTHOR> http://www.iflscience.com/editors-blog/marijuana-contains-alien-dna-from-outside-of-our-solar-system-nasa-confirms/ <comment><AUTHOR> > People are more willing to share an article than read it. :)</comment>


FlightCar (YC W13) Launches Its 'Airbnb For Airport Car Rentals' Service At SFO - <AUTHOR> http://techcrunch.com/2013/02/15/flightcar/

<comment><USER2> I was skeptical about this idea (as I was about Airbnb) till I saw how YC alumni reacted to it. They all seem to want to use it.

<comment><AUTHOR> It truly does seem like the craziest ideas sometimes turn out to be the most popular, but I guess that's because the people behind them have a creative vision of the future, aka true entrepreneurs.

One of my concerns for FlightCar is the amount of competition they face in other cities. Expansion could be tough but again, their prioritization of customer service is a great initial platform whereupon users will be naturally attracted to using their service.

One idea I had was to create a rewards system such that people gain 'points' by putting their car up for rent. And once a number of points is accumulated, prizes are offered or something of the sort. Could be a neat marketing tool as well as an edge on similar services.

</comment></comment><comment><AUTHOR> It is great that FlightCar is doing the little things to prioritize customer service. As noted in the article, their idea of specializing in multi-day rentals is unique and in time, I believe they will beat out their competition.

<comment><USER3> Thanks for the kind comment! We really do believe it's all about the customer in this industry plagued with terrible stories of HR gone wrong. :)

<comment><AUTHOR> No problem! I guess one concern of mine is that I can't seem to find is whether or not some level of background checks are being done on the drivers who put their cars up for rent. It could be a potential security hazard right?

<comment><USER3> We do an MVR check--not the same as a background check, but we do double check identities as we have physical hand-offs. Its good to note that the MVR assures that they're a relatively good driver, and that our theft, collision, and liability insurances will kick in if anything goes wrong.

<comment><AUTHOR> Gotcha, that sounds great...good luck with everything! Bring it to the less exciting of Pittsburgh as soon as you can :D</comment></comment></comment></comment></comment>


Money Trees – Rap Genius Response to Heroku - <AUTHOR> http://rapgenius.com/Lemon-money-trees-rap-genius-response-to-heroku-lyrics <comment><USER2> > _This works for future customers, since once Heroku makes these > documentation changes, everyone who signs up will understand exactly how > routing works. But it does nothing to address the time and money that > existing customers have spent over the past few years. What does Heroku owe > them?_

Lawyers -- well, judges really -- are good at coming up with answers for this exact sort of question.

I am not being facetious. There are legal rules for assessing losses in even very complex, very entangled situations. If you feel Heroku has dudded you, find a torts lawyer.

Heck, Salesforce.com have deep pockets. Round up a few other $20k/month customers and start a class action.

Web companies need to realise that boring old-fashioned rules like "your claims should not be misleading" apply to them too.

(IANAL, TINLA)

<comment><USER3> ummmmm......salesforce bought heroku

<comment><USER4> The implication there was that because Salesforce owns Heroku, they would be able to pay up major cash if Rap Genius got some other Heroku customers involved in the lawsuit.

<comment><USER5> I don't think you can sue Heroku because it was "slow". Were any SLAs offered by Heroku explicitly not met? If so that's a different story.

If AWS was able to be sued by every customer who didn't understand how their infrastructure config worked with the underlying EC2 network, or did something incorrectly for a time due to a documentation mistake there would be more lawyers working at Amazon than engineers.

<comment><USER6> The argument is not about sueing Heroku because of them being slow. It's sueing them because they lied.

<comment><USER5> Define "lied". Not having clear documentation or you not understanding 100% how their back end system is implemented does not mean they were lying.

</comment></comment></comment></comment></comment></comment><comment><AUTHOR> If you use Heroku and New Relic, make sure you install the gem we wrote to make New Relic report correct queue times: <https://github.com/RapGenius/heroku-true-relic>

<comment><USER7> Hey Tom, I'm the guy who wrote the queue time instrumentation in the New Relic ruby agent. Note that I no longer work for New Relic and my opinions are my own, not those of New Relic or my current employer.

I don't recommend that you use this patch - work needs to be done on Heroku's end, this is not a satisfactory workaround. The ideal would be for them to add a timestamp to the headers at the front end of the dyno machine (i.e. in apache/nginx/whatever) to allow the calculation to include a local-machine- relative timestamp rather than one reliant on two servers being in sync.

The major issue is that servers on AWS do not have synchronized clocks, in general. I'm not sure how Heroku manage their servers, but I do know that in the samples I saw several years ago, we had a very large variance in the queue time reported based solely on that clock skew.

The New Relic reported value is an average, which is a poor choice for something like this, but it's very difficult to graphically illustrate queue time across a network of machines without resorting to it.

I'd be happy to discuss it further, and I know that sgrock [1] is also around the neighborhood - he's one of the current Ruby Agent maintainers.

1: <http://news.ycombinator.com/user?id=sgrock>

</comment></comment><comment><USER8> Man, there is a LOT of expertise over there at Rap Genius just to have a website where you can figure out what "hollatickin" means.

<comment><USER9> It's not just about rap: [http://rapgenius.com/Marc-andreessen-why-andreessen- horowitz...](http://rapgenius.com/Marc-andreessen-why-andreessen-horowitz-is- investing-in-rap-genius-lyrics)

<comment><USER10> The tech. isn't exactly crazy though. A JavaScript popup over some text.

<comment><USER8> The implementation is very good though. It's clean, easy to use, and very useful. I could imagine it being very useful on something like Wikipedia. I don't always need to go to an entire article, maybe I only need to see the first paragraph on hover.

<comment><USER11> Wikipedia actually already has this in the form of Navigation Popups: [http://en.wikipedia.org/wiki/Wikipedia:Tools/Navigation_popu...](http://en.wikipedia.org/wiki/Wikipedia:Tools/Navigation_popups)

<comment><USER8> hey, you're right! I usually surf with JS off and never noticed!

</comment></comment></comment></comment></comment></comment><comment><USER12> My guess as an armchair observer (and tiny-scale Heroku user) would be that Heroku will offer some affected customers refunds, especially if those customers "threw dynos" at latency problems that were aggravated by the drift in Bamboo routing behavior and hidden by the misleading NewRelic monitoring.

I don't think Adam@Heroku's response on the 11th is that bad. He accepts the feedback and also wants Heroku to help RapGenius 'modernize their stack'. That's not a full and proper solution, nor a remedy for the lost cost/effort so far, but it would have offered a lot of performance and cost relief.

In fact, I think that's why this problem festered: many customers managed to soften the pain by going to Cedar, multiple-workers, app-optimizations, and more dynos... so deeper investigations kept getting backburnered, both inside and outside Heroku, until now.

RapGenius has done us a mitzvah by finally digging deeper, but I'm still eager to see what Heroku thinks the right remedies are, beyond RapGenius's 'must do' ultimatums.

<comment><USER7> > hidden by the misleading NewRelic monitoring.

The assumptions built into the queue time and queue depth monitoring were essentially the same - that routing would hold requests until a dyno was free, and all queueing happened at the routing fabric level not the dyno level.

Unfortunately, so far as I am aware the only way to get a 'true' round trip time for a given web page is to look at it from the user's perspective as they request it - as a percentage of time, the network roundtrip is the only number you really care about.

If they had been using New Relic correctly (note that I don't work for or speak for New Relic, I'm just a former employee), they'd have seen on the javascript-enabled monitoring that requests were taking a long time. The server-side time is only a portion of that, but it's clearly delineated.

I think this whole thing is composed of two issues: Rap Genius realized that requests were queueing at the dyno level (bad) and decided that they needed numbers to back that up. Unfortunately they picked a number (queue time) that doesn't have much functional basis on Heroku's stack at the moment, which weakens their argument.

What I would like to see change, is to see an additional header placed on the front end of the dyno machine by Nginx or Apache or Yaws or whatever web server runs local to the dyno, immediately as the request hits the machine. That would enable the current New Relic Agent to pick up the queue time spent on the local machine correctly, and basically entirely eliminate the problem of inaccurate queue time statistics.

There's actually code in there already to handle this already - add an HTTP_X_REQUEST_START header to your requests as they enter the machine and it'll be recorded. Not sure how it's displayed these days, I haven't been privy for a couple years now, but the code still exists and records statistics in the Agent.

<comment><USER13> > If they had been using New Relic correctly (note that I don't work for or > speak for New Relic, I'm just a former employee), they'd have seen on the > javascript-enabled monitoring that requests were taking a long time. The > server-side time is only a portion of that, but it's clearly delineated.

Note that they were using New Relic as part of an expensive add-on package from Heroku. It gave them a queue time value in it's reports, but it was extremely misleading, since the only value it showed was for the router queue time, which should have always been extremely low (and was displayed as such). It didn't say "router queue time" or "dyno queue time shown elsewhere".

Since New Relic is supposed to be showing them everything that happens with their request on Heroku's servers, it seems logical that it would include dyno queue times.

Javascript-enabled monitoring would only show you that the request times are much longer than what Heroku says they are, then you still have to troubleshoot down to figuring out why.

> Unfortunately they picked a number (queue time) that doesn't have much > functional basis on Heroku's stack at the moment, which weakens their > argument.

I don't think it weakens their argument as it clearly shows that the biggest problem they have is not only out of their control (even with very short run- times, the higher the number of requests you have per minute, the more this problem is going to affect you) but that even buying very expensive tools integrated into Heroku's "stack" will not help you to see where the problem is. The tools were basically hiding the one problem that was solely Heroku's responsibility.

Do not forget that even while there were a lot of statistics about how long running requests can cause other, much shorter requests to take just as long and even timeout, the heart of the matter is that even with a high number of extremely short requests, the router can end up sending many requests to a single dyno while other dynos remain idle. There were plenty of graphs, even animated to show you the effect over time for this random dyno routing.

<comment><USER7> > It didn't say "router queue time" or "dyno queue time shown elsewhere".

I can tell you that personally when I was writing the code that calculates that queue time value, several years ago, we didn't think such time existed. It was either router time or nothing.

> The tools were basically hiding the one problem that was solely Heroku's > responsibility.

Right, absolutely agree, but the problem is their current queue time measurements do not make a strong argument for this due to the issue of clock skew. They're essentially picking a more-or-less nonsense number and saying that it demonstrates that Heroku is bad. The only numbers that are fundamentally reliable are the total request time from the javascript side.

</comment></comment></comment></comment><comment><USER14> There are still some important points missing from the discussion:

1\. Operating at scale with parallel routing. 2\. Handle faults while operating at scale with parallel routing 3\. Providing correct statistical models for the situation. The one we have right now is a crude approximation. 4\. Measuring on the real system for problems.

The optimum routing is to have each dyno with 0 or 1 job at a time and a global queue of all incoming requests. But this is a latency problem then since it takes time for a dyno to tell that it is "ready". The net result is very bad performance and the global queue is a single point of failure. The solution is to queue because this removes the latency --- but with the price you see RG paying if a Dyno can only serve one request at a time.

If a dyno does not report "ready" to the routing mesh, then you can't route optimally:

Queue length doesn't work since a request in queue may take 7000ms while still having a length of 1. Another queue with length 5 consisting of 5 70ms requests is better to route to.

The time the last request spent in queue is not useful either because the very next message may be a 7000ms one.

So to solve this problem, you must do something else. You cannot use "intelligent routing" unless you can describe how it will work distributed with, say, 8 routing machines while avoiding latency. And while you are at it, you better measure your solution in a real-world scenario.

<comment><USER15> "The optimum routing is to have each dyno with 0 or 1 job at a time and a global queue of all incoming requests. But this is a latency problem then since it takes time for a dyno to tell that it is "ready". The net result is very bad performance and the global queue is a single point of failure. The solution is to queue because this removes the latency --- but with the price you see RG paying if a Dyno can only serve one request at a time."

You are right that there is an inherent latency hit using intelligent routing versus random routing. However in a resource constrained world, where no one can afford infinite dynos, there is also an average latency hit to random routing -- but rather than being largely fixed for all requests it is instead highly variable. While the magnitude of the two need to be factored in, ceteris paribus low variance in latency is better.

As for your single point of failure point, there are distributed queue algorithms that handle router failure gracefully.

<comment><USER16> Could you point me to such distributed queue algorithms that can handle failure gracefully? I'm interested in reading on the topic.

<comment><USER15> Sure. See <http://research.microsoft.com/pubs/153348/idleq.pdf>, <http://arxiv.org/ftp/arxiv/papers/1103/1103.2408.pdf>, and [http://static.googleusercontent.com/external_content/untrust...](http://static.googleusercontent.com/external_content/untrusted_dlcp/research.google.com/en/us/archive/spanner- osdi2012.pdf).

</comment></comment></comment></comment><comment><USER17> This incident has done wonders for RapGenius's technical brand. I don't know how many people would've identified them as a 'tech company' before, but that number has surely gone up.

</comment><comment><USER18> Guys, you've made a lot more money than me, so you don't need my advice. But if you want money back, you should probably be communicating in private through your lawyers. Posts like this look like you're trying to get (more) attention.

<comment><USER19> I'm super glad they are taking this public and getting tons of attention. I'm consulting for some people who are plagued with the same H12 errors. We've spent tons of time and money trying to mitigate these errors as much as possible to no avail. Bringing this problem up to Heroku through their paid support channels has always come back to them finger pointing at everyone but themselves. I'm glad someone is stepping up and pushing back.

<comment><USER18> The first post, definitely. But this one?

Maybe I'd feel differently if I were a Heroku customer, so I'll defer to people directly affected.

</comment></comment></comment><comment><USER20> Heroku's suggestion: "modernize and optimize your web stack."

I don't have any experience with Ruby web stacks so I'm curious if this is actually an option for you guys? What would it take to do that? Would the performance increase on Heroku be worth it?

It also seems like if you wanted to self host you would probably need to do those same improvements, right?

Please don't take my comment the wrong way, I'm not trying to say Heroku is somehow excused from their mistakes here. I'm just trying to understand that suggestion from Heroku.

<comment><USER21> Here's a discussion about that point in the original article's comments: <http://news.ycombinator.com/item?id=5216553>

The conclusion is that it will buy you a bit more time, but does not fix the underlying issue.

In Rap Genius' case, they're large enough that they would still have significant issues even if they switched to cedar and unicorn with 2-4 worker processes.

<comment><USER22> Agreed. I am working with a client who is slightly lower on the Heroku customer food chain and using Unicorn with four workers right now. Our next step is Puma though that will likely not be the end point.

<comment><USER23> Certainly investigate Puma on Rubinius or JRuby, and make sure you have config.threadsafe! turned on.

</comment></comment></comment></comment><comment><USER24> I've lost a lot of faith in Heroku this last week. Going to be doing a lot of investigating Cloud66/Elastic Beanstalk + EC2 for my Rails app. Good excuse to up my sysadmin abilities a bit.

</comment><comment><USER15> Why does Adam Wiggins repeatedly use the word 'evolve' as a transitive verb in an awkward fashion? Is this some sort of start-up usage that I managed to avoid thus far?

"We're working on evolving away from the global backlog concept in order to provide better support for different concurrency models, and the docs are no longer accurate."

"Getting user perspective is very helpful and I'll apply your feedback as we continue to evolve our product."

"You're correct that we've made some product decisions over the past few years that have evolved our HTTP routing layer away from the "intelligent routing" approach that we used in 2009."

Evolve to me connotates natural selection -- which is rather more haphazard than I would hope for from a engineering process.

<comment><USER25> People don't like change.

Using the word "change" makes it more obvious and the changes they made to their system (that were debatably for the worse) deliberately.

</comment></comment><comment><USER26> Maybe this is offtopic, but I really don't like the way Rap Genius does links. It makes it so I essentially have to click on each link twice to get to what it actually goes to...

<comment><USER27> They're not links, they're text annotations. Rap Genius is a text annotation platform currently focused on rap lyrics. In this case most of the annotations happen to be a link + context, which is pretty rare.

<comment><USER26> Yea, I got that... but in the context of a tech blog post, the interface just doesn't seem to make sense or at least in my head. Its a blog post, not rap lyrics.

<comment><USER27> I think their hypothesis is something like all long-form text should/will be annotated. While I think it's important to dog-food, I agree that the execution in this case isn't great. The "links" really should just be links.

</comment></comment></comment></comment><comment><USER28> I'm sorry, but I don't understand any of this hating on Rap Genius.

There's a reason they are the fastest growing YC company ever, and got a16z in for 15M -- because they are straight killers. They have quietly created an internet empire until this point, and are building something that people love and use everyday.

A lot of folks wouldn't have the chutzpah to call out Heroku like that or are just too small to make this kind of attention. To me it seems as though they are helping Ruby devs save money and time. 8 dynos vs 4 dynos is a hell of a big difference when you're starting out. Also, seems like something that would be pretty fun to do if you worked there.

<comment><USER29> Yet they are spending a crazy amount on hosting with Heroku when they could be doing it themselves and get cost savings plus know the entire stack.

<comment><USER13> TL;DR - They've said repeatedly that they would rather work on their product/projects than have to deal with all of the details themselves.

The crazy amount they're spending should give you an idea of how much throughput they have and what would be required for their own setup. That means a lot of design, additional time implementing, more time updating to new tech (including on the software side, in gems, etc), then in the end, ongoing maintenance.

There's probably a lot more that I'm not mentioning as hey, like them, I'm not interested in spending my time developing, purchasing and administrating my own hosting systems, either. I have apps to write.

<comment><USER29> But it seems like they had to deal with the details themselves, plus they are paying a crazy amount of money. Seems like a lose/lose situation for them.

Also the amount of money really gives no indication of how optimized a site is or how much traffic it gets. I've seen sites that are in the top 500 spending ~3K/mo on hosting and sites that get 60K uniques a month spending 2x that.

</comment></comment></comment></comment><comment><USER30> Thank you so much for forcing Heroku to confront this issue!

We've been seeing strange delays and optimizing based on New Relic for a long time... and whenever we reported this to Heroku, they would not admit to an issue.

We ended up using threads (on cedar stack) to get more concurrency per dyno.

</comment><comment><USER21> Wonder how all of these people are feeling right now..

<http://success.heroku.com/>

<comment><USER31> "Rap Genius became a cult phenomenon and scaled from zero to ten million users with ease thanks to the Heroku process model."

Good for at least one chortle.

</comment></comment><comment><USER32> "Explain Now, as Rap Genius is widely known for its expertise in queuing theory" Is this true, or are they being sarcastic that if they could do it Heroku really should've?

<comment><USER23> There's actually a text annotation that explains that, demonstrating how their platform works.

<comment><USER32> Thanks Steve, when I wrote the question there wasn't one...

<comment><USER23> Ah. No worries. :)

</comment></comment></comment></comment><comment><USER33> Ironically, it's possible to get a huge gain over purely random load balancing by examining just two queues at random -- essentially, you should always be doing this since the cost is O(1) and the improvement is large.[0] This doesn't require any distributed locking and at least would qualify as "intelligent" routing -- probably the bare minimum needed to justify that marketing label.

Oh, and it also scales incredibly well. Like I said, there's no reason not to use it over purely random load balancing.

[0] [http://www.eecs.harvard.edu/~michaelm/postscripts/mythesis.p...](http://www.eecs.harvard.edu/~michaelm/postscripts/mythesis.pdf)

<comment><USER34> Wow, the gain from using two choices is indeed impressive! According to section 3.6 (page 72): ``[For a system with 500 servers, i]n equilibrium, the expected time a [request] spends in the system given one choice ( _d_ =1) is 1/(1-λ) {where λ is the average request rate}. Hence ... when λ = 0.99 the expected time in the system when _d_ = 1 is 100.00; with two choices {when _d_ =2}, this drops to under 6.''

</comment></comment><comment><USER35> Heroku should have done something about the issue earlier, but it seems like the problem was just poor prioritization/time management on their end. Yes, these posts got them to finally get moving, but i wonder if perhaps RapGenius could have had the same effect by continuing to bug them privately in the same unyielding manner, instead of going public with it so quickly. That would have allowed Heroku to have focused their energy on fixing the problem, rather than upon worrying about PR and class action lawsuits.

Also, on the topic of lawsuits, how many small startups will go out of business if they get hit with a class action lawsuit every time their documentation accidentally diverges from reality? In this case, RapGenius is small and Salesforce is big, but the legal system will apply the same standard when the plaintiff is big and the defendent is poor. If this becomes precedent, then soon we will have lawyers trying to treat any public post by company employees as 'documentation', forcing startups to have a policy of not allowing their employees to freely help others with their product in public forums. Also, any small startup with a large competitor will have the large competitor paying people to sign up for the product with the sole intent of finding a bug in the documentation so that the small startup can be sued out of business.

<comment><USER2> I'm the guy who mentioned lawsuits.

> _Also, on the topic of lawsuits, how many small startups will go out of > business if they get hit with a class action lawsuit every time their > documentation accidentally diverges from reality?_

I was chatting with a law academic of my acquaintance; her specialty is torts and in particularly, remedies to torts. We discussed some of the different sorts of actions you could bring.

Heroku in their Terms of Service 11.1 have language that basically says "we can change our technicals without telling you". And that's very reasonable. It would be impossible to test every minor change with every client's application. It would also be very annoying for clients to get dozens of emails per day of the form "Updated foolib-2.5.3-55 to foolib-2.5.3-56a".

But torts are a different beast; they live outside and alongside contracts. She could tell me what actions you could take in Australian torts law. We have a tort for "misleading and deceptive conduct" [1] which would probably be whistled up for this case, given the magnitude and length of the divergence, but that particular tort seems to be an Australian-only innovation. US law has, she told me, a lot of unique features that she hasn't studied very closely. Heroku's ToS requires all disputes to be settled under California law in Californian courts.

> _Also, any small startup with a large competitor will have the large > competitor paying people to sign up for the product with the sole intent of > finding a bug in the documentation so that the small startup can be sued out > of business._

Heroku also has ToS language to deter anti-competitive behaviour; not to mention anti-competitive practice laws. Plus, interference with a contract is itself legally troublesome.

Generally speaking, if you can think of a HUGE GAPING PROBLEM in the law, the lawyers and judges have already thought of it and closed it. Usually hundreds of years ago.

[1] There is also in Australian law a _statutory_ offence of misleading and deceptive conduct, but that would not lead to remedies for Heroku customers.

Of course, I am not a lawyer and this is not legal advice.

</comment></comment><comment><USER36> I agree that Heroku's response is pretty unbelievable and their engineering choices very suspect. Reading the email chain between Tom & Adam really drives home how badly this has been handled by Heroku.

Heroku is massively crippling its own product with random routing. Other cloud providers have been able to get this right, and Heroku very obviously knows what kind of applications are running on its server (e.g. deploy a Rails application, Heroku says "Rails" in the console). It would not be difficult to apply different routing schemes for each type of application.

Given that this has been going on for years now, Heroku is either acting with pronounced malice or incompetence. Any competent engineer would not be satisfied with switching the routers over to random and calling it a day. How could that have possibly been approved, then remained for years? They must not have realized what a grave mistake it is.

The #1 thing they should be doing _right now_ (aside from damage control) is to move the routers over to round-robin routing. Random is the most naive scheme possible and is laughably inappropriate for this situation.

See for yourself using this simulator: <http://ukautz.github.com/pages/routing-simulator.html>

</comment><comment><USER37> To what extent would using something like Amazon's ELB mitigate this sort of issue in a bring-your-own-cloud approach? Completely?

I've been looking at using something like Cloud66 and an ELB to move off of Heroku.

<comment><USER38> I don't think it would help at all, as ELB doesn't intelligently route either.

<comment><USER37> My understanding was that you can specify how the "health check" works for each EC2 node, so if you can report that a particular node is having issues, ELB can observe that. But upon further reading, you're right: you can't do anything other than report that a node is "up" or "down", it seems.

<comment><USER39> well. I'm talking out of my ass here, but I have this idea:

Lets say you set the health check to check an URL of your app that maps to a very, very cheap rack app that does nothing but return some really short string.

Then you set the health check timeout to a very, very low value (like 5-10ms or something).

Now all hosts that don't respond within that low timeout are seen as down with no requests routed to them.

So if that one node that is able to process one request is busy, the health check will fail and the request will be routed elsewhere.

This is a very poor mans solution with some drawbacks:

1) there's still a race condition here between determining that the host is up and sending a request to it, so you might still end up with a request being queued.

2) now you are practically doubling the latency between the load balancer and the app server

3) you are creating quite a bit of load to that small rack app which might have a negative impact on the overall performance.

So in the end, this might be a very bad idea (I didn't think this through fully and I'm not in a position for trying it out), but it might also be a stop-gap measure until the problem is solved for real. Maybe worth trying this out in a staging environment - or with a percentage of hosts.

<comment><USER40> ELB health checks have a minimum interval of 2 seconds.

Also, when you hit the threshold for consecutive failures (2 or 3), the ELB immediately closes all connections to that backend without waiting for their responses to finish.

</comment></comment></comment></comment></comment><comment><USER41> This is big stuff...

Sorry to see Rap Genius investing all that money in New Relic, I can't really imagine being on their shoes.

I would be so pissed.

PS: Heroku user here

</comment><comment><USER42> Am I the only one who read "crap genius"?

</comment><comment><USER43> Holy crap - over $60k to get app performance graphs! Wow - that is super expensive!!

</comment><comment><USER44> This is unrelated, but every time I see that domain my mind thinks it's either rapegenius.com or ragepenius.com. Surely I can't be the only one?

<comment><USER45> This comment seems to be made every time rapgenius.com is on HN, every time it gets downvoted. No you're not the only one who thinks that, but why does it even matter.</comment></comment>


NYSE: no technical problems during plunge - <AUTHOR> http://finance.yahoo.com/tech-ticker/nyse-no-technical-problems-duirn-plunge-480290.html?tickers= <comment><USER2> Could it be possible, that a collaboration between two organizations, one entering the bid or ask at 0.01 and another quickly buying it or selling it could be arranged such that the particular trade could go through and could trigger lots of stop losses?

You can submit a trade at any amount. During high volume episodes, you'll see trades go through that are vastly "out of whack" so to speak.

It could be a mistake or it could be a strategically planned incident to cause this.

By my math, it doesn't really take a huge amount of money to move a stock in a particular direction. If you spend a couple million buying a stock at high asks, then the value of the underlying market cap of the company could climb by billions in a single day.

There are companies who trade large orders in the millions of shares over long periods of time to avoid pushing the stock too hard in the short term. If you _wanted_ to push the stock hard in one direction, knowing you'll reverse that action with more money, it could be, and actually _is_ lucrative. Companies do it all the time. This one was more noticeable, but not out of the realm of possibility.</comment>