How many deaths make a disaster newsworthy? - <AUTHOR> https://ourworldindata.org/how-many-deaths-make-a-natural-disaster-newsworthy <comment> GuB-42 What is interesting is the Touhoku earthquake and tsunami and the resulting Fukushima nuclear disaster.

Which one is the best known? In the west, I'd say Fukushima. And yet, the nuclear disaster caused zero direct fatalities. The consequences, the evacuation in particular, killed maybe a few hundreds. That's definitely not insignificant, but to think that it overshadowed an earthquake and tsunami that killed more than 15000...

<comment><USER2> Japan is basically an honorary European nation as far as the West cares. They kind of earned it. Ever since they realized they were behind in the 1800s they've done everything the western nations have done (good and bad) and done them as well or better.

<comment><USER3> In one sense, this comment...

is holding Japan in a great light

but also looking down on it

as though Europe is in a position to confer honor

a Western point of view

<comment><USER4> whether you like it or not the world still revolves around the west. even if it is less so than in the past, militarily and economically the world is led by the west.

<comment><USER5> > _whether you like it or not the world still revolves around the west_

People in the west like to think that.

After one has lived for a while in an Asian, African, Latin American etc country, they get to see that billions of people, the 80% of the global population, could and don't give a fuck about the going's on in the west -- except when they're forced to deal with it (e.g. militarily).

E.g. the Chinese might care for selling to US/Europe -- but that doesn't mean they also care for our beliefs, history, philosophy, moral outrages, outlook on life, charts, stars, concerns, or any other such thing that we take as some kind of "physical law".

<comment><USER6> No way.. I have spent the last 6 years living in Mexico, Colombia, and Japan. Our culture dominates, absolutely. They strive, especially in Mexico and Colombia, to be like us in the West.

<comment><USER5> If you mean in trying to make money as well, have luxuries like in the West and things like that, that has been a global concern since before West and East were met.

Even so, Mexico is next to the states and has some millions of US immigrants and wannabe immigrants, and still -- they have their own culture, morals, etc, and the typical western concerns are not their thing.

So, I was talking more about the cultural and societal aspects. They'll still watch some popular Hollywood movie or have teenagers listening to Cardie B, but they live in their own, different, world, as oblivious to the West as a Oklahoman farmer or a NY gallery owner is oblivious to them and their world.

<comment><USER6> You should do some traveling to major cities in Latin America.

</comment></comment></comment></comment></comment></comment></comment></comment><comment><USER7> Is no one else going to say this? Fine.

Most so-called "journalism" is in fact a rush to capture as many eyeballs as possible for the purpose of displaying advertising. These patterns make a lot more sense once viewed through the adtech lens: news outlets aren't publishers or providers of information; they're sellers of an audience to advertisers. This is their primary mission, fueled in turn by the desire to self- perpetuate.

<comment><USER8> How do state broadcasters fit into this theory? The CBC and BBC produce content in a similar style to the for-profit broadcasters (better, but definitely similar in more ways than its not), but without the advertising mission.

<comment><USER9> State broadcasters have a larger surface area offering a more political goal. After all, it's state-run.

Most people can see this effect more clearly on the state-run news outside their own country.

<comment><USER10> Not always - the BBC was mentioned above and they are far less politicised than many of the other U.K. news outlets. It is argued that any bias comes from the demographic of those who want to work there rather than political pressure from above. A good local news source that is government funded, Radio New Zealand, would fall into the same catagory in my opinion. [https://en.m.wikipedia.org/wiki/Criticism_of_the_BBC](https://en.m.wikipedia.org/wiki/Criticism_of_the_BBC)

</comment></comment></comment></comment><comment><USER11> The newsworthiness of disasters has nothing to do with the number of deaths. It has to do with how dramatic and visually interesting the circumstances were (volcanoes and earthquakes are more dramatic than droughts and famine) and, at least for American media, whether the victims were American. Bonus points if the victims were children.

<comment><USER12> While I partly agree, I remember examples such as the terrorist attacks in Paris where it was covered on every media for a long period in time, and a few months later there was a mass killing in Afghanistan with way more deaths and obviously was done in the same dramatic way but most media didn't even cover it. I'd say it has to do with how unexpected is that event to happen. Same applies to people's grief. Everyone had some sort of support for paris up, or even with the newspaper attack ze suis etc, noone cared about what happened in Afghanistan. :/

<comment><USER13> This is news in general. If it's not "new" it's not "news".

<comment><USER14> I don't think you looked at parent comment closely enough.

</comment></comment></comment></comment><comment><USER15> There is obviously good amount of relevance (of all shades, be it political, regional, communal etc) affecting reporting, and discretion applied that skips certain events, but I want to say something tangential.

To me, it appears that the modern epidemic of equating live TV coverage where reporters blabber inconsequrentialities with an event being in the news is the cause for the feeling that the article espouses. Doesn't a 60 second mention about an event suffice? The larger point is, how does this 'coverage' really help the cause? Surely, reporting a disaster, with brief visuals, will alert the volunteers to do their bit for humanity, but showing private human emotions for days and days is something I never understood. It can't be anything but politically motivated propaganda. It also smacks of commodification of victims for a sort of depraved entertainment value. I often feel that a disaster that suits a tv station's agenda is a relief to them because they don't have to worry about programming and can simply milk it till they get bored.

One could argue, correctly, that when an event is reported only briefly on tv it doesn't remain in people's memory. This is where I favor newspapers to tv news. Although there is still a bias here too - an event being reported on front page, with pics etc, or in the fifth page column 2, or nowhere at all - it is relatively indelible. The cliched Hollywood character that pins newspaper clippings to remember an event sort of corroborates what I am saying.

That said, I must say that I am not a fan of all these award winning gut wrenching pictures of human tragedies because I refuse to believe that the photographer is not manipulating. News via visual media is always less trustworthy to me than news via written word. Atleast with the latter, we can sense the tone, inclinations, sophistry etc which is extremely hard with visual medium. For instance, every documentary looks manipulative to me, so I never value news disseminated via visual medium, except for some events like Hindenberg disaster, Tsunamis etc which can't be described in words.

<comment><USER16> Its not the death count, type of disaster or sympathy for similar/popular. Its novelty of news. A disaster that is unique, novel or unexpected will get the attention of news readers.

A terrorist attack in a third world will not get the same attention as a tsunami or mudslide(because the later occur less often) and much less than a new disease that kills a dozen people. There is a subconscious risk-estimating going on "is disaster X relevant/close/threat to me", but the novelty of new information is the key for news coverage and subsequent interest.

An easy way to increase novelty is to add something unique: instead of 20 people died in X, add their names and their occupations, how they died, in- depth material/interviews - all things that increase engagement and empathy.

A simple news broadcast that searches for scoops and immediate facts is quite dry and non-appealing, like a weather report for most people. People better absorb news in format of something similar to opinion piece/entertainment, regardless of its accuracy/neutrality.

All coverage is building engagement. People relate to the event from more angles. Emotional connections form, sympathy and sense of relatedness. To some it might be a soap opera, but it also relieves their personal fears and problems - in a sort of twisted escapism, like watching disaster movies or horror films.

<comment><USER17> Power=Reach x Persuasiveness according to Scott Adams, a change from the old Power = Money x Willingness to do evil. By this, reach increases as a function of novelty and existing reach, if we can take fore granted that humans "like" shiny. Ie. the coverage is rational self interest.

</comment></comment></comment><comment><USER18> What is missing is how relatable the victim of the disaster is.

Self > Family > Neighbor > City > State > Country > Region

... also in a different direction, race, gender, economic status, shared culture, etc.

Something along the lines of "how much the victim reminds me of myself". In other words the trolley dilemma.

In some senses it might be morally objectionable to value some people more than other people, but in other senses it makes sense. Do you care exactly as much about your child as you do about the other 2 billion children in the world? Would you forgo feeding your child in order to feed someone else's? Of course not. It is a difficult subject.

<comment><USER19> You can add "species" at the front of your second list.

</comment></comment><comment><USER20> In 1990 around 170,000 Kashmiri Hindus were pushed out of Kashmir, the eventual number over time became 400,000 but it's estimated over 2000 were killed by Muslims in Kashmir. This was in the news but hardly made it to regular headlines like the Rohingyas did, even the aftermath is reported but hardly in the same way. The reason? Because Hindus are he majority in India, however in Kashmir they are the minority. My point is that sometimes it's about preconceptions about the media as to who matters and who doesn't rather than the number of deaths. There is a value on a human head, sad really.

</comment><comment><USER21> i remember hearing somewhere that UNICEF determined that contribution rates were maximal if they showed a poor little boy in a starving nation, or a poor little girl, similarly. But if they showed the boy and girl together the contribution rates went down significantly. A conjecture was the more numbers of people in despair there is, the less empowered you feel to help.

<comment><USER22> What about an alternative conjecture - they can help each other?

</comment></comment><comment><USER23> This article seems to either miss the point or deplore the reality that the news media is an entertainment business, not an important information dissemination service. What gets "on the news" is what will get eyes on the commercials/clicks on the ads, in the judgment of the editors.

</comment><comment><USER24> I've always dreamt of a news services that serves structured and data driven news. E.g. each news report contains attributes about the number of casualties, geographical area affected, estimated economic damages, the period of time and the type of event (e.g. crime, military operation, statement, weather, natural disaster, legal, election outcome, public gathering, trend/speculation etc) and the level of reliability (verified, reported, alleged etc). The article body is secondary.

<comment><USER25> I believe something like this is valuable to insurance agencies and quite likely also hedge funds and investment banks.

You could even hand out free access to NGOs and have a super-nice landing page with a world map with circles in different colors and sizes to get an overview of what's going on.

Finally, I think _I_ would love it, too, though I am not sure whether I'd pay for it. Also, expect to be censored, so China is probably not a market.

If you manage to do it, I think it's a valid business model.

Also, it'd be a great counter to biased media and, thereby, a service to the world.

<comment><USER26> I wonder if there's something amounting to this for a Bloomberg terminal.

</comment></comment></comment><comment><USER27> This reminds me of a supposed quote by Joseph Stalin:

"A single death is a tragedy; a million deaths is a statistic."

<comment><USER28> Or

"Kill one man, and you are a murderer. Kill millions of men, and you are a conqueror. Kill them all, and you are a God" -Jean Rostand (and Megadeth :))

<comment><USER29> Should be ... "You were a god"

</comment></comment></comment><comment><USER30> > "I tell people that if it's in the news, don't worry about it. The very > definition of "news" is "something that hardly ever happens." It's when > something isn't in the news, when it's so common that it's no longer news -- > car crashes, domestic violence -- that you should start worrying." (Bruce > Schneier)

</comment><comment><USER31> The ranking looks like it matches the frequency of occurrence pretty closely. So it's not surprising news outlets report on deaths that happen in uncommon ways even if a much larger number of people just died from a more common cause of death.

</comment><comment><USER32> Deaths per mile is the metric used by many journalists. 1 death in your neighbourhood matters more than around 10 deaths 100 miles away. I don't think the relationship is exactly linear.

</comment><comment><USER33> I mean, I guess this is not too surprising. There is that well-referenced Schneier article about how we react most to risks that are unfamiliar and rare (overestimating and valuing them), while believing as low risk those things which are mundane and common but much more likely to kill us day to day.

General hunger is familiar to everyone, even though dying of it now should be immoral to allow to happen. But being killed by a volcano is new and scary.

Hence the data they observe.

<comment><USER34> You could say the same about bias on contentious issues: "tribalism is an evolutionary trait, so such-and-such a news outlet's tendentious coverage isn't surprising".

Or about poor software design: "Concrete thinking is natural, so it's not surprising the engineers didn't think through possible surprising codepaths".

Journalism aspires to be, at its best, a profession whose associated skills, standards, and ethics exist precisely to countermand the kinds of 'unsurprising' traits you mention. As with reducing bias, application of rational news values is never going to be perfect. But like other skills, it can be improved.

</comment></comment><comment><USER35> It's not a question of numbers, but relevancy. I would say that a disaster in Central Europe and North America will always get to the news, even if it didn't cause a large number of deaths. If it happens anywhere else in the World, it takes a lot more to be newsworthy.

I don't agree with this binary view of the world, but that's how it is. As a European.

<comment><USER36> What do you mean when you use the word "relevancy"?

<comment><USER37> Probably proximity to home country mixed with the number of victims that are of proximal origin to ones own country/region.

<comment><USER35> Yes, relevancy as proximity to our country/region. I think I would first hear about a disaster that killed one single person from Portugal (as a Portuguese myself) anywhere in a world than one that killed fifty from any other nationality (outside of Europe and North America).

That would be different for people around the world according to their country and region.

</comment></comment></comment></comment><comment><USER38> It's about relatability, these stats just show how is population of these areas relatable to Americans. Not surprisingly rich Europeans are the most relatable, followed by poor S and C America, but still mostly same race and cultural background and then there is big nothing with vast cultural gap, different race etc. with Asia and Africa. I just don't understand Pacific, especially if it includes AU/NZ they should be just next to Europe, but maybe nothing really happening in AU/NZ besides ocasional earthquake and the news are overshadowed by less interesting unknown countries.

</comment><comment><USER39> It doesn't even take one death from a volcano, apparently: [https://www.reuters.com/article/us-hawaii-volcano/hawaii- rep...](https://www.reuters.com/article/us-hawaii-volcano/hawaii-reports- first-serious-injury-from-volcano-idUSKCN1IK0VH)

</comment><comment><USER40> its not about the number of deaths, its the nationality of deaths that most news stations use as a barometer of reporting. The British press gave coverage of the 2004 Tsunami but that coverage increased as a by-product of the news channels discovering British nationals being in the affected areas. Then it suddenly became relevant. Otherwise its "just" another Syria/Yemen/Darfur/Burma.

</comment><comment><USER41> IIRC in The Tin Men by Michael Frayn (1965) the computer used the number dead divided by the distance away to determine newsworthiness. Using some of the observations here I am sure we could improve on that.

It is gut wrenchingly sad that every death is not treated equaly ... though much in the same way, every life.

</comment><comment><USER42> With regard to famine: the word "news" means "new things". It is literally the plural of "new". Famines are typically long-term events that have built up over time. Famine in (parts of) Africa? Old story. Famine in New York? New, and therefore "news".

</comment><comment><USER43> It isn't the number of deaths, rather, it is the imminance of the predicament. The desperation caused by the situation. Single astronaut dies during space walk, big news. Epidemic, famine, these are slow processes that don't immediately invoke the fear reflex or curiosity as much.

<comment><USER44> Yeah, I remember reading about this in PSYC 101. The idea is that scale of tragedy is proportional to people's perception of how "avoidable" it was.

</comment></comment><comment><USER45> Depends on the circumstances. One person dying on a commercial airliner recently became very newsworthy.

<comment><USER46> Context?

<comment><USER47> i believe they're referencing the southwest airlines flight in which an engine had an explosive failure and the shrapnel from that explosion went through the plane's main compartment windows, killing one passenger and maybe injuring a few more, i cant quite remember

<comment><USER48> What probably gathered the most attention was that the woman by the window was being sucked out but other passengers helped her by pulling her. I don't know if it was she who died. It wasn't mentioned in the article.

[https://news.ycombinator.com/item?id=16860902](https://news.ycombinator.com/item?id=16860902)

EDIT: This makes me think that part of what makes news newsworthy is describing something in a way that provokes the readers to put themselves in the story. In this case it's terrifying to consider being in the shoes of that woman. Also, per GuB-42's comment, I think on average people fear nuclear fallout more than an earthquake or a tsunami, considering that most of us have experience with none of those scenarios.

<comment><USER49> She was the one that died. [https://www.theguardian.com/business/2018/apr/17/philadelphi...](https://www.theguardian.com/business/2018/apr/17/philadelphia- plane-emergency-southwest-landing-engine-explosion-latest)

</comment></comment></comment></comment></comment><comment><USER50> What's important with news headlines is if it makes it feel like it could have been you.

</comment><comment><USER5> It depends. US/European or developing world deaths?

Also, if it concerns developing world deaths, regardless of their number, are they in an area where US/Europeans have strategic interests, and can they exploit them to further them?

<comment><USER51> is more of a black / white thing sadly. Well white / other really. Probably a 1 to 100 ratio. If not more. 10 white deaths is more news worthy than a 1000 non-whites.

</comment></comment><comment><USER52> It'll be interesting to see man made disasters like a shooting or a terror attack.

How many people die in the Middle East from a terrorist before it makes news? Same goes for a shooting in USA.

How many get shot before it makes national headlines.

<comment><USER53> I could totally be wrong about this, but my impression is part of the reason terrorist attacks in the Middle East are less visible in American media is they don't fit the narrative of the rest of the news. Classically there are supposed to be two native groups in the Middle East: Terrorists with a political agenda and unworldly innocents who just want to raise their olives and apricots. The terrorists are supposed to be rising up against the worldly and technologically advanced West, so it doesn't make sense for them to be blowing up the cafe on the other side of town.

<comment><USER54> I'd rather say that it's about the relevance of the conflict to local readers. Middle Eastern terrorists harming some westerners (no matter if it happens in the west or to some tourists there) matter because it's part of an "their tribe vs our tribe" conflict; Middle Eastern terrorists harming a lot of locals does not matter as much to the local reader because it doesn't involve "our tribe" at all, it's just their own internal affair or civil war or even genocide.

News is not about reporting what happens, it's about reporting what's relevant - the events that can or will influence _us_ in some way, so whether something is newsworthy depends on the audience. A limited conflict that affects the gas price is more relevant to local readers than a deadly conflict with little international influence.

</comment></comment></comment><comment><USER55> One if your name is Tesla.

<comment><USER56> Elon couldn't keep it together through an earnings call, of course the media is going to go after him if only just for the reaction

</comment></comment><comment><USER57> Interesting question. It'd be interesting to train a regression model to answer this question and help the media decide what to show or not depending on the predicted impact.

</comment><comment><USER58> Tesla made a broken ankle newsworthy. So there is one data point.

<comment><USER56> Tesla also has a CEO that the media knows will react. Musk is the elementary school child who keeps crying when the bully comes around. What makes Tesla newsworthy is the guaranteed reaction. Elon can't keep it together for an earnings call so of course they are going to keep poking him

</comment></comment><comment> amiga-workbench Totally depends on the prevailing sociopolitical narrative.

</comment><comment><USER59> Only 1 needed if it's in a Tesla.

</comment><comment><USER60> 1 if it's from a shark

</comment><comment><USER61> How valid are the statistics, or rather the conclusions, for some of the (maybe unspoken) claims on some charts? Take the last one, "How many deaths does it take for a disaster in different continents to receive newscoverage?". This suggests that disasters in Europe get reported on much sooner than those in Asia in Africe; which I believe, I'm just wondering to what extent we can conclude that from this data. But it also (maybe only in my perception?) seems to make a moral judgement on that - that that is unfair, biased, discrimination, or just overall morally objectionable. Many comments in this thread seem to focus on that point, or just take it as a given.

The op says for this specific chart 'Note: This data is controlled for several factors, including the number killed and affected, country, year, month, and type of disaster.'. There are no famines in Europe (ok maybe depending on definitions and time period, there were some in the 60's and 70's in former communist countries?), which make the largest number of casualties by far. So how is it 'controlled for' by disaster type? When you look at 'data' for that chart, it doesn't even mention 'disaster type' under 'controlled for'! So what is it?

Furthermore, this chart suggests that every disaster in Europe with at least 1 casualty is reported on. That seems, let's say, rather unlikely. So is this data normalized with Europe as the base? (I'll admit that I only skimmed the actual paper). But that would be weird - what is the point of this graph? What could it show, and what _does_ it show?

Look I'm not saying these people (i.e., the authors of the original paper) deliberately misrepresented anything, or that they set out to prove a point. It just looks more like a 'hey we have a cool idea for what we can do with data, let's make regression models, it'll be fun (and probably at least somewhat informative to others as well)' than some fundamental inquiry into the morality of news reporting. And that's fine - much (dare I say most) published papers have 'only' that level of depth; I've done it myself. But my problem is the big jump from numerical relations between some 'data' and broader conclusions (not by the authors, usually, but by people reporting on it, like the OP).

In this case, the numbers support the intuitions most people would have. I'm not even doubting the results, or the overall trend or bias. What I'm wondering is to what extent we can claim that this sort of analysis actually 'proves' those intuitions. Are so-so analyses, and hand-wavy generalizations made from them, better than having none at all? I used to think 'math is factual and any data that is not completely bogus is better than none at all', but nowadays, I'm not so sure any more.</comment>


Twitter Co-Founder Ev Williams Is Leaving the Company's Board - <AUTHOR> https://www.cnbc.com/2019/02/22/twitter-co-founder-ev-williams-leaving-board.html <comment><USER2> From Ev:

"Thank you, @jack and @biz for starting this crazy company with me—and continuing to make it better and better."

Do you think he really means that? Has Twitter changed at all in years? It seems like the service is objectively worse than it used to be (no 3rd party apps, no chronological timeline...).</comment>


My url isn't your url - <AUTHOR> https://daniel.haxx.se/blog/2016/05/11/my-url-isnt-your-url/ <comment><USER2> It seems nobody really understands how to handle non-ascii URLs. To get it right for Scrapy (i.e. do same as browsers) we checked how several browsers behave; the results was consistent but surprising, if not crazy. Before sending an URL to a server (e.g. when user clicks a link) browser does the following:

1\. Url is parsed into components.

2\. Domain name is encoded using punycode.

3\. URL path is encoded to UTF-8 and then percent-escaped.

4\. Query string and fragment are encoded using _page encoding_ and then percent-escaped. For URLs entered to address bar it seems that UTF-8 is used, not page encoding.

5\. The result is joined back to a single byte string and sent to a server.

So this means parts of a single URL are encoded using 3 different encodings, and one needs to know the encoding of a page URL was extracted from to send it correctly (i.e. to send it the way browsers do). I haven't seen this algorithm stated explicitly anywhere, but this is how all browsers work. It doesn't make any sense.

<comment><USER3> > For URLs entered to address bar it seems that UTF-8 is used, not page > encoding.

Maybe I'm misunderstanding, but how could a browser use page-encoding before the page has been fetched?

<comment><USER4> That's how: [https://html.spec.whatwg.org/multipage/syntax.html#determini...](https://html.spec.whatwg.org/multipage/syntax.html#determining- the-character-encoding)

<comment><USER5> And how would you deal with an URL to the HTML itself?

<comment><USER2> There are 3 ways to get an URL:

a) URL is entered to the address bar. Default encoding (it seems UTF-8) is used for query and fragment. b) User clicks on a link. Page URL is used; page encoding is already known. c) Redirect to a new URL happens. I can't say out of my head what happens in this case.

<comment><USER6> This is just a recipe for massive confusion though.

You're saying if I click on a link with an IRI in a non-UTF8 html page, it means one thing.

But if I copy and paste that link into a text file, and later paste it into a browser, it'll mean something else entirely? Or if I save it to a bookmark using some service?

<comment><USER2> Yeah, it is a recipe for confusion. Browsers use a workaround: when you copy an URL of a link (using "Copy Link Address" in Chrome or "Copy Link Location" in Firefox) it copies escaped URL, not original URL.

</comment></comment></comment></comment></comment></comment></comment><comment><USER7> This seems to fall into the common trap of misunderstanding the causality in standardising legacy features of the web platform. The post says:

> The [WHATWG] spec says so because browsers have implemented the spec.

But that's exactly wrong. The WHATWG spec is based on the observed behaviour of browsers. By documenting what existing browsers do, and scoping out who is prepared to change their implementation when there are differences, it tries to specify a behaviour that is either already implemented or able to be implemented with as few changes as possible. It is _possible_ to specify behaviour that is stricter than any current implementation, but the burden of proof for doing so is non-trivial; you have to demonstrate, to the satisfaction of all involved parties, that the proposed change will not break enough web content to cause a compatibility problem. Doing that is non- trivial, but possible if you are sufficiently motivated.

The complaint that the WHATWG spec is written by a closed group also seems like a misunderstanding. Compared to pay-to-play organisatons like the W3C, or pay-to-meet organisations like IETF, there is no concept of formal membership, or need to pay to get your voice heard. There's a mailing list, an open GitHub repository and an IRC channel. Anyone, especially prospective implementors of the spec, is welcome to participate in those forums, and they will have the same privileges as any other contributor. I imagine that for URL in particular, contributions from the CURL comminity would be most welcome.

<comment><USER8> It is good to have clarity on how WHATWG is working. Thankyou! Sounds like an HTML5 style treatment of the problem.

I'm not clear though why the effective starting point is the _superset_ of all variants currently implemented by one of the involved parties. Given that the extant standards are quite strict, couldn't an argument be made that the "burden of proof" would be on why these needed to be extended, on a case by case basis?

Regardless of the particular agreements reached, an up-to-date standard will certainly benefit everyone; it seems like it would be an excellent outcome if the standard came with a reference implementation too :-)

<comment><USER9> Browser vendors will be hesitant to remove any special variant that might break existing websites.

<comment><USER10> The number of websites with this issue is so small that in my 9 years of crawling the web, I'd never noticed the issue.

<comment><USER11> Why do you think you would notice it? I imagine the only giveaway would be the url in status bar when you hover mouse over the link. If you are a programmer (or very pedant person) you might notice it, otherwise probably not.

Also, I hope they acted on some real-world statistics, provided by Google's or other crawlers...

<comment><USER10> I was the CTO of a web-scale search engine, and now I work on the Internet Archive's Wayback Machine. I'm not referring to things I noticed as a web end- user.

</comment></comment></comment></comment></comment></comment><comment><USER8> When the robustness principle meets the tragedy of the commons, good people get driven to the brink of madness.

There are surely arguments for allowing any old garbage in URLs/URIs/IRIs - hey, it's easy and fun! - but, gee, it's not like bunging an identifier through a URI escape function is going to triple the code base.

Compare that with the surface area for potential bugs in the parsing code which somehow didn't account for several megabytes of whitespace embedded in a URL or multiple code pages in a single string or whatever zalgoesque horror is showing up on the security bulletins this week. It's a lot easier to proof code that only needs to deal with a strict subset of 7-bit clean ASCII and can politely decline embedded emojis.

When one web client starts being overly zealous in what it accepts, that puts an implicit onus on everybody else to start accepting that too ("all browsers do"!). Where would you draw the line? Well, we've got a couple of RFCs lying around, how about we go with those.

This stuff doesn't have to be hard. Surely the act of issuing an HTTP redirect isn't the Last Great Unsolved Problem of web engineering! I say rejoice in the beauty of URIs in canonical form. Be miserly in what you accept for a change.

<comment><USER12> Most of the world is outside the (English-speaking) United States of America and doesn't speak an ASCII language. That's why Unicode exists - couldn't we all just use it? That's what the IRI RFC says in essence. EDIT: Perhaps I misread and you're saying we need two layers like the RFCs intended it: the user-visible Unicode IRIs and the protocol-level ASCII URIs. cURL lives between these two and arguably would be simpler if there was no such separation and everything was Unicode to begin with.

<comment><USER8> Agreed. I love me all the Unicodes, all the time! For typing, for display, for whatever internal app logic, variable names, the names of children etc.

The IRI RFC does however specify that for transmission/interchange the IRIs need to be percent-encoded to form valid URIs. That's pretty straightforward! A good user agent will fully transcode whatever characters I type in the address bar, generating a valid IRI. A bad user agent will complain and say that whatever it was I typed was very nice but not a valid interweb, and I have a bad day. A broken user agent may generate some broken encoding (or just pass on whatever I typed in UTF-8 or CP-1252), and some web admins may have a bad day, if they weren't already.

But the location bar is not the only place that URLs come from (less and less every day). When a web application (e.g. 301 redirect) or a resource (e.g. external document reference) uses an RFC-3987 noncompliant identifier, just how far should the user-agent bend over backwards to fetch the resource? Some browsers seem to do a lot, but it's not clearly documented just how much (see elsewhere in the comments), and among the article's laments was the observation that when any browser accepts dodgy identifiers, it places a burden on everyone else to accept the same.

Seems that WHATWG hopes to formally codify just how dodgy you can get away with without the identifier being "too dodgy", but I get the impression that this will be an expansive definition ("hey, if they did it, you can do it! we believe in you"). This is an understandable approach, as they are trying to document the state of the web, but it does put extra burden onto anyone dealing methodically with URLs henceforth, rather than being strict and drawing some clear boundaries up for people constructing URLs. I'm expecting to see a lot of "SHOULD" and not a lot of "MUST NOT" :-/

</comment></comment></comment><comment><USER13> Page seems to be struggling.

[http://webcache.googleusercontent.com/search?q=cache:https:/...](http://webcache.googleusercontent.com/search?q=cache:https://daniel.haxx.se/blog/2016/05/11/my- url-isnt-your-url/&num=1&strip=1&vwsrc=0)

<comment><USER14> Mildly amused that the website of the guy who makes cURL wasn't ready for the HN hug of death.

</comment></comment><comment><USER15> It seems to me that much of the article can be boiled down to the fact that the standards are being set by people writing software that

a) has to be usable by users with the lowest common denominator of tech savvy and

b) doesn't have to parse URLs out of other text because it encounters URLs only in defined fields

As a result, this software is written to be lenient about trying to interpret URLs input by the user. So it already has to have facilities to transform URLs with improperly encoded special characters such as spaces. This allows it to in turn be more lenient about what it accepts from servers.

With no clear standard for these transformations and the browser makers not sharing their parsing and transformation libraries, the writers of other software that has to deal with URLs are left playing catch-up and trying to reverse-engineer the browsers' undocumented hacks for dealing with not-very- well-formed URLs.

Moreover, many have the additional burden of having to be able to parse URLs out of other text where the delineation of URL start/end may not be clear. Some of the forms supported by browser makers (who don't have to do this) can't reasonably be supported in this context, leading to frustration for developers (let alone users).

</comment><comment><USER16> There's already a solution in the spec for the slash issue

Either you have a valid scheme or you don't, and either you have a relative path or an absolute path

Double slash after a valid scheme implies an authority, if a 3rd slash exists it's an absolute path, anything past the 3rd slash has to be normalized. So, [http://////](http://////) is actually [http:///](http:///)

If you have an invalid scheme or a relative path such as http:/ or +[http://](http://) it would actually be normalized to ./http:/ ./+http:/

[https://tools.ietf.org/html/rfc3986#section-4.2](https://tools.ietf.org/html/rfc3986#section-4.2)

</comment><comment><USER12> _curl gets confused by non-ASCII letters in the path part but percent encodes such byte values in the outgoing requests – which causes "interesting" side- effects when the non-ASCII characters are provided in other encodings than UTF-8 which for example is standard on Windows…_

I don't think cURL implements this percent encoding yet - instead, it sends out binary paths on UTF-8 locale and Linux likewise. This was recently added in cURL's TODO though: [https://curl.haxx.se/docs/todo.html#Add_support_for_IRIs](https://curl.haxx.se/docs/todo.html#Add_support_for_IRIs)

</comment><comment><USER17> FTA:

> The term URL was later changed to become URI, Uniform Resource Identifiers

rfc3986:

> A URI can be further classified as a locator, a name, or both. The term > "Uniform Resource Locator" (URL) refers to the subset of URIs that, in > addition to identifying a resource, provide a means of locating the resource > by describing its primary access mechanism

Personally, I don't treat errors like that as massively damaging to the credibility of the rest of the article, but it is rather jarring, and I know that people do post "Stopped reading at..." comment about this sort of thing.

<comment><USER6> Eh, I think your comment only confirms the main point of the article, that it's become massively confusing.

I was around around then, and definitely recall the conventional wisdom being "don't call them URLs anymore, call them URIs, that's the new standard." At any rate, rfc3686 claimed to be a new standard for _syntax_ -- according to the passage you quote, whether it's a "URL" or a non-URL URI is not a matter of syntax at all. The syntax spec is now called "URI". That rfc claims.

Except somehow the result has been just massive confusion over what standard should be used how for syntax of URLs/URIs, with the WHATWG just stirring the pot further (obligatory reference to xkcd there's too many standards let's create another). I don't know that there's anything wrong or to blame in the written text of rfc3986, but something in the social practice of rfc standards as manifested in URL/URI-related things, has resulted in massive confusion and a situation described in OP where there is effectively no formal standard for URLs/URIs consistently used across software.

</comment></comment><comment><USER18> Part of the messyness of URLs might stem from the fact that they occur in a lot of different contexts, with very different properties and reserved characters:

\- Somewhere within some unstructured text: Here, finding the exact end of the URL is difficult and disallowing spaces can have a great benefit. Even then you can have ambiguities: Does punctuation at the end belong to the URL? What about URLs that are wrapped in <> brackets?

Whether or not it's even possible that the URL contains non-ascii chars depends on the type of the document.

\- As a value of a HTML attribute without surrounding quotes: Slightly less ambiguous but including a space would break the HTML parser. The parser works fine with unicode characters however, so there is less pressure for authors to encode them.

\- As a value of a quoted HTML or XML attribute, HTTP header or data field: Charset, Start and end of the URL are communicated out-of-band so there is no pressure at all to normalize the URL.

\- In the browser address bar: Start and end are unambiguous but there are other concerns: You have to detect if it's an actual URL or a search term, you have to deal with omitted parts (e.g. no scheme) and people WILL input unicode characters. You also want to display unicode characters to the user, except when you don't want to for security reasons. And then users will copy the URL in the address bar and paste them into an HTML document...

<comment><USER19> I think we should discount _The Widget Formerly Known As The Address Bar_ from these sorts of discussions. The omnibox[chrome]/awesomebar[ff]/frustrationmaker[edge]/etc. is a field where a human can type or paste all sorts of guff, and where the browser can present the human with a representation of the location of the cat video currently playing on the screen (complete with optional scheme, padlock icon, elided path elements, fancy colours and shading, Unicode snowmen, etc.)

Sometimes the guff from the human looks exactly the same as a URL, sometimes it's close enough to work out what they meant, and the rest of the time it's just guff. If it looks exactly like a URL or IRI, it's trivial for the browser to turn it into a proper URI and emit an appropriate request. If it's close, then some assumptions can be made, but the browser is still responsible for constructing the actual URI. That doesn't mean that what the human typed has any bearing on the underlying protocols or standards, or the definition of a URI/URL.

</comment></comment><comment><USER20>
        (and yeah, file: URLs most have three but lets ignore that for now)

I don't want to be nitpicky but actually a file url still has two slashes, the third slash is actually the root path i.e.: file:///home where the two slashes are slash-slash-colon and there third is actually the start of the path. (actually it just emit's the host)

Actually there is one exception, but just ignore this widly 'unused' operating system which doesn't seem to care about posix and rfc's anyway.

<comment><USER19> Actually, no. From RFC 1738, Section 3.10:

       A file URL takes the form:

           file://<host>/<path>

       where <host> is the fully qualified domain name of the system on
       which the <path> is accessible, and <path> is a hierarchical
       directory path of the form <directory>/<directory>/.../<name>.

And then in Section 5 (the BNF):

       ... elements may be preceded
       with <n>* to designate n or more repetitions of the following
       element; n defaults to 0.

       fileurl        = "file://" [ host | "localhost" ] "/" fpath

       fpath          = fsegment *[ "/" fsegment ]
       fsegment       = *[ uchar | "?" | ":" | "@" | "&" | "=" ]

So that first slash isn't part of the path, and the path starts with a directory name. The fact that the root directory in a UNIXy environment doesn't have a name (or has a zero-length name) is a source of much confusion. (Nobody would accept "file:////etc/passwd" as a URL, but that's how I read the spec.)

This is also why Firefox insists that there are five consecutive slashes when encoding a UNC string (a Windows/Samba share path, like "\\\server\share\path\to\file.txt") into a URL (e.g.: "file://///server/share/path/to/file.txt")

And let us not even begin on the fact that file: URLs are specified in an "obsolete" spec, and explicitly allow characters in the path that are disallowed by RFC 3986. I'm working on fixing that [[https://tools.ietf.org/html/draft-ietf-appsawg-file- scheme](https://tools.ietf.org/html/draft-ietf-appsawg-file-scheme)] but it takes time.

</comment></comment><comment><USER21> punycode, that is used to encode unicode urls is also used for support of unicode characters in DNS.

</comment><comment><USER6> The forking of the URL standard has been indeed annoying. It's a standards fail.

</comment><comment><USER22> ... now try handling BIDI IRI

</comment><comment><USER23> > The Web Hypertext Application Technology Working Group

That name alone should tell you that they don't give a hoot about anything other than WWW and HTTP.</comment>


What Comes After Hard Drives? - <AUTHOR> http://www.physorg.com/news175505861.html

<comment><AUTHOR> Link to paper on which article is based: [http://ieeexplore.ieee.org/xpl/freeabs_all.jsp?tp=&arnum...](http://ieeexplore.ieee.org/xpl/freeabs_all.jsp?tp=&arnumber=5257331&isnumber=5256997). Sweet paywall.</comment>


PeerTube – Federated video streaming platform using P2P directly in the browser - <AUTHOR> https://github.com/Chocobozzz/PeerTube/?src=hn <comment><USER2> This is really cool ! Framasoft is doing a great job with their project to «ungoogle the internet»[1].

I really wish Mozilla tried doing the same thing, with their much bigger manpower and communication impact.

[1]: [https://degooglisons-internet.org/liste?l=en](https://degooglisons- internet.org/liste?l=en)

<comment><USER3> Un Google the internet? I hope so but almost every remotely commercial site out there uses Google services.

<comment><USER2> People use Google's services (and other privately owned, and «you are the product», ones) because it's the most convenient way they know about. Sometimes it's because it's the most convenient way (lack of credible competition, self-hosting isn't an option for many companies), and sometimes it's because the alternatives are less known (commercial products based on OpenStreetMap are way better than what google maps offer, but few people know about them).

An organization like Mozilla could make open-source-based and privacy-aware services for many Google products with ease if they wanted to. (I mean, Framasoft is doing it already and it's a really tiny French non-profit organization !)

</comment></comment></comment><comment><USER4> I think for this to be successful you need to encourage users to server the videos. There are a couple things that might help this:

\- Popularity could be based not only on views, but how many total minutes of the video were shared by other people.

\- When someone watches a video, they have to keep hosting it until they share as much as they've watched it (so if they watched 10 minutes, they have to share it until they've distributed 10 minutes worth of the video to other people.

<comment><USER5> Make them share for ~10% longer than they consumed for a more robust network.

<comment><USER6> Most people have terrible upload speeds compared to download speeds. In my neighborhood most people (by choice because they aren't tech savy and don't care) have AT&T DSL with 40Mbps download but 2Mbps upload. This means that for a good quality 10 minute video at 720p with a bitrate of 3mbps they would have to seed for significantly longer than viewing just to get a ratio of 1 without degrading downstream performance.

<comment><USER7> Just to add a datapoint to this, my home service is gigabit down... _10 megabit_ up.

That's 1000 / 10.

<comment><USER8> This makes me want to go test me upload on my gigabit down connection...

</comment></comment></comment></comment></comment><comment><USER9> It would have been cooler for this to use IPFS instead of its own P2P content distribution.

Still cool though.

<comment><USER10> IPFS does not work in the browser yet.

<comment><USER11> D.tube works just fine:

[https://d.tube/](https://d.tube/)

<comment><USER10> D.tube makes the browser download from a gateway (ipfs.io) using HTTP.

<comment><USER12> You can set it to connect to your own gateway.

<comment><USER13> Doesn't change the fact that your browser doesn't connect to the ipfs network

</comment></comment></comment></comment></comment></comment><comment><USER14> Perhaps also needs onion routing to protect users from liabilities.

<comment><USER15> it says it works in browser so you just run it from tor browser

<comment><USER16> AFAIK Tor browser deactivates WebRTC to avoid leaking real IPs, so at least the P2P aspect wouldn't work. Not sure if seeding from the server works without it.

</comment></comment></comment><comment><USER17> awesome, except it doesn't look like there are any incentives to running a server. unless i missed something?

<comment><USER18> Like Mastodon et al, this doesn't need to be built on traditional incentives. The incentives are that you want to share cool videos with people. Any one server that gets overloaded with users encourages others to host another server and build a community on it, and the load (and the cost) naturally balances and is distributed across the whole network.

<comment><USER17> Definitely an interesting digital ecosystem experiment. Hopefully it can sustain itself. Will be interesting to see how it plays out and evolves.

</comment></comment></comment><comment><USER19> Problem is being able to monetize.

<comment> loup-vaillant Monetise what? For whom? The thing is distributed, it's not like the host has to shoulder most of the bandwidth costs.

If one wants to live off their videos, that's another problem, but most videos don't mean to be monetised at all —no ads, no Patreon, no nothing.

<comment><USER20> They can just use Patreon in exactly the same way they do for YouTube.</comment></comment></comment>


Study users by letting them design your UI - <AUTHOR> http://svenfoo.geekheim.de/index.php/2005-08-24/paper-prototyping-for-fun-and-profit/

<comment><USER2> I think this works extremely well when your users are familiar with the tasks associated with the type of thing your making. But if you're creating something entirely new, they'll need more guidance from the designer to get users started.

One thing to watch out for though is that you can't always rely on what people tell you they like or want to use. There is a huge difference between what people say and what they do. Again, this difference is further magnified if the thing you're creating is something new. It is very hard for people to imagine their life with it.

Other strategies for involving your user in the design process which work very well are: contextual inquiries (to understand what things your audience needs) <http://www.usabilitynet.org/tools/contextualinquiry.htm>, think aloud studies (to understand how well your product is meeting those needs) <http://en.wikipedia.org/wiki/Think_aloud_protocol>, and card sorting (to understand how the user expects the product to be organized) [http://www.boxesandarrows.com/view/card_sorting_a_definitive...](http://www.boxesandarrows.com/view/card_sorting_a_definitive_guide).

</comment><comment><USER3> I guess you could hold a contest where users have to both submit their design ideas and vote on what they like, and giving a prize to those whose ideas you use; acceptance of which indicates they agree that you're allowed to use their ideas.</comment>


The most popular, and fastest declining, programming languages - <AUTHOR> https://www.techrepublic.com/article/the-most-popular-and-fastest-declining-programming-languages/ <comment><USER2> This doesn't look very reliable to me. Visual basic is #6, up one percent to 5.23% from a year ago (and I checked, a year ago it was about 4.x% on their chart before so a "20% gain" in a year). I know there are lots of vb apps out there but growing that much? This article is just a teaser over the real data at [https://www.tiobe.com/tiobe-index/](https://www.tiobe.com/tiobe-index/).

<comment><USER3> Visual Basic is all the rage in medium to large corporations. The ability to write custom scripts to fill Excel spreadsheets is underrated.</comment></comment>


OpenSUSE 13.2 released - <AUTHOR> https://news.opensuse.org/2014/11/04/opensuse-13-2-green-light-to-freedom/

<comment><USER2> As always, you couldn't want a better KDE integration. Good distro for GUI fanatics

</comment><comment><USER3> Wohoooo</comment>


Ask HN: As Covid-19, why is the world and human so fragile In a high-tech world? - <AUTHOR> Maybe for most people the high-tech is not that important for their life,
Maybe it is time to slow down and think what are the most important for a basic happy life, how to more reasonable and appropriate use all these high-techs,
rather than continue to crazily speed economy and create more 'high-tech'. <comment><USER2> Maybe :P

But in all seriousness, maybe your view of the world and the people in it, is a bit skewed towards your current life?

Individual people can be fragile but as a species we are pretty resilient, even as far as the Wuhan Fever goes, some people get it and show no symptoms at all. So don't despair too much and buy into all the doom and gloom on the news, the human race and our world will exist no matter what problems fate throws our way.

As far as it being time to slow down and think what is most important for you to have a happy life, that is probably a good idea. If you can find happiness in the simple things thats great, just don't expect everyone else give up their widgets and tech too!

No matter how fast(or slow) technology or the economy changes there is still lots of happiness(and sadness) to be found if you look for it.</comment>


Linux x86 Program Start Up - <AUTHOR> http://dbp-consulting.com/tutorials/debugging/linuxProgramStartup.html <comment><USER2> Coming from DOS/Windows the amount of... extra cruft that gets linked into a Linux binary by default seems rather excessive. The equivalent startup code in Windows for a C program doesn't involve constructors or destructors, for one thing. From the entry point, there's a total of ~7 functions called in the default stub - most of them for parsing the environment variables and creating argc/argv, and for programs that don't need that it's quite easy to eliminate those too.

(The whole ELF/dynamic linking system is another thing that has always seemed overly complex to me, compared to PE/DLLs.)

<comment><USER3> But Windows still supports constructors and destructors, right? I don't understand how it eliminate the complexity.

Do you mean that the windows equivalent of _start is nested less deeply and with more reasonable naming conventions?

</comment></comment><comment><USER4> On the other side of the spectrum, a tiny ELF executable is possible at 45 bytes: [http://www.muppetlabs.com/~breadbox/software/tiny/teensy.htm...](http://www.muppetlabs.com/~breadbox/software/tiny/teensy.html)

Previous discussion: [https://news.ycombinator.com/item?id=5016434](https://news.ycombinator.com/item?id=5016434)

</comment><comment><USER5> A different take on the same matter: [http://www.bottomupcs.com/starting_a_process.html](http://www.bottomupcs.com/starting_a_process.html)

</comment><comment><USER6> I posted this the last time a similar article came up and a couple people seemed interested. I'm still trying to gauge whether anyone would be interested in a podcast about this kind of stuff. I've started putting together notes for this podcast here: [https://docs.google.com/document/d/1Nn01yDS5PkiegHq4Oxz43xUQ...](https://docs.google.com/document/d/1Nn01yDS5PkiegHq4Oxz43xUQRLHqbc5l5GCRjTEzRw8/edit?usp=sharing)

</comment><comment><USER7> Anyone got similar article for windows?

</comment><comment><USER8> This appears to only be about userland stuff. Would be nice to explain what "execve" does.

<comment><USER9> It's a hack left over from the early days when AT&T UNIX became BSD. Under pre-BSD UNIX systems, a program was started by forking a new process, which meant sharing the code and copying the data. Then the new process did an exec, which loaded a new code segment and wiped the data segment that had just been copied. Copying the data segment and immediately overwriting it was inefficient, so several new flavors of "exec" were developed to get around that.

There probably should have just been a "run" call, which started a subprocess given the indicated executable. Some UNIX-like operating systems have that. It has the nice property that it scales to clusters; there's no reason a subprocess can't run on another machine. QNX, for example, supports that.

<comment><USER10> > Copying the data segment and immediately overwriting it was inefficient, so > several new flavors of "exec" were developed to get around that.

The several different flavors of exec are just convenience functions that lead to an execve syscall (execl* makes the vector out of some of its args; exec*p searches the path first, etc). Maybe you're thinking of vfork (a restricted fork), which seems to have gone away about when copy-on-write came in.

<comment><USER9> Right.

</comment></comment></comment></comment><comment><USER11> This looks like a fantastic resource, but I was off put by the (albeit tongue- in-cheek) 'RMUVI - Real Men Use VI' line in the article.

<comment><USER12> Because of the "men", because of the "vi", because of the "real", or because it's a jab at the readership?

Some time ago, I would have just assumed you were an emacs user, but these days there's a much larger diversity of people on the internet, who get offended by very many things.

<comment><USER13> It might as well say "Real _hackers_ use vi", "Real _programmers_ use vi", or whatever. I do not see why it needs to be gender related.

The author should write whatever he pleases. But I believe it would be proper to use a language directed to a broader audience when writing a public blog post. I also believe that the author was trying to be funny and had no bad intents what so ever, but either way these tiny things matter if we ever want to see diversity in our field.

I would also like to add that your argumentation is quite unpleasant to me. I read it as you believe that diversity is this annoying thing that destroys the good ol' "all dude internet". Would you care to clarify what you meant?

<comment><USER14> > I would also like to add that your argumentation is quite unpleasant to me. > I read it as you believe that diversity is this annoying thing that destroys > the good ol' "all dude internet".

So you're saying that men (read: dudes) can't be offended by this, also? That insults me.

Seriously though: "real men" phrases are often used in a serious way to try to police men's behaviour (you probably know this already, but anyway). So I, as a man, often feel queasy about "real men" quips. But real men aren't supposed to feel uneasy about such things, so I don't know.. ;)

I don't personally feel uneasy about this: it feels like satire that plays on the old trope of "real men do <thing that is hard, often unnecessarily so>", and all over "trivial" things like text editors. I can see how some people might be offended by the phrase, though. I'm just talking for myself here.

(I hope no vi users were offended by this post.)

<comment><USER15> Real men don't get insulted about real men posts.

<comment><USER14> Exactly ;)</comment></comment></comment></comment></comment></comment>


The End of Bash Scripts?  - <AUTHOR> http://hpc-opinion.blogspot.com/2011/08/end-of-bash-scripts.html

<comment><USER2> I think the guy doesn't understand copyright: "There is a general understanding that unless explicitly marked in the contents of the script (the text file that is the Bash program), a Bash script is freely available for use and modification by anyone."

Really?

The fact that bash scripts are not compiled and distributed in a source code form doesn't mean they are not covered by copyrights. Just like ruby, or perl.

I don't know the details, but it is entirely possible that someone copied some scripts around and they are now used anywhere w/o proper redistribution permission.

TRade secrets claims are strange though.

</comment><comment><USER3> I'm not a lawyer, but it seems absurd for them to claim Bash scripts in products shipped to customers can contain trade secrets.

Also, the original article goes into much more depth: <http://www.clustermonkey.net//content/view/308/1/></comment>


Ask HN: How to learn OpenCL - <AUTHOR> OpenCL seems to only have minimal examples, and although there is a fair to good amount of documentation, I feel the need to ask if there are any good books/tutorials on how to use/learn OpenCL? <comment><USER2> Intel and AMD both have some good documentation on their site for getting going with OpenCL, so that's a good place to start. For example:

[http://developer.amd.com/tools-and-sdks/heterogeneous- comput...](http://developer.amd.com/tools-and-sdks/heterogeneous- computing/amd-accelerated-parallel-processing-app-sdk/introductory-tutorial- to-opencl/)

The Khronos website has a huge page with a list of OpenCL tutorials and books:

[https://www.khronos.org/opencl/resources](https://www.khronos.org/opencl/resources)

Amazon has a number of OpenCL books available:

* [http://www.amazon.com/OpenCL-Action-Accelerate-Graphics-Comp...](http://www.amazon.com/OpenCL-Action-Accelerate-Graphics-Computations/dp/1617290173)

* [http://www.amazon.com/Heterogeneous-Computing-OpenCL-Second-...](http://www.amazon.com/Heterogeneous-Computing-OpenCL-Second-Edition/dp/0124058949)

* [http://www.amazon.com/OpenCL-Programming-Guide-Aaftab-Munshi...](http://www.amazon.com/OpenCL-Programming-Guide-Aaftab-Munshi/dp/0321749642/)

This book is available on Amazon but the previous edition is available for free:

[http://www.fixstars.com/en/opencl/book/](http://www.fixstars.com/en/opencl/book/)

Intel's website also has some "Getting Started" articles and optimization guides for OpenCL (for CPU, GPU, and Xeon Phi):

[http://software.intel.com/en- us/vcsource/tools/opencl](http://software.intel.com/en- us/vcsource/tools/opencl)

<comment><USER3> This is just a small sample of the OpenCL kernels available in the AMD SDK (it runs everywhere, at least it did last time I checked).

      AMD-APP-SDK-v2.4-lnx64/samples/opencl/bin/x86/AESEncryptDecrypt_Kernels.cl
      AMD-APP-SDK-v2.4-lnx64/samples/opencl/bin/x86/BinarySearch_Kernels.cl
      AMD-APP-SDK-v2.4-lnx64/samples/opencl/bin/x86/BinomialOption_Kernels.cl
      AMD-APP-SDK-v2.4-lnx64/samples/opencl/bin/x86/BitonicSort_Kernels.cl
      AMD-APP-SDK-v2.4-lnx64/samples/opencl/bin/x86/BlackScholesDP_Kernels.cl
      AMD-APP-SDK-v2.4-lnx64/samples/opencl/bin/x86/BlackScholes_Kernels.cl
      AMD-APP-SDK-v2.4-lnx64/samples/opencl/bin/x86/BoxFilterGL_Kernels.cl
      AMD-APP-SDK-v2.4-lnx64/samples/opencl/bin/x86/BoxFilter_Kernels.cl
      AMD-APP-SDK-v2.4-lnx64/samples/opencl/bin/x86/BufferBandwidth_Kernels.cl
      AMD-APP-SDK-v2.4-lnx64/samples/opencl/bin/x86/ConstantBandwidth_Kernels.cl
      AMD-APP-SDK-v2.4-lnx64/samples/opencl/bin/x86/DCT_Kernels.cl
      AMD-APP-SDK-v2.4-lnx64/samples/opencl/bin/x86/DeviceFission_Kernels.cl
      AMD-APP-SDK-v2.4-lnx64/samples/opencl/bin/x86/DwtHaar1D_Kernels.cl
      AMD-APP-SDK-v2.4-lnx64/samples/opencl/bin/x86/EigenValue_Kernels.cl
      AMD-APP-SDK-v2.4-lnx64/samples/opencl/bin/x86/FFT_Kernels.cl
      AMD-APP-SDK-v2.4-lnx64/samples/opencl/bin/x86/FastWalshTransform_Kernels.cl
      AMD-APP-SDK-v2.4-lnx64/samples/opencl/bin/x86/FloydWarshall_Kernels.cl
      AMD-APP-SDK-v2.4-lnx64/samples/opencl/bin/x86/FluidSimulation2D_Kernels.cl
      AMD-APP-SDK-v2.4-lnx64/samples/opencl/bin/x86/GaussianNoise_Kernels.cl
      AMD-APP-SDK-v2.4-lnx64/samples/opencl/bin/x86/GlobalMemoryBandwidth_Kernels.cl
      AMD-APP-SDK-v2.4-lnx64/samples/opencl/bin/x86/HelloCL_Kernels.cl

I'd hook into OpenCL from the high level language of your choice. Look at

* [http://mathema.tician.de/software/pyopencl/](http://mathema.tician.de/software/pyopencl/)

* [http://www.drdobbs.com/open-source/easy-opencl-with-python/2...](http://www.drdobbs.com/open-source/easy-opencl-with-python/240162614)

Or JRuby or Jython with [https://code.google.com/p/aparapi/](https://code.google.com/p/aparapi/) (still have to write your inner kernel in .java and send that to aparapi)

If you go the "whole-stack-in-c" route much of your time will be spent doing memory operations.

</comment></comment><comment><USER4> If you're a complete beginner in data parallel programming, and you're having trouble finding good intro material for OpenCL, it might almost be worthwhile to check out CUDA instead. In terms of the programming model, OpenCL and CUDA are identical - significant differences don't come about until you start optimizing for specific devices.

I learned CUDA first on my own, and then took an OpenCL class and found that the whole first section was completely redundant. There's also a pretty great wealth of CUDA material online and a few published books if that's your sort of thing.

<comment><USER5> To add some reasons why you'd want to learn CUDA first: It turns out that simple things are a lot simpler, and take a lot less code, in Cuda than OpenCL. With Cuda, your kernel and host code will be close together in the same file. You'll need a /lot/ less boilerplate than are needed in OpenCL to accomplish even the most simple things. OpenCL exposes you to a lot more concepts, and a lot more extrinsic complexity, than Cuda. All this means just playing around a lot easier to do in Cuda.

Just take a look at some simple examples in both, and you'll quickly see what I mean. Even though I'm a fan of OpenCL because it is available on more platforms, Cuda is a lot better suited as a learning platform.

</comment></comment><comment><USER6> A word of warning: OpenCL, and heterogeneous computing in general, is very very difficult. It will take a lot of effort to get even the simplest hello world application working.

And when writing OpenCL, even you are using a single API to write code, if you want to get high performance you will need to rewrite parts of your application for each hardware you intend to run on if you want good performance. This is obvious since the code may end up running on Intel x86's or Intel, AMD or Nvidia GPU architectures which are all very different. If you're lucky, it's enough to rewrite your kernel code (the code running on the device). But you might also need changes to the host side code (running on the cpu) and make changes to the way you manage the memory and DMA transfers, etc.

Finally, when it comes to the basics, OpenCL is not too different from CUDA and there's a lot more material available on CUDA (because it's been around longer and is perhaps used a bit more). You should be able to pick up a book or a tutorial on CUDA and translate it to OpenCL without too much effort.

Finally, even though it may take quite a lot of learning to get started, parallel programming on GPUs is quite fun and it is very rewarding to see your code run with very high performance.

</comment><comment><USER7> Disclaimer: this is a rant, it is obvious that I don't like OpenCL and that I think that it was designed by monkeys so take it with a grain of salt.

In short: don't learn OpenCL. Both CUDA and C++AMP are good languages for programming heterogeneous machines and nVidia's Thrust and Microsoft's PPL are both excellent libraries to write efficient and reusable code. These language extensions are also strongly typed and come with really good tools. My advice is: learn any of them instead.

Why not OpenCL? AMD's Bolt library is the live proof that OpenCL is fxxxxx up beyond all repair. It is not meant for humans to write, nor for machines to understand.

Kernels are just character strings!!! This is just so wrong! Forget about using functors and lambdas as kernels, and forget about mixing kernels with templates. You will be better off using Python and PyOpenCL (which is great) that using C and C++. In C++ generating kernels is really hard, and generating kernels from expression templates is insanely hard.

Furthermore, this also means that the language is not typed at all!! Forgetting a semicolon in C results in a runtime error! Do you want syntax highlighting? Write your kernels in separate files! This is even worse than the way people used to write functors far away from the call site in C++03, at least they were in the same file!

As stated above my advice is don't learn it. Let it die. Your time is better spent learning CUDA/C++AMP and their libraries. The design rules for OpenCL have been "let's not learn anything from OpenGL" \+ "we need something, this is something, let's standarize this". This of course has resulted in an hilarious language that came after CUDA and was worse in every possible way.

</comment><comment><USER8> If you're using C++, check out Boost.Compute [1]. It provides a high-level STL-like API for OpenCL (without preventing you from directly using the low- level OpenCL APIs). It simplifies common tasks such as copying data to/from the device and also provides a number of built-in algorithms (e.g. sorting/reducing/transforming, etc).

[1] [https://github.com/kylelutz/compute](https://github.com/kylelutz/compute)

</comment><comment><USER9> I've started with this[1] blog post, then worked quite a bit on adding proper error checking to the example code to figure out why it failed :), now the author has merged my changes. So perhaps it's a worthwhile example to start from now. I haven't done much with OpenCL yet, though, in the end I figured out that my ~7-8 year old laptop ran SIMD optimized C code faster on the host CPU than on the GPU (I wrote this[2] with heavy SIMD optimization work, not sure anymore whether this was what I tested against OpenCL though), which is a reason why.

[1] [http://www.thebigblob.com/getting-started-with-opencl-and- gp...](http://www.thebigblob.com/getting-started-with-opencl-and-gpu- computing/) [2] [https://github.com/pflanze/mandelbrot/tree/master/c](https://github.com/pflanze/mandelbrot/tree/master/c)

</comment><comment><USER10> Perhaps Apple's documentation and WWDC video might be of help: [https://developer.apple.com/opencl/](https://developer.apple.com/opencl/)

To watch the video you need to be a registered Apple developer.

<comment><USER2> To clarify -- you can read the documentation and code samples without having to register as an Apple Developer. You only need to do that to watch the tutorial videos or access the developer forums.

</comment></comment><comment><USER11> One great way to start is to use OpenCL libraries. We work on clMath ([https://github.com/clMathLibraries](https://github.com/clMathLibraries)) and ArrayFire ([http://arrayfire.com](http://arrayfire.com)) which are both easy to pick up. Once you get comfortable with libraries, you can start trying to write your own kernels, and you'll know which things you'll need to write that aren't already in a freely available library. Good luck!

</comment><comment><USER12> Mess around with open-source applications (such as Rodinia benchmark suite and NAS parallel benchmarks) after going through the basic tutorials on AMD, Intel and Nvidia webpages.

<comment><USER13> A good open-source OpenCL application might be cgminer or bfgminer, might have to get an older version though, it looks like the latest ones have abandoned OpenCL/GPU support.

<comment><USER14> Any idea why they abandoned it? CPU-only mining seems to be kinda pointless these days...

<comment><USER15> Cgminer and bfgminer are focused on usb based ASIC miners now. Bfgminer still supports cpu and gpu mining, it's just unsupported and not compiled by default.

</comment></comment></comment></comment><comment><USER16> You might wish to check out the University of Illinois's "Heterogeneous Parallel Programming" course, offered through Coursera:

[https://www.coursera.org/course/hetero](https://www.coursera.org/course/hetero)

The course is already currently ongoing, but it's not too late to enroll. The course is mainly focused on CUDA (since it's easier to learn, the professor believes), but covers OpenCL as well.</comment>


More Facebook users are on Android than iOS, but just barely - <AUTHOR> http://thenextweb.com/facebook/2012/10/08/more-android-users-use-facebook-than-ios-users-but-just-barely/

<comment><USER2> Where are the other 61% of mobile users coming from?? I know facebook has been focusing on making their mobile site available on all platforms, but I didn't think they would realistically have more than half the visits coming from non- smartphone devices. I doubt Blackberry and WP is more than 1 or 2%.

Also, does android come with facebook already installed? I don't think iOS does, which could account for the difference between mobile web and app.

<comment><USER3> It's not a comparison of mobile usage but of all usage, so the other 61% are just pc's</comment></comment>


Free Interactive Python Tutorial - <AUTHOR> http://www.learnpython.org/

<comment><USER2> Looking good!</comment>


DARPA-backed IoT-sensing security startup Qadium raises 20M Series A - <AUTHOR> http://www.forbes.com/sites/thomasbrewster/2016/06/05/qadium-iot-google-security-darpa-cia/ <comment><USER2> How large do they expect this search space to be compared to 'websites'?</comment>


How to Monetize MySpace - <AUTHOR> http://robertyoung.vox.com/library/post/how-to-monetize-myspace.html <comment><USER2> There are certainly a lot of ways to monetize MySpace beyond ad revenue, and using it as a music distribution channel is one of the easiest. (maybe News Corp. should buy Amie Street, and integrate it into myspace?)

But shouldn't they start by doing a better job capitalizing on the advertising potential? Theoretically, MySpace should know a massive amount of information about each user (everything they list on their profile page), including location, age, interests, media consumption patterns, etc. They could be selling very highly-targeted ads based on all of those parameters. Wouldn't it be more lucrative NOT to sell a graphical ad on the homepage at a daily rate of $250,000 (as the post reports), but instead to make the homepage ad dynamically tailored to each user?</comment>


LangId Tells You The Language of Your Text - <AUTHOR> http://langid.net/

<comment><USER2> To help with all the many postings expressing surprise at a misidentification: How long was your sample? According to the link posted by fixz, [http://alias-i.com/lingpipe/demos/tutorial/langid/read- me.ht...](http://alias-i.com/lingpipe/demos/tutorial/langid/read-me.html) , this is using letter-frequency statistics. It's very easy to fool such a system by feeding it a very short sample. For a similar problem that can arise with trying to auto-detect encodings, see: [http://blogs.msdn.com/oldnewthing/archive/2004/03/24/95235.a...](http://blogs.msdn.com/oldnewthing/archive/2004/03/24/95235.aspx)

Try feeding it a bit more than a single word or sentence.

<comment><USER3> Yeah you got that right. The more you type the more google apis should be accurate.

</comment></comment><comment><USER4> I love this tool. Been bookmarked.

There's something seriously wrong with the web (or my usage of it). Whenever I need something like this, I cant find it. Go on google, search for any kind of tool. No matter how you word it, the results never get passed 1999.

Yesterday I needed something to convert a couple timezones. Try finding something via google. Every relevant keyword match was a pile of shit tool belongs in 1995.

One idea is that the people who are good at making tools are the exact opposite of the people who are good at SEO.

Or I could just be doing something wrong. Not using google right, or maybe I should be using a different search engine. Help would be appreciated!

</comment><comment><USER5> Semi-open source code and tutorial that you can use to understand what they're doing.

[http://alias-i.com/lingpipe/demos/tutorial/langid/read- me.ht...](http://alias-i.com/lingpipe/demos/tutorial/langid/read-me.html)

</comment><comment><USER6> Nice implementation of the Google Ajax Language API: [http://code.google.com/apis/ajaxlanguage/documentation/#Dete...](http://code.google.com/apis/ajaxlanguage/documentation/#Detect)

<comment><USER7> It's funny--I was going to post that exact phrase and link.

</comment></comment><comment><USER8> I tried this obscure eastern European language. It identifies polite phrases correctly, but curse words get identified as Portuguese for some reason.

They need to bone up on their international cursing.

<comment><USER9> I tried just the word "reddit" and it identified it as Portuguese as well. Maybe that's a default for unknown words.

</comment></comment><comment><USER10> NLP is a hard nut to crack! Even with the Google API :)

From impressive ... identifying Greek written in English, correctly to returning French and Czech (flag missing!) for gibberish!

Nice Toy!

</comment><comment><USER11> It also picked up Persian while transliterated. That's impressive to me, because Persian does not have a standard method of transliteration.

<comment><USER12> Transliterated Russian seems to get ID'd correctly, but only sometimes. (Anything with "zdrastvuite" in it, for example, works fine. But other simple Russian 101 sentences --- e.g. "U menya est odin karandash" meaning "I have a pencil" --- get ID'd as French. Unless of course I'm spelling it wrong.).

But I haven't seen it correctly ID transliterated Chinese or Korean yet.

</comment></comment><comment><USER13> Identified several common Japanese phrases as Vietnamese. True, the difference is subtle, but...

</comment><comment><USER14> Would be even nicer if it provided a link to translate or just showed a translation as well.

</comment><comment><USER15> I went with "ni hao". It told me it was Vietnamese. Possibly, but not only.

</comment><comment><USER16> Hm. It identified Bislama - pidgin English from Vanuatu - as Indonesian.

<comment><USER17> Yes, "bislama" means "with peace" in various languages in the muslim parts of the world (to me it sounds very north african.)

</comment></comment><comment><USER18> Did anyone else expect this to be for programming languages?

</comment><comment><USER19> Close but not really close: identifies faroese as icelandic.

</comment><comment><USER20> An unfair test, but it "identified" lojban as Italian.

</comment><comment><USER21> wtf? Identifies Arabic correctly but displays the Saudi flag. I like the service, though, despite of this minor annoyance.

<comment><USER17> I saw that too :-/

They're confusing the saudi flag with the arab league flag, both green with white arabic text.

[http://upload.wikimedia.org/wikipedia/commons/thumb/2/2b/Fla...](http://upload.wikimedia.org/wikipedia/commons/thumb/2/2b/Flag_of_the_Arab_League.svg/800px- Flag_of_the_Arab_League.svg.png)

</comment></comment><comment><USER3> Added 'Identify by Link' feature!</comment>


15 Key Takeaways Every Start Up Founder Needs to Know to Avoid Failure - <AUTHOR>

Hey Everyone,
I will be posting a 5000-word biography / essay on my life and what I have learned over the past 6 months developing my startup. I thought I would give you a sneak peak of the Medium post before it's live. Here are the 15 Key Take Away's that I go over throughout the post. The post was written to focus new startup founders on the only things they are suppose to concentrate on in order to ensure they don't fail. If you follow these take away's your chances of success will increase dramatically. If there are any other points you think I left out please leave a comment and I might add them to the post.
I will make sure to add the full post once it is live on Monday March 9, 2015
15 Key Take Away's from my Medium post "If You're Crazy Start a Startup." • Find amazing co-founders that are just as passionate as you. • Prepare to give up your social life and if you are in a relationship make sure they are extremely supportive of your idea. • Create daily, weekly, and monthly goals and don't let the due date slip. • Grow as fast as you humanly can. • Focus on building an amazing product first before anything else. • Money doesn't grow on tree's you gotta work hard for it. (I know this is cheesy, but it's so true) • Build things that don't scale. • Create an amazing user experience and differentiate yourself from others. • Talk to users, talk to users, talk to users.
• Never stop learning.
• Don't start a startup until you have a problem you are facing and are passionate enough to dedicate your life to solving that problem. • Be unique / memorable and seperate your brand from your competitors. • Write down your core values right away and only hire people that breathe your mission. • Forget about competitors. If your product is better, than your competitors should be worried about you.
• Eat healthy and exercise regularly. <comment><USER2> If I may, I'd like to add one to your great 15. Don't build until you've interviewed a ton of people about whether they'd use the product you plan to create. If most do, build. If most don't...don't build.

<comment><AUTHOR> Thanks for the tip. This kinda goes along with talking to users. But is a bit more in depth. I will definitely make sure to add it.</comment></comment>


The Secret Science Behind Big Data And Word Of Mouth - <AUTHOR> http://techcrunch.com/2013/06/15/the-secret-science-behind-big-data-and-word-of-mouth/ <comment><USER2> The linked study goes into much greater detail: [http://jonahberger.com/wp- content/uploads/2013/02/ViralityB....](http://jonahberger.com/wp- content/uploads/2013/02/ViralityB.pdf)

<comment><AUTHOR> yes</comment></comment>


Free Computer Science E-Books - <AUTHOR> http://www.e-booksdirectory.com/listing.php?category=24 <comment><USER2> Thanks, this will be useful. As I said on my blog in "The Web Is Still Not Adequate for Serious Study", one of the biggest problems with using the web for study is finding things, like this, that may be out there somewhere.

<comment><USER3> I've found that Wikipedias references are usually really good and then just using wikipedia and wikibooks to become familiar with the subject enough too investigate further is quite effective.

Although if I want to really study something beyond introductory reading I use scholar.google.com and looking at the articles published in the last decade or so as well as looking at the papers citing them too be quite effective. Although that is most effective when you are researching a narrow subject and it does sometimes require you too pay too have access too the journals, something which my university provides.

Lastly there are quite a lot of recorded univerity lectures published online that cover just about any subject you'd want to study from the comfort of your computer. I'm verry happy google has made an effort to gather them at: <http://www.youtube.com/edu> recently altough there's other places. For computer-science in particular I've found this blog to be highly good at gathering the interesting ones: <http://freescienceonline.blogspot.com/>

I realize you might already know all of this but as a student I personally am quite pleased with what the Internet provides for me especially as I like (and it enables me) to go far above and beyond what is assigned as course work, when I study something.

</comment></comment><comment><USER4> Very cool! I'm not complaining, but I do wish they provided more formats than just PDF (PRC, LIT, etc.)

</comment><comment><USER5> Any recommendations from this list?</comment>