Well-Kept Gardens Die by Pacifism - <AUTHOR> http://lesswrong.com/lw/c1/wellkept_gardens_die_by_pacifism/ <comment><USER2> To be honest, while I do see his point to some degree (just about every webmaster forum has become an utter cesspool due to a lack of moderation and a flood of idiotic posts and signature spammers), I'd also like to point out that being too harsh can kill a site just as quickly.

For one thing, it makes newer users feel a bit too scared to start contributing, especially if everything slightly below quality standards get removed immediatedly. And given that a community depends on new people as its lifeblood, the result can easily be an ever dwindling pool of site veterans talking among themselves til everyone loses interest and stops visiting.

So you want to remove the obvious idiots and poor contributors, while not scaring away people who might otherwise become decent users in the process.

There's also a related issue you need to consider here, and that's that to some degree, you need to avoid being too strict in terms of what's 'relevant' to the community. If you've got a programming forum, it may seem tempting to shut down any 'low quality' conversation about unrelated matters like TV or films or sports or whatever else, since they're not related to the site. But people don't purely talk about one thing over and over. It's best to have a place to discuss these 'unrelated' things (with some semblance of quality control) so as to bring the community closer together and to see other members as friends and associates rather than just virtual colleagues.

In other words, keep quality up, but don't brutally remove anything that's even the slightest bit dodgy or shut down everything not 100% related to the topic, otherwise you'll likely end up with a perfectly immaculate ghost town.

<comment><USER3> I saw a really nice post from either sama or pg or dang about /noobcomments. Someone had suggested that was a page where new users were going to get rapid downvotes and flags. This person said that they regularly read noobcomments and upvote the good posts and provide advice to the others.

(I have absolutely no idea what I need to plug into the search box to find this post. Sorry.)

</comment></comment><comment><USER4> This has been posted 4 or 5 times before, with a frequency of 2 or 3 years. I would like to add that it is not only about online communities, but it applies to real communities as well. All of the developed communities in the world are so pacific (not willing to defend) that it is quite easy for wilder communities to invade. It is happening more now than before.

<comment><USER4> In other words, in medieval ages, if barbarians had a simple goal of kill all Romans and Romans had a complex rule that says kill only those persons who are found to be harming good Romans, then obviously barbarians would win by the simplicity of their work.

</comment></comment><comment><USER5> _Because I really do honestly think that if you want to downvote a comment that seems low-quality... and yet you hesitate, wondering if maybe you 're downvoting just because you disagree with the conclusion or dislike the author... then nine times of ten, I bet, nine times out of ten at least, it is a comment that really is low-quality._

Five years on Reddit (and a few more on HN) has proven beyond any possible doubt for me that downvotes are used "9 times out of 10" more for disagreement rather than an honest value judgement on quality.

It's simply far too easy to mentally conflate "this comment is _wrong_ (because of my own biases)" with "this comment is _low quality_ ", especially in the midst of a heated discussion.

Speaking of biases, the author here clearly has his own. He lives in a world where groupthink/hiveminding/circlejerking (pick your favorite term, they are all synonymous) doesn't exist, where astroturfing never happens, where moderators never abuse their power, where karma is always used for its stated purpose, and where censorship isn't a documented and heinous thing.

..which is pretty hilarious when you remember this is an article about moderation apologia.

<comment><USER6> I honestly believe that "quality" always correlates with personal biases and collective norms.

Even the guideline for posting here - "anything that good hackers would find interesting" \- assumes that 1) you should be a "hacker" (a subjective term, especially on a site with as diverse a community as this) and qualifies this with "good" (another subjective term.) Are we to assume that stories which are popular (high karma) are necessarily posted by better hackers than those which aren't? One should post "anything that gratifies one's intellectual curiosity," implicitly provided one has the _correct_ sort of intellect. While this is designed to bias the site in favor of high karma posters (and, obviously, whatever the mods happen to want) the end result is that quality is in the eye of the beholder.

It's all arbitrary and it's all subjective.

<comment><USER5> I don't think the problem is the subjectiveness as much as the knee-jerk reaction.

If you're in a heated argument with someone, most any human's first reaction given the ability to "punish" their opposer will be to click the downvote button.

The mental heuristic, in our case "anything that good hackers would find interesting", just never gets run. It's effectively dead code, because the stack of "do I agree with this" if/then blocks happens a few hundred lines up.

It takes a lot of effort to override that reflex.</comment></comment></comment>


Vim and Ctags tips and tricks - <AUTHOR> http://stackoverflow.com/questions/563616/vim-and-ctags-tips-and-tricks <comment><USER2> Give GNU Global a try [http://www.gnu.org/software/global/](http://www.gnu.org/software/global/)

[https://github.com/OpenGrok/OpenGrok/wiki/Comparison-with- Si...](https://github.com/OpenGrok/OpenGrok/wiki/Comparison-with-Similar- Tools)

<comment><USER3> Thanks for posting this. Am surprised to have not heard of this project before. It's a bit disappointing that it's not included in either of the Distro's that I use regularly. And even worse that the source for the project is maintained in CVS. It seems like perhaps the project is a tad neglected.

<comment><USER4> it's in fedora and debian (package name global). what distro do you use? it seems easy to package...

</comment></comment></comment><comment><USER5> In addition to Ctrl+T to go back from a definition, Ctrl+O also seems to work, which I find more ergonomic on Qwerty.

</comment><comment><USER6> I find it useful to display the output of tag commands in a location list rather than the default vim pager. (Use "ltag" instead of "tag" from the vim command line).

</comment><comment><USER7> Before clicking this link, I wager that it's already been closed as "not constructive" by zealot S.O. mods.

 _Edit:_ yep. Thanks Kev, good work!

<comment><USER8> To be fair SE have made it clear that they don't want that type of question there, so at least they're consistent.

I'm surprised that HN hasn't spotted the market opportunity for a site like SE but allowing only non-specific discussion style questions.

<comment><USER9> The better market opportunity would be SE that figured out a way to serve their customer base better. Splintering isn't always the best answer. For instance, the very valid "meta is murder" ideas lead SE to ban meta, and the need for meta led them to create a separate place to discuss meta issues for SE. But...it wasn't the SE community having the meta discussions then. It was a separate community with some overlap. IMO it was a bad move. To be fair, doing it right is an unsolved problem.</comment></comment></comment>


Thomas Harriot - <AUTHOR> http://en.wikipedia.org/wiki/Thomas_Harriot

<comment><USER2> You might be interested in Stephen Greenblatt's essay "Invisible Bullets", especially the first part of it, where he discusses Harriot quite a bit.

Here is a brief summary of what he has to say about Harriot:

[https://en.wikipedia.org/wiki/Subversion_and_containment#The...](https://en.wikipedia.org/wiki/Subversion_and_containment#The_English_in_the_New_World)

<comment><AUTHOR> Cool. Will check out. I'm surprised he doesn't get more of the unsung hero fanfare usually reserved for people like Tesla.</comment></comment>


Ask HN: What should I do before releasing a project as OSS? - <AUTHOR>

I've seen a few interesting tools transition from being private internal tools at some company, to useful open source software and libraries (such as Flanker)
I'd love to learn some more on best practices on how to clean things up, and get them ready for the world.
For example:
Should you open up your internal tickets, with all the history and cruft, or create a new clean issue tracker. (Maybe Github Issues?)
What about the git history? Intuitively it seems useful, but is anyone really going to find value in branches to work around issues that since went away?
If the install is complicated, is it worthwhile to create specific to install on Ubuntu/Fedora/OSX, or is it better to try to script it to run automatically?
Should one provide a Vagrant box? Vmware image?
I'd love any advice, guides, or thoughts on best practices for releasing an formerly closed project to a larger world. <comment><USER2> As per [https://news.ycombinator.com/item?id=6911908](https://news.ycombinator.com/item?id=6911908), you should check and double check that you are not exposing any secret keys or credentials.</comment>


How does an Angel Investor get invited to a YC demo day? - <AUTHOR> <comment><USER2> I'm pretty sure you can apply for an invite at [http://www.ycombinator.com/demoday](http://www.ycombinator.com/demoday).

</comment><comment><USER3> This type of question is probably better addressed to email@example.com.</comment>


Illusory ownership of an invisible body reduces social anxiety responses - <AUTHOR> http://www.nature.com/srep/2015/150423/srep09831/full/srep09831.html

<comment><USER2> potential future solution to public speaking woes? I assume this also reduces general (including positive) social engagement, which ought to be studied</comment>


Joel Spolsky (Building Stack Overflow and Values Developers Care About) - <AUTHOR> https://channel9.msdn.com/Blogs/Seth-Juarez/Joel-Spolsky-Talks-About-Building-Stack-Overflow-and-Values-Developers-Care-About <comment><USER2> Seemed like a little bit of a puff piece. Would have liked to hear questions like "Is Quora a competitor?" or about the recent claims that Stack Overflow was going downhill.

<comment><AUTHOR> For what it's worth, they addressed the "going downhill" issue in one of their latest podcasts. [http://blog.stackoverflow.com/2016/10/Podcast-89-The- Decline...](http://blog.stackoverflow.com/2016/10/Podcast-89-The-Decline-of- Stack-Overflow-Has-Been-Greatly-Exaggerated/)</comment></comment>


China used prisoners in lucrative internet gaming work (2011) - <AUTHOR> https://www.theguardian.com/world/2011/may/25/china-prisoners-internet-gaming-scam <comment><USER2> This was in 2011. They probably got them finding near hash collisions (by hand, with pencil and paper) for mining cryptocurrency in 2019.</comment>


The Case Against Lawns - <AUTHOR> https://www.curbed.com/2019/3/13/18262285/mcmansion-hell-kate-wagner-lawn-care-mowing <comment><USER2> "After the neighbors replaced the nearby woods with an oversized house and lawn"

The lawn is the least of the problems cited here.

And its quite geographically specific. If you get heavy rain, you want grass to absorb run off. Because lawns aren't going to be replaced by trees and natural wilderness. They'll be replaced by paving.

<comment><USER3> I felt it was pretty clear that while details (like the one about the neighbors) were geographically specific, the point wasn't to get rid of all grass. The term "lawn" in this essay is a short-hand for "turf grass lawn".

For example, the starting suggestion was "Getting rid of turf grass and replacing it with native grasses, prairie, or whatever natural ground cover happens to be inherent to the place you live and that doesn't require fertilization, pesticide use, or mowing".

If you need grass to absorb run off, then go ahead and use grass which has evolved in and for that local climate.

I felt like the author should have pointed out some of the politics involved, like city or neighborhood rules which place limits on the height of the grass. One of the houses near me when I lived in Illinois used prairie grasses. I loved it. But the grasses were a _lot_ higher than the turf grass of the neighbors, which upsets people who 1) value the turf grass lawn look and 2) believe, perhaps correctly, that their house value depends on having neighbors who agree with 1).

</comment></comment><comment><USER4> What I've noticed about lawns:

From looking at pre-1990s houses in the suburbs, I have noticed that lawns typically comprise a large percentage of the lot size. It seems to me, back when those houses were built, a desirable quality of a suburban home was having a 'lawn where the kids could play around'. Lawns also were (and still are) a personal show of accomplishment, so people took interest in their lawns and the lawns of others. (If you watch 80s and 90s movies with suburban shots - like Donnie Darko - it also shows "the American lawn" in the suburbs).

In newer houses built after the 2000s, I've seen house size comprise more of the total lot size, and lawn size reduced. I figure this is a sign of the trend to have a big house over a lawn. It's fairly economical today to build a template-style cul-de-sac or subdivision of "large" homes built closer together (thus smaller lawns) to capitalize on the land. Plus, if someone is spending a lot of money on a house, then I suppose they want to get their money's worth in amenities (2 car garage, 3+ full baths, nice kitchen, big living room, etc), which means more house is needed. There's also less reason to care about having a lawn if it's going to get less use out of it. On speculation, maybe 70% of the lot is house, with the remaining 30% of the lot being a small front yard and larger fenced back yard.

Aside:

As for having natural lawns that are more locally ecological friendly as opposed to manicured grass, I think the main impediment is people's perceptions and fear of "pests". Either neighbors get pissed off or they complain about property values, local ordinances, HOA bylaws, etc to some local government about "non-standard" lawns that promote ecological cohabitation.

It's kind of weird so many people in the suburbs have this view. Like they are actually afraid of having nature next to them, when the suburbs are a place outside the city built right next to nature and rural land.

I used to live close to a small park, and I would leave out unused food at night on ocassion for deers, rabbits, and stray cats that would wander around, instead of throwing it away. My neighbor saw me doing that and threatened to call the cops on me, and report me to the health board and animal control. If animals were wrecking havok on neighbors yards I would understand, but neighbors like these are so protectionist of their yards, they will blow up a petty thing to huge proportions and find a way to get "the law" on their side to make life miserable for someone. As if I am putting them in grave danger because a deer ate some berries in my backyard. I don't understand why people would buy a house next to a park if they really needed to be so fenced off.</comment>


Ocean carbon uptake widely underestimated - <AUTHOR> https://www.sciencedaily.com/releases/2020/09/200904090312.htm <comment><USER2> This seems particularly troubling in terms of oceanic acidification, right? Carbonic acid is formed in the CO2 uptake. As I recall, it's the same method our bodies use to detect that we need more oxygen: blood acidity from CO2.

So if the ocean is taking up 10% of global petrochemical emissions _more_ as CO2 doesn't that mean the rate of acidification will be compounding even faster?

<comment><USER3> Worse than just the direct acidification: if we're already seeing the increase in surface temperatures that we are, with oceans absorbing more carbon than we thought, what happens when ocean sequestration decreases? It will do that nonlinearly, we just don't know exactly when.

(It's expected that when the pH of the Southern Ocean gets past a threshold, phytoplankton (with carbonate shells) will start to be replaced by algae (without). Dead phytoplankton falling to the ocean floor is a major component of carbon sequestration by the oceans.)

It suggests that we could pass a threshold and be in for a world of hurt very soon.</comment></comment>


Rethinking the Commodore 64 Memory Map (2018) - <AUTHOR> http://www.c64os.com/post?p=57 <comment><USER2> Having architect elegant solutions, and watched much of the nuance of the elegance languish in the ecosystem, I can't help but wonder: Did the original creators have _exactly this_ in mind, and it's only just now being capitalized upon?

<comment><USER3> Perhaps, but it is more likely they provided many options, hoping it worked out.

Sometimes it did not. We look back, and some machines were almost there, like the Apple GS. Still great, but with just a bit more thought, could have been amazing.

Many of the better effects on C64, and other computers, were actually exploits. Bugs in the logic were found, then abused, then perfected.

Most of the better titles on the old 2600 (VCS), for example, exceeded engineering specs. A bit of flexibility left in turned out to make all the difference in the world.

Older machines used a combination of software and hardware. That combination turns out to be quite potent over time as people continue to explore what may be possible.

Some have coaxed better color, interlaced video, and more out of a C64, despite that not being intended.

The memory map options were likely flexible, "just in case", and where that was the thinking back then, it often became the max potential case.

<comment><USER4> "Older machines used a combination of software and hardware. That combination turns out to be quite potent over time as people continue to explore what may be possible."

Makes me wonder what undiscovered things are possible with say, a SPARCStation 1+, or an Indigo2 R10000 when bit-banging the hardware directly.

<comment><USER3> I have wondered that about SGI machines too.

One guy did explore the largely unused, shared memory graphics system that was used on the O2 and NT workstations SGI made.

He made a very precise compositor. Was capable of many real time effects and sub pixel accuracy. The thing could take huge images and do real time transforms, effects at a couple hundred MHZ clock.

I ran it once, in the very early 00's. Can't for the life of me seem to remember enough to get a URL.

</comment></comment></comment></comment><comment><USER5> It's great to see someone putting so much thought into this. I've been coding heavily in hires (320x200) mode the last few weeks on this platform.

Having the memory banks swappable has all kinds of fun quirks. For example, although the VIC-II could address many different 8K memory locations for the hi-res screen, only four are okay, because others would be the character ROM ($1000 and $9000). Therefore, people tend to put the music code at $1000.

Also there's only one color memory for all the character mode pages and character memory is the color memory for hires mode!

</comment><comment><USER6> Zooming into the grid representation of 64KB of memory reveals some fun artifacts. Tiny o's appear kind of regularly. I don't know what kind of resampling would account for that... maybe a mistake or a counterintuitive way to shift error among the cells?

<comment><USER7> The minor gridlines are dotted. The "tiny o's" appear where the spaces of two lines are aligned, so four dots appear around the crossing point.

</comment></comment><comment><USER3> What I love about all this is we managed to get networked together in time to enjoy the roots of computing together.

I had no idea the fun 8 bit, and older machines would still see development and innovation.

Makes me happy, and I know there is fun to be had when I get old.

</comment><comment><USER8> It's incredible the amount of research and brainstorming it took to develop this solution. Does anyone know more about the author? Is this a hobby project? Either way, these optimizations [system design] are super impressive.

<comment><USER9> C64os is by Greg Nacu from the Toronto area. It's a hobby project along with a portable c64 he's built that's more portable than an SX-64 and much more versatile due to modern components.</comment></comment>


The Buzzfeed Lesson - <AUTHOR> https://stratechery.com/2019/the-buzzfeed-lesson/ <comment><USER2> I am still waiting for a good micropayments platform to allow me to support arbitrary web sites with relatively low-friction (and high-privacy) small payments to consume their content. I would happily pay $0.25 or thereabouts to read articles on some of the sites I consider to be good content creators.

That said, I have somewhat high expectations. E.g., I would hope that such a platform doesn't take too large a cut (e.g., Apple app-store sized cut) because that's offensive to both readers and creators.

<comment><USER3> Completely disagree. Having to make a purchase decision is stressful and a cognitive nuisance. I'd much rather pay $10 a month for Netflix and not think about it, than $.20 to watch an episode.

<comment><USER4> Netflix would prefer you do this, too. In subscription models, you are almost guaranteed to be paying more than you're using. Only when a service is first starting and trying to attract subscribers is it likely you are paying less than you use.

<comment><USER5> How do you measure the value or cost of how much someone uses for s product like Netflix? Compared to equivalent iTunes purchases, a Netflix subscription is inexpensive. If you are referring to the bandwidth, the cost is close to zero anywhere.

<comment><USER4> No, I'm not talking about bandwidth. iTunes isn't generally selling one-time access, but perpetual ownership (kinda) of a single episode. This accounts for the outlandish $2 per episode sort of thing.

A non-subscription model for a Netflix service would be closer to fractions of a penny per view.

<comment><USER5> Fractions of a penny per viewing can only refer to bandwidth costs; it's 1-2 orders of magnitude too small to cover the costs of producing or licensing a show, let alone Netflix' other fixed operating costs.

Everything Netflix talks about is getting users to spend more time watching shows, not less. They certainly want subscriptions because they make more money, but they do it by aggregating a ton of content into one subscription and providing it at effectively no variable delivery cost. The result is that users pay less per hour watched than can be provided by anyone a la carte.

<comment><USER4> I don't follow your reasoning. Whatever the delivery costs may be, they don't determine the service cost. That's determined by value to the customer (though delivery costs sets the minimal viable service cost).

Their adherence to a subscription modivates the view of getting users on the platform for as long as possible, regardless of what they're watching. A pay per view model would twist motivations another way.

<comment><USER5> Yes, that is my original point I wanted to make in disagreement with you. Netflix does not make more money off a subscription because of underuse, they make money because of, if anything, overuse which is possible because the value to the watcher (some number below the a la carte rental price of $2-5 set by competitors) greatly exceeds the variable cost of providing the view of the show (nearly $0.)

</comment></comment></comment></comment></comment></comment></comment></comment><comment><USER6> There is a reason why we don't have any VC investments in the media, or for that matter, in the content creation on the internet. The reason is because Google and FB monopolized search, advertising, and more importantly, surveillance of individuals. So, the website that caters clinical information to doctors competes with Candy Crush, because doctors get served relevant ads on games, etc.

In the early days of internet, we had plethora of new media networks. No more. The same with science websites, smart websites for children, hobby websites, etc etc. Big tech is killing content on the internet, and instead we have fake news controversies and privacy scandals.

This is very serious stuff for society and democracy and knowledge.

PS [https://www.cnbc.com/2019/01/28/tech-monopolies-are- killing-...](https://www.cnbc.com/2019/01/28/tech-monopolies-are-killing- journalism-ocasio-cortez-says.html)

Edit. Think about it: Google and FB have 80% share of the ad market, and they actually don't create the content. The rest 20% goes to all the publishers from the entire world.

<comment><USER7> Who is "we"? There are tons of VC investments in media.

<comment><USER6> Ok, which ones? Give me examples.

<comment><USER7> The Athletic, Gimlet, Cheddar, Scroll, Piano, Tastemade...

<comment><USER8> Out of these, I have heard of only Gimlet Media and that's because I'm neck deep into podcasts.

</comment></comment></comment></comment></comment><comment><USER9> A number of HN comments have pointed to the internet-native advantage Buzzfeed (BF) had, and that the layoffs are disconcerting or puzzling in this respect. It's a fair and natural point. However, there are a number of fundamental shifts occurring as the digital economy grows as a percentage of the total economy. Many companies misaligned to these on-going shifts will either change or go under.

Shouldn't an internet-native company already be well aligned to these shifts? Not necessarily. For instance, one of the shifts underway is the increasing transience of markets. That is, markets are created, undergo change, and die- off at an accelerating pace. This is just one fundamental shift underway, there are others, and companies will not exist very long if merely a portion of the company is aligned for this new environment.

<comment><USER10> I always wonder how one arrives at such conclusions with this kind of confidence. Are you reading and essentially consolidating a lot of your acquired knowledge? Is this the tenor observant people are agreeing upon at conferences? Are you in some kind of overview position?

I am merely asking because from my perspective I feel I could never conclude such things and I have no idea how it happens?

<comment><USER9> I don't understand your question. Alternatively, if you're making an argument, then I don't what your argument is.

</comment></comment></comment><comment><USER11> Earlier he said the world needs good journalism and therefore journalism needs a good business model.

I see this sentiment often but I see no support for it. There is no link between the premise and the conclusion. Plenty of things in the world have no business model: much fits under the umbrella of philanthropy. Others are provided by government, which provides some things for which there used to be a business model but now isn't (public transportation for instance).

So it is with journalism. Philanthropy for instance could support it. So could government. In the US the idea of government doing this raises hackles, but it's not uncommon worldwide; see the BBC for instance. Even in the US you see some journalism funded with government and philanthropy (Frontline on PBS.)

</comment><comment><USER12> _BuzzFeed sucks, their journalists were providing benefit to the business, so they pivoted._

This isn't about BuzzFeed "sucking".

But rather, about its screwing over its employees while trying to appear "hip" and "anti-establishment".

 _Can we move on?_

Until Jonah digs into his piggybank and coughs up the backpay owed to his laid-off employees - no.

<comment><USER13> What backpay is owed to employees? If you're referred to unpaid PTO, if it was in their employement contract I agree. Otherwise, just another case of clueless millennials signing contracts without reading them.

<comment><USER14> What contract? Why do you mention one?

</comment></comment></comment><comment><USER15> The article is correct that competing for the kind of content that you find on the front page of glossy magazines in the supermarket lines is a race towards the bottom on the internet.

The article is also correct that this is because online aggregators unlike supermarket tabloid stands present no supply line incumbent advantage.

Offline content is a combination of ads plus upfront price. That requires some form of friction and seems to be where content is moving. This fits better with the necessary expectations of incumbent content providers that their way of reaching customers should have some incumbent distribution advantage, because in the end it needs to be a stable job for people providing the content and for the shareholders taking the risk of providing the company that create those jobs.

</comment><comment><USER16> The simultaneous layoffs in many news outlets prove they lost access to a common source of funding, perhaps that source was something like Countering Foreign Propaganda and Disinformation Act which recently expired.

</comment><comment><USER13> Why are we acting like a journo company laying off 1000 people is the end of the world? Are people not aware of how many other companies across many sectors have layoffs all the time? This isn't news. BuzzFeed sucks, their journalists were providing benefit to the business, so they pivoted. Can we move on?

<comment><USER17> It's more than that. buzzfeed is essentially a type of media company, one that is suppose to 'survive' in the new landscape where aggregates like fb, and google dominates. to learn and understand why it is struggling has more implications not just to the long term viability of journalism, but also to how aggregators dominate and there fore, should be regulated, and what can company do in a age where controlling demand is much more important than controlling supply.

<comment><USER13> maybe people are finally sick of trashy clickbait tabloid articles?

<comment><USER18> Not only that, but we really don't need journalists in the traditional sense as much as we used to. It wasn't even that long ago that newspapers and TV were people's eyes and ears. Now anyone with a phone can record video of events as they happen, and anyone with a YouTube channel can report on the news and make political commentary for a mere fraction of the overhead that media firms operate on.

This isn't to say that there isn't value in traditional journalism, and I think the viability of member-supported radio is a testament to that. But the old guard of for-profit journalism is obsolete, and it's quickening its own death by going hog-wild with unverified sources, hit pieces, and lack of fact checking.

<comment><USER19> _we really don 't need journalists in the traditional sense as much as we used to_

Are you under the impression that government corruption and injustice are things of the past?

 _Now anyone with a phone can record video of events as they happen, and anyone with a YouTube channel can report on the news and make political commentary for a mere fraction of the overhead that media firms operate on._

Except that we live in a world of deepfakes, astroturfing, and a dozen other reasons not to trust what is posted on the internet by any random person.

And building on my comment above, Random Joe isn't going to spend tens or hundreds of thousands of dollars investigating illegal arms sales, contract fraud, or hundreds of other things that big journalism companies do routinely.

 _member-supported radio is a testament to that_

Member-supported radio doesn't cover 1/1000th of what local newspapers do.

I think you may not understand the value of professional journalism, past and present, perhaps because you haven't been exposed to it, or its consequences.

<comment><USER18> > Are you under the impression that government corruption and injustice are > things of the past?

Oh, come now.

> Except that we live in a world of deepfakes, astroturfing, and a dozen other > reasons not to trust what is posted on the internet by any random person.

And professional journalists are going to be immune to these things? A lot of the time, journalists actually _repost_ what they see on Twitter and YouTube. I suppose that in an ideal world journalists would wise up in the face of deep fakes and trust their various sources a lot less, but caution != profit.

> And building on my comment above, Random Joe isn't going to spend tens or > hundreds of thousands of dollars investigating illegal arms sales, contract > fraud, or hundreds of other things that big journalism companies do > routinely.

I don't agree that you need Big Journalism for that job to be done. Individuals in the press who are outside Big Journalism, as you put it, can do the same job independently and for a fraction of the cost.

> Member-supported radio doesn't cover 1/1000th of what local newspapers do.

Sure, _local_ newspapers have an edge over public radio, but you can also use that same argument against Big Journalism, which despite its resources can't do the same job that local newspapers do.

> I think you may not understand the value of professional journalism, past > and present, perhaps because you haven't been exposed to it, or its > consequences.

That's not a fair assessment at all.

<comment><USER20> I don't comprehend the romance of journalism like the parent comment, especially in light of a Harvey Weinstein scandal which revealed he had a payroll of journalists. These are people that are profit motivated like any other human being, and will often kiss the ring rather than shine light upon it.

Idealism often dies when people start owning larger personal responsibilities.

</comment></comment></comment></comment></comment></comment></comment><comment><USER21> We need to stop calling people who write articles about memes or what random people are saying on twitter "Journalists".

Your listicle is not defending democracy, and confusing these two groups has led to a great devaluation of trust in the remaining real journalists.

<comment><USER22> That was the takeaway I got from reading this thread, a BuzzFeed person discussing BuzzFeed contributions: [https://twitter.com/katienotopoulos/status/10896299402266296...](https://twitter.com/katienotopoulos/status/1089629940226629632)

> Wonder why suddenly quizzes took over your Facebook? That was because of > @LouisPeitzman, who made one viral quiz that unleashed everything else (he > probably can apologize to you personally for that).

This isn't a layoff of journalists, its a layoff of custodians of low-brow content farm writing. If that's all, then the main problem in the journalism- apocalypse is that these people were called journalists in the first place.

<comment><USER23> I clicked on the twitter link, I don't know why really, I probably shouldn't have

> "A bunch of people were laid off (and more to come! Fuck!) who quite > literally created and shaped the internet as we know it today."

I think this what people mean when they talk about Poe's law [https://en.wikipedia.org/wiki/Poe%27s_law](https://en.wikipedia.org/wiki/Poe%27s_law). For a short while I thought I was reading a parody.

> "BuzzFeed changed the internet, for better or worse (if you say worse to me > right now please eat my ass)," ... "They gave moral clarity, heart, and a > mission that was never craven click-lust, but driven by serious > consideration of humans"

It seems the team builders there failed on the moral clarity bit. At least there is more work to be done, but time has run out it seems.

It's interesting to observe how confident they are that they shaped the internet for the better. Something tells this is not an anomaly and most of them share that belief. This is a great example self and group brainwashing. And notice the part where if you disagree with them, you get immediately attacked. Clearly the qualities of a sophisticated investigative journalist, ready to report news in an unbiased and objective manner.

</comment></comment></comment><comment><USER24> It's a bunch of people with insane degrees (literally a PhD in "romantic comedies", WTF why is this even possible) living in places they can't afford and writing nonsense.

These people loved telling factory workers and especially coal miners to "learn to code". You could tell there was a certain smug glee, taking perverse pleasure in the fact that aging white males (can't imagine any of the other coal miners) in MAGA hats (all coal miners wear them all day long, right?) would have to get educated and somehow enlightened.

Now these writers are annoyed that people on Twitter are saying they should learn to code. Well... that's just funny, and very well deserved.

Personally, I'd rather they didn't all learn to code. Steel mills are hiring. Mao said to make steel.

<comment><USER25> How is a PhD in romantic comedies any more ridiculous than a PhD focusing on the works of Ovid, Aristophanes, or Shakespeare?

<comment><USER26> One of these things enriches society and sheds light on the universal experiences in human existence.

The other fuels consumerism, depression, and relegates women to either damsels in distress, objects of conquest, or abuse victims.

<comment><USER27> > The other fuels consumerism, depression, and relegates women to either > damsels in distress, objects of conquest, or abuse victims.

I wonder in what ways, and what are the effects of such depictions. If only there was some group of people interested in researching such a thesis...

</comment></comment></comment></comment><comment><USER28> The author lost me at _" Back in 2015 I wrote that BuzzFeed [Was] the Most Important News Organization in the World."_

<comment><USER17> do you know who this author is?

<comment><USER29> who is it? i don't see an author listed on the page.

<comment><USER30> Well it's a one author site. It's Ben Thompson, he has been doing this for quite some time and is an important figure. Here is the about page: [https://stratechery.com/about/](https://stratechery.com/about/)

There is also his Wikipedia which gives a good amount of info: [https://en.wikipedia.org/wiki/Ben_Thompson_(writer)](https://en.wikipedia.org/wiki/Ben_Thompson_\(writer\))

</comment></comment></comment></comment><comment><USER31> 'journalism' and publishing is always about advertising revenue, and historically the big platforms with the most reach (Murdoch's tabloid newspaper empire, TV) were the big money earners because of number of eyeballs and ears served. There are no barriers to publishing online today but it is still all about the platform and number of people using it, and that platform today is arguably Google and FB/Insta/WA. Who creates and writes the 'journalism', however cogent, perceptive and well thought out is largely irrelevant. How often do you remember who specifically actually wrote an article on a site like Buzzfeed? You do a lot better building a niche brand like stratechery as Ben has done...

</comment><comment><USER32> "The inferior product is advertising"

Got that right. Partly because adverts are boring and uncreative ... much like Buzzfeed

"now advertisers ... can reach the exact customers [at] Facebook and Google."

That has yet to be demonstrated. Was it ever true? Is it still?

"Back in 2015 I wrote that BuzzFeed [Was] the Most Important News Organization in the World... So what went wrong?"

I glance at many newsfeeds every day. I found little value in Buzzfeed. So, you were wrong.</comment>


Ember 3.0 Released - <AUTHOR> https://emberjs.com/blog/2018/02/14/ember-3-0-released.html <comment><USER2> I stopped working with ember about a year and a half ago and it's interesting to see how major progress seems to have halted. It became pretty obviously hindered even before then with the controllers to routable components initiative, but I can't seem to . I don't check in on it often enough to be an expert in the project now, but I get a feeling that the new glimmer stuff is a similarly distracting big idea that will get in the way of many smaller improvements that could go out in that time.

That said I do still sometimes miss ember, specifically ember data. I was pretty hesitant to start using using ember with all of it's magic, but having all of that magic really felt like it kept us from over thinking most small problems we had to solve. Also the codebase was extremely easy to digest, so things that seemed like the framework was standing in the way of usually only required an hour of poking through the library code and you could build out functionality you needed with private APIs. Got really familiar with the data codebase and got to pull off some fun hacks.

<comment><USER3> From my perspective, progress hasn't halted but there was a deliberate effort to reign in the pace of changes. I appreciate how much easier the migrations were through the 2.x series vs 1.x.

Changes to the framework are well documented and the roadmap is easy to find: [https://emberjs.com/statusboard/](https://emberjs.com/statusboard/)

</comment></comment><comment><USER4> Anyone knows why the script-tag functionality is gone?

I have always used PHP + EmberJS via script tags so it seems very unfamiliar to start using NodeJS instead :)

</comment><comment><USER5> Brave decision of actually removing code instead of bloating with more features. Well done!

<comment><USER3> I think Ember point releases are typically focused on removing long-deprecated features so that upgrade is super easy. New stuff gets added/deprecated in between.</comment></comment>


Is blogging dead? - <AUTHOR> http://eisokant.com/2008/07/14/is-blogging-dead/

<comment><USER2> "Jason is right when he is saying that these days blogs are often no longer about the integrity of the content but the number of backlinks it receives, the number of page views and how high it ranks in Google."

I started blogging over 2 years ago and that's exactly how it was back then. It's always about readership, even if one doesn't run ads. Sure, people play games to be seen sometimes, but it's like television, or maybe any media. A few people will find short-term success (reality TV stars) and serious content providers will find longer life (think 60 Minutes).

Sure, the technology will change as well, but as long as the internet remains reasonably neutral, people will blog/micorblog/videoblog/etc.

</comment><comment><USER3> I've been blogging on and off since 1999 and I've seen a lot of friends and other good bloggers stop maintaining their sites over the years for various reasons. Some got married and didn't have their single-person adventures to write about anymore; others got discouraged when they looked at their visitor stats; still others just lost interest.

But for the most part, many people who have stopped blogging have come to the realization that maintaining a good blog takes time -- lots of it. A good, thoughtful post can take a couple hours to write and edit. And maintaining an overall theme for your blog that's compelling and not just "random stuff I post for the occasional friend who drops by" is even harder. In other words, having a good blog is like creating and managing a product.

As many bloggers begin to realize this, and begin to see other options (microblogging, etc.) they move on. But that's not to say that the entire form of media called blogging is dead. It just means it's stabilizing and becoming recognized as just one of the many options out there for publishing content, as opposed to being the "everyone must have" option it once was.

</comment><comment><USER4> No.

In related news: The practice of posting controversial link-bait to attract attention from other bloggers is also alive and well.

</comment><comment><USER5> Yours is. Yikes.

Fix your CSS so we can read your article in IE7.

<comment><AUTHOR> Thank you for letting me know, I am going to get right on it!

</comment></comment><comment><USER6> No.

</comment><comment><USER7> No. I still make $12 a day through backhat blogging :P. It can't be dead to me.

</comment><comment><USER8> blogging is not dead, it just went mainstream.</comment>


0.30000000000000004 - <AUTHOR> https://0.30000000000000004.com/ <comment><USER2> The big issue here is what you're going to use your numbers for. If you're going to do a lot of fast floating point operations for something like graphics or neural networks, these errors are fine. Speed is more important than exact accuracy.

If you're handling money, or numbers representing some other real, important concern where accuracy matters, most likely any number you intend to show to the user as a number, floats are not what you need.

Back when I started using Groovy, I was very pleased to discover that Groovy's default decimal number literal was translated to a BigDecimal rather than a float. For any sort of website, 9 times out of 10, that's what you need.

I'd really appreciate it if Javascript had a native decimal number type like that.

<comment><USER3> Decimal numbers are not conceptually any more or less exact than binary numbers. For example, you can't represent 1/3 exactly in decimal, just like you can't represent 1/5 exactly in binary.

When handling money, we care about _faithfully reproducing the human-centric quirks of decimal numbers_ , not "being more accurate". There's no reason in principle to regard a system that can't represent 1/3 as being fundamentally more accurate because it happens to be able to represent 1/5.

<comment><USER4> Money are really best dealt with as integers, any time you'd use a non-integer number, use some fixed multiple that makes it an integer, then divide by the excess factor at the end of the calculation. For instance computing 2.15% yearly interest on a bank account might be done as follows:

      DaysInYear = 366
      InterestRate = 215
      DayBalanceSum = 0
      for each Day in Year
        DayBalanceSum += Day.Balance
      InterestRaw = DayBalanceSum * InterestRate
      InterestRaw += DaysInYear * 5000

      Interest = InterestRaw / (DaysInYear * 10000)
      Balance += Interest

Balance should always be expressed in the smallest fraction of currency that we conventionally round to, like 1 yen or 1/100 dollar. Adding in half of the divisor before dividing effectively turns floor division into correctly rounded division.

<comment><USER5> This is called fixed-point arithmetic:

[https://en.wikipedia.org/wiki/Fixed- point_arithmetic](https://en.wikipedia.org/wiki/Fixed-point_arithmetic)

> In computing, a fixed-point number representation is a real data type for a > number that has a fixed number of digits after (and sometimes also before) > the radix point.

> A value of a fixed-point data type is essentially an integer that is scaled > by an implicit specific factor determined by the type.

<comment><USER4> Yeah, though that notion tends to come with some conceptual shortcomings, like presuming a power of 10 radix. In the above code the radix is implicitly different on leap years, applying such tricks is usually not possible with a fixed point library or language construct.

</comment></comment></comment></comment></comment><comment><USER6> MS Excel tries to be clever and disguise the most common places this is noticed.

Give it =0.1+0.2-0.3 and it will see what you are trying to do and return 0.

Give it anything slightly more complicated such as =(0.1+0.2-0.3) and this won't trip, in this example displaying 5.55112E-17 or similar.

<comment><USER7> Are you sure it is not showing the exact answer because the the the cell precision set to a single decimal digit?

<comment><USER8> Yup: [https://i.imgur.com/VuawaE1.png](https://i.imgur.com/VuawaE1.png), on Excel v1911 (Build 12228.20332).

</comment></comment></comment><comment><USER9> I remember in college when we learned about this and I had the thought, "Why don't we just store the numerator and denominator?", and threw together a little C++ class complete with (then novel, to me) operator-overloads, which implemented the concept. I felt very proud of myself. Then years later I learned that it's a thing people actually use: [https://en.wikipedia.org/wiki/Rational_data_type](https://en.wikipedia.org/wiki/Rational_data_type)

<comment><USER10> It's actually in use in many places, for things like handling currency and money, and for when you get funny corner cases involving rounding such numbers and pooling the change.

Whenever I see someone handling currency in floats, something inside me wither and die a small death.

<comment><USER9> I haven't worked in fintech but I've read that money is often represented (at least in storage) as plain integers, since for example US currency only ever goes to two decimal places. But I guess once you start operating on it you run into potential truncation unless you use rationals.

<comment><USER11> US currency can go to more than two decimal places...

[http://blogs.reuters.com/ben-walsh/2013/11/18/do-stocks- real...](http://blogs.reuters.com/ben-walsh/2013/11/18/do-stocks-really-trade- for-fractions-of-a-penny-sort-of/)

I guess it's time for someone to write an "Assumptions Programmers make about money" post.

<comment><USER12> No it can't. There are systems that track things worth less than a penny for later billing, but at the end of the month when they bill someone, they do some sort of rounding.

<comment><USER13> Someone should tell that to everyone who ever used a ½¢ coin in the US. Also, US law explicitly states (31 USC §5101) that the unit of 1/1000th of a dollar is a mill.

<comment><USER14> > _Someone should tell that to everyone who ever used a ½¢ coin in the US._

All 10 of them?

<comment><USER15> I've got some at home, but admittedly I'd never use one as currency. I also have US ½¢ paper notes too.

</comment></comment></comment></comment></comment></comment></comment></comment><comment><USER16> A thread from 2017.00000000000: [https://news.ycombinator.com/item?id=14018450](https://news.ycombinator.com/item?id=14018450)

2015.000000000000: [https://news.ycombinator.com/item?id=10558871](https://news.ycombinator.com/item?id=10558871)

<comment><USER3> FWIW, both of those can be expressed exactly by floating-point numbers ;)

<comment><USER17> Right all integers up to 2^53 (or something like that) can be (in double precision).

I assume that's the reason they made the mantissa linear, even though having the whole thing logarithmic makes more sense.

<comment><USER18> What. Mantissa is already logarithmic, bit number _n_ has value _2^(n - N-1)_ for an _N_ -bit mantissa.

This is how positional number systems work at all.

<comment><USER17> The mantissa is linear. It's unrelated to how positional number systems work. A floating point value is split into two numbers - the exponent and the mantissa. Normally they are used to represent a final number like:

        x = 2^e * (1 + m)

Where e is the exponent and m is the mantissa (varying _linearly_ from 0 to 1).

But you could have a fully exponential number format:

        x = 2^(m + o)

As pointed out though, it makes addition much more complicated, you can't exactly represent integers, and someone told me it makes quantisation noise worse too. Bad idea.

</comment></comment></comment></comment></comment><comment> mark-r Also the subject of one of the most popular questions on StackOverflow: [https://stackoverflow.com/q/588004/5987](https://stackoverflow.com/q/588004/5987)

</comment><comment><USER19> While it's true that floating point has its limitations, this stuff about not using it for money seems overblown to me. I've worked in finance for many years, and it really doesn't matter that much. There are de minimis clauses in contracts that basically say "forget about the fractions of a cent". Of course it might still trip up your position checking code, but that's easily fixed with a tiny tolerance.

<comment><USER20> when the fractions actually dont matter... its so painless just to just store everything in pennies rather than dollars (multiply everything by 100)

<comment><USER21> It's not painless. E.g. dividing $100.00 by 12 month in integer cents requires 11 $8.33 and one $8.37 (or better 4x(2x8.33+8.34), depending on definition of 'better'). You can forget this $0.04, but it will jump around in reports until you get rid of it – it requires someone's attention anyway, no matter how small it is. Otoh, in unrounded floating point that will lead to a mismatch between (integer) payments and calculations. In rounded fp it's the same problem, except when you're trying very hard for error bits to accumulate (like cross-multiplying dataset sums with no intermediate rounding, which is nonsense in financial calc and where regular fixpoint integers will overflow anyway).

What I'm trying to show here is that _both_ integers and floating point are not suitable for doing 'simple' financial math. But we get used to this Bresenhamming in integers and do not perceive it as solving an error correction problem.

<comment><USER22> This struck home with me when one day a friend and I bought the same thing and he paid a penny more.

I realized something I didn't ever notice or appreciate in 20+ years: oh yeah, they can't just round off the penny in their favour every time. And the code that handles tracking when to charge someone an extra penny must be irritating to have developed and manage. All of a sudden you've got state.

<comment><USER23> What kind of thing and store was it?

</comment></comment></comment></comment></comment><comment> GuB-42 That's one of the worst domain name ever. When the topic comes along, I always remember about "that single-serving website with a domain name that looks like a number" and then take a surprisingly long time searching for it.

I have written a test framework and I am quite familiar with these problems, and comparing floating point numbers is a PITA. I had users complaining that 0.3 is not 0.3.

The code managing these comparisons turned out to be more complex than expected. The idea is that values are represented as ranges, so, for example, the IEEE-754 "0.3" is represented as ]0.299~, 0.300~[ which makes it equal to a true 0.3, because 0.3 is within that range.

<comment><USER24> It's the first result for "floating point site" on Google. Sure the domain itself is impossible to remember, but you don't have to remember the actual number, just what it stands for.

<comment><USER25> Remember filter bubble. My first result is not your first result. (although in this case it happens to be, but we both probably search a lot on programming)

<comment><USER24> Also did it in an InPrivate window to confirm, which is still somewhat targeted but far less so than on my actual account. It's still first.

And, at the end of the day, even if there's a filter bubble and it's the reason I see it first, then so what? The people looking for this site are likely going to fit into the same set of targeted demographics as you and me and most people on this site. So unless you also want to cater to 65-year old retirees that don't care about computer science and what floating numbers are, then why does the filter bubble even matter?

</comment></comment></comment></comment><comment><USER26> This is a good thing to be aware of.

Also the "field" of floating point numbers is not commutative†, (can run on JS console:)

x=0;for (let i=0; i<10000; i++) { x+=0.0000000000000000001; }; x+=1

\--> 1.000000000000001

x=1;for (let i=0; i<10000; i++) { x+=0.0000000000000000001; };

\--> 1

Although most of the time a+b===b+a can be relied on. And for most of the stuff we do on the web it's fine!††

† edit: Please s/commutative/associative/, thanks for the comments below.

†† edit: that's wrong! Replace with (a+b)+c === a+(b+c)

<comment><USER27> Note that the addition is commutative [1], i.e. a+b==b+a always.

What is failing is associativity, i.e. (a+b)+c==a+(b+c)

For example

(.0000000000000001 + .0000000000000001 ) + 1.0

\--> 1.0000000000000002

.0000000000000001 + (.0000000000000001 + 1.0)

\--> 1.0

In your example, you are mixing both properties,

(.0000000000000001 + .0000000000000001) + 1.0

\--> 1.0000000000000002

(1.0 + .0000000000000001) + .0000000000000001

\--> 1.0

but the difference is caused by the lack of associativity, not by the lack of commutativity.

[1] Perhaps you must exclude -0.0. I think it is commutative even with -0.0, but I'm never 100% sure.

<comment><USER28> I tried to determine how to perform IEEE 754 addition (in order to see whether it's commutative) by reading the standard: [https://sci- hub.tw/10.1109/IEEESTD.2019.8766229](https://sci- hub.tw/10.1109/IEEESTD.2019.8766229)

(Well, it's a big document. I searched for the string "addition", which occurs just 41 times.)

I failed, but I believe I can show that the standard requires addition to be commutative in all cases:

1\. "Clause 5 of this standard specifies the result of a single arithmetic operation." (§10.1)

2\. "All conforming implementations of this standard shall provide the operations listed in this clause for all supported arithmetic formats, except as stated below. Unless otherwise specified, each of the computational operations specified by this standard that returns a numeric result shall be performed as if it first produced an intermediate result correct to infinite precision and with unbounded range, and then rounded that intermediate result, if necessary, to fit in the destination's format" (§5.1)

Obviously, addition of real numbers is commutative, so the intermediate result produced for addition(a,b) must be equal to that produced for addition(b,a). I hope, but cannot guarantee, that the rounding applied to that intermediate result would not then depend on the order of operands provided to the addition operator.

3\. "The operation addition(x, y) computes x+y. The preferred exponent is min(Q(x), Q(y))." (§5.4.1). This is the entire definition of addition, as far as I could find. (It's also defined, just above this statement, as being a general-computational operation. According to §5.1, a general-computational operation is one which produces floating-point or integer results, rounds all results according to §4, and might signal floating-point exceptions according to §7.)

4\. The standard _encourages_ programming language implementations to treat IEEE 754 addition as commutative (§10.4):

> A language implementation preserves the literal meaning of the source code > by, for example:

> \- Applying the properties of real numbers to floating-point expressions > only when they preserve numerical results and flags raised:

> \-- Applying the commutative law only to operations, such as addition and > multiplication, for which neither the numerical values of the results, nor > the representations of the results, depend on the order of the operands.

> \-- Applying the associative or distributive laws only when they preserve > numerical results and flags raised.

> \-- Applying the identity laws (0 + x and 1 × x) only when they preserve > numerical results and flags raised.

This looks like a guarantee that, in IEEE 754 addition, "the representation of the result" (i.e. the sign/exponent/significand triple, or a special infinite or NaN value - §3.2) does not "depend on the order of the operands". §3.2 specifically allows an implementation to map multiple bitstrings ("encodings") to a single "representation", so it's possible that the bit pattern of the result of an addition may differ depending on the order of the addends.

5\. "Except for the quantize operation, the value of a floating-point result (and hence its cohort) is determined by the operation and the operands' values; it is never dependent on the representation or encoding of an operand."

"The selection of a particular representation for a floating-point result is dependent on the operands' representations, as described below, but is not affected by their encoding." (both from §5.2)

HOWEVER...

6\. §6, dealing with infinite and NaN values, implicitly contemplates that there might be a distinction between addition(a,b) and addition(b,a):

> Operations on infinite operands are usually exact and therefore signal no > exceptions, including, among others,

> \- addition(∞, x), addition(x, ∞), subtraction(∞, x), or subtraction(x, ∞), > for finite x (§6.1)

</comment></comment></comment><comment><USER29> I feel like it should really be emphasised that the reason this occurs is due to a mismatch between binary exponentiation and decimal exponentiation.

 _0.1 = 1 × 10^-1_ , but there is no integer significand _s_ and integer exponent _e_ such that _0.1 = s × 2^e_.

When this issue comes up, people seem to often talk about fixing it by using decimal floats or fixed-point numbers (using some _10^x_ divisor). If you change the base, you solve the problem of representing _0.1_ , but whatever base you choose, you're going to have unrepresentable rationals. Base 2 fails to represent _1 /10_ just as base 10 fails to represent _1 /3_. All you're doing by using something based around the number _10_ is supporting numbers that we expect to be able to write on paper, not solving some fundamental issue of number representation.

Also, binary-coded decimal is irrelevant. The thing you're wanting to change is _which_ base is used, not how any integers are represented in memory.

<comment><USER30> Agree. All of these floating point quirks are not actually problems if you think of them as being finite precision approximations to real numbers, not in any particular base. Just like physical measurements of continuous quantities. You wouldn't be surprised to find an error in the 15th significant figure of some measurement or attempt to compare them for equality or whatever. So don't do it with floating point numbers either and everything will work perfectly.

Yes, there are some exceptions where you can reliably compare equality or get exact decimal values or whatever, but those are kind of hacks that you can only take advantage of by breaking the abstraction.

</comment></comment><comment><USER31> One small tip about printf for floating point numbers. In addition to "%f", you can also print them using "%g". While the precision specifier in %f refers to digits after the decimal period, in %g the precision refers to the number of significant digits. The %g version is also allowed to use exponential notation, which often results in more pleasant-looking output than %f.

       printf("%.4g", 1.125e10) --> 1.125e+10
       printf("%.4f", 1.125e10) --> 11250000000.0000

<comment><USER32> And %e _always_ uses exponential notation. Then there's %a, which can be exact for binary floats.

</comment></comment><comment><USER33> One of my favorite things about Perl 6 is that decimal-looking literals are stored as rationals. If you actually want a float, you have to use scientific notation.

Edit: Oh wait, it's listed in the main article under Raku. Forgot about the name change.

</comment><comment><USER34> That's only formatting.

The other (and more important) matter, — that is not even mentioned, — is comparison. E. g. in "rational by default in this specific case" languages (Perl 6),

      > 0.1+0.2==0.3
      True

Or, APL (now they are floats there! But comparison is special)

          0.1+0.2
      0.3
          ⎕PP←20 ⋄ 0.1+0.2
      0.30000000000000004
          (0.1+0.2) ≡ 0.3
      1

<comment><USER35> Exactly what are the rules for the "special comparison" in APL? That sounds horrifying to me.

<comment><USER34> Assume the values could be equal if the relative error of the operation is greater than a small predefined value (called "⎕ct", comparison tolerance, and you can change it).

<comment><USER36> but this is not an equivalence relation. You may have a=b and b=c but a!=c

it's horrifying!

</comment></comment></comment></comment><comment><USER37> The runner up for length is FORTRAN with: 0.300000000000000000000000000000000039

And the length (but not value) winner is GO with: 0.299999999999999988897769753748434595763683319091796875

<comment><USER38> Those look like the same length

<comment><USER39> Huh? The fortran one is 38 characters long with 33 0s after the 3. The go one is 56 characters long with 15 9s after the 2.

<comment><USER38> I'm on mobile. Must be the issue.

</comment></comment></comment></comment><comment><USER40> > It's actually pretty simple

The explanation then goes on to be very complex. e.g. "it can only express fractions that use a prime factor of the base".

Please don't say things like this when explaining things to people, it makes them feel stupid if it doesn't click with the first explanation.

I suggest instead "It's actually rather interesting".

<comment><USER41> Ditto as I now feel stupid.

I read the rest of your reply but I also haven't let go of the possibility that we're both (or precisely 100.000000001% of us collectively) are as thick as a stump.

<comment><USER42> To be fair, this is also done in every other STEM field, and CS is no exception.

We could all learn a lot more from each other if everything wasn't a contest all the time.

</comment></comment></comment><comment><USER43> Postgresql figured this out many years ago with their Decimal/Numeric type. It can handle any size number and it performs fractional arithmetic perfectly accurately - how amazingly for the 21st Century! Is comically tragic to me that all of the mainstream programming languages are still so far behind, so primitive that they do not have a native accurate number type that can handle fractions.

<comment><USER44> > how amazingly for the 21st Century!

Most languages have classes for that, some had them for decades in fact. Hardware floating point numbers target performance and most likely beat any of those classes by orders of magnitude.

</comment></comment><comment><USER45> I still remember when I encountered this and nobody else in the office knew about it either. We speculated about broken CPUs and compilers until somebody found a newsgroup post that explained everything. Makes me wonder why we haven't switched to a better floating point model in the last decades. It will probably be slower but a lot of problems could be avoided.

<comment><USER29> Unless you have a floating point model that supports arbitrary bases, you're always going to have the issue. Binary floats are unable to represent 1/10 just as decimal floats are unable to represent 1/3.

And in case anyone's wondering about handling it by representing the repeating digits instead, here's the decimal representation of 1/12345 using repeating digits:

      0.0[0008100445524503847711624139327663021466180639935196435803969218307006885378
      69582827055488051842851356824625354394491697043337383556095585257189145402997164
      84406642365330093155123531794248683677602268124746861077359254759011745646010530
      57918185500202511138112596192790603483191575536654515998379910895099230457675172
      13446739570676387201296071283920615633859862292426083434588902389631429728635074
      92912110166059133252328878088294856217091940056703118671526933981368975293641150
      26326447954637505062778452814904819765087079789388416362899959497772377480761441
      87930336168489266909680032401782098015390846496557310652085864722559740785743215
      87687322802754151478331308221952207371405427298501417577966788173349534224382341
      02875658161198865937626569461320372620494127176994734710409072498987444309437019
      03604698258404212231672742]

<comment><USER46> > Unless you have a floating point model that supports arbitrary bases

See also binary coded decimals.

[https://en.wikipedia.org/wiki/Binary- coded_decimal](https://en.wikipedia.org/wiki/Binary-coded_decimal)

<comment><USER47> That's not a floating point.

<comment><USER46> From the article:

> Programmable calculators manufactured by Texas Instruments, Hewlett-Packard, > and others typically employ a floating-point BCD format, typically with two > or three digits for the (decimal) exponent.

<comment><USER47> Then that's how they're encoding the components of the float. BCD itself is not a floating-point, it's just a different way of encoding a fixed-point or integer. If all you want to do is use floating point but expand the logarithm and mantissa then that's completely tangential to whether or not they're stored as BCD or regular binary values.

</comment></comment></comment></comment></comment></comment><comment><USER48> In JavaScript, you could use a library like decimal.js. For simple situations, could you not just convert the final result to a precision of 15 or less?

      > 0.1 + 0.2;
      < 0.30000000000000004

      > (0.1 + 0.2).toPrecision(15);
      < "0.300000000000000"

From Wikipedia: "If a decimal string with at most 15 significant digits is converted to IEEE 754 double-precision representation, and then converted back to a decimal string with the same number of digits, the final result should match the original string." \--- [https://en.wikipedia.org/wiki/Double- precision_floating-poin...](https://en.wikipedia.org/wiki/Double- precision_floating-point_format)

</comment><comment><USER49> That is why I only used base 2310 for my floating point numbers :-). FWIW there are some really interesting decimal format floating point libraries out there (see [http://speleotrove.com/decimal/](http://speleotrove.com/decimal/) and [https://github.com/MARTIMM/Decimal](https://github.com/MARTIMM/Decimal)) and the early computers had decimal as a native type ([https://en.wikipedia.org/wiki/Decimal_computer#Early_compute...](https://en.wikipedia.org/wiki/Decimal_computer#Early_computers))

<comment><USER50> The multiplication of the first 5 primes ;)

</comment></comment><comment><USER51> This is part of the reason Swift Numerics is helping to make it much nicer to do numerical computing in Swift.

[https://swift.org/blog/numerics/](https://swift.org/blog/numerics/)

<comment><USER52> Swift also has decimal (so does objective-c) which handles this properly. See [https://lists.swift.org/pipermail/swift-users/Week-of- Mon-20...](https://lists.swift.org/pipermail/swift-users/Week-of- Mon-20161219/004220.html) to see how swift's implementation of decimal differs from obj-c.

</comment></comment><comment><USER53> This is a great shibboleth for identifying mature programmers who understand the complexity of computers, vs arrogant people who wonder aloud how systems developers and language designers could get such a "simple" thing wrong.

<comment><USER54> " vs arrogant people who wonder aloud how systems developers and language designers could get such a "simple" thing wrong."

I never heard anyone complain that it would be simple to fix. But complaining? Yes - and rightfully so. Not every webprogrammer need to know the hw details and don't want to, so it is understandable that this causes irritation.

</comment></comment><comment><USER55> Interesting, I searched for "1.2-1.0" on google. The calculator comes up and it briefly flashes 0.19999999999999996 (and no calculator buttons) before changing to 0.2. This happens inconsistently on reload.

</comment><comment><USER56> Swi-Prolog (listed int he article) also supports rationals:

      ?- A is rationalize(0.1 + 0.2), format('~50f~n', [A]).
      0.30000000000000000000000000000000000000000000000000
      A = 3 rdiv 10.

</comment><comment><USER57> This specific issue nearly drove me insane trying to debug a SQL -> C++/Scala/OCaml transpiler years ago. We were using the TPC-H benchmark as part of our test suite, and (unbeknownst to me), the validation parameters for one of the queries (Q6) triggered this behavior (0.6+0.1 != 0.7), but only in the C/Scala targets. OCaml (around which we had built most of our debugging infrastructure) handled the math correctly...

Fun times.

</comment><comment><USER58> When did RFC1035 get thrown under the bus? According to it, with respect to domain name labels, "They must start with a letter" (2.3.1).

<comment><USER59> Long, long ago. 3com.com wanted to exist.

<comment><USER60> Amazingly, 3.com apparently _didn 't_ want to exist.

</comment></comment></comment><comment><USER61> I wish high level languages (specifically python) would default to using decimal, and only use a float when cast specifically. From what I understand that would make things slower, but as a higher level language you're already making the trade of running things slower to be easier to understand.

That said, it's one of my favorite trivia gotchas.

</comment><comment><USER62> Fixed-point calculations seem to be somewhat of a lost art these days.

It used to be widespread because floating point processors were rare and any floating point computation was costly.

That's not longer the case and everyone seems to immediately use floating point arithmetic without being fully aware of the limitations and/or without considering the precision needed.

</comment><comment><USER63> As soon as I've started developing real-life business apps I've started to dream about a POWER which is said to have hardware decimal type support. Javs's BigDecimal solves the problem on x86 but it is at least an order of magnitude more slow than FPU-accelerated types.

<comment><USER64> Well, if your decimals are fixed-point decimals, which is the case in finance, decimal calculations are very cheap integer calculations (with simple additional scaling in multiplication/division).

I just use Zarith (bignum library) in OCaml for decimal calculation, and pretty content with performance.

I don't think much domains needs decimal floating point that much, honestly, at least in finance and scientific calculations.

But I could be wrong, and would be interested in cases where decimal floating- point calculations are preferable over these done in decimal fixed-point or IEEE floating-point ones.

<comment><USER63> Why doesn't everybody do it this way then? We would probably have a transparent built-in decimal type in every major language by now if there were no problems with this.

<comment><USER64> > Why doesn't everybody do it this way then?

Why? Fintech uses decimal fixed-point all the way, there are libraries for them for any major language. Apps like GnuCash or ledger use them as well.

<comment><USER63> But Java has BigDecimal in its standard library and it's soooo slow I doubt it is implemented this way.

</comment></comment></comment></comment></comment><comment><USER65> In the Go example, can someone explain the difference between the first and the last case?

<comment><USER66> There's a link right below. It seems like

1\. Constants have arbitrary precision 2\. When you assign them, they lose precision (example 2) 3\. You can format at as a arbitrary precision in a string (example 3)

In that last example, they are getting 54 significant digits in base 10.

<comment><USER65> Thanks. What I didn't realize is that although the sum is done precisely, the resulting 0.3 will be represented approximately once converted to float64. In the first case formatting hides that, in the last it doesn't.

<comment><USER66> I think in the last example, it's going straight from arbitrary precision to 54 significant digit, bypassing float64 entirely, hence why it looks different from the middle example.

</comment></comment></comment></comment><comment><USER67> Mods: Can we have a top level menu option called "Floating point explained"?

</comment><comment><USER68> Not surprisingly Common Lisp gets it right. I don't mean this is snark (I don't mean to imply you are a weenie if you don't use lisp) but just to show that it picked a different kind of region in the language design domain.

</comment><comment><USER69> Computer languages should default to fixed precision decimals and offer floats with special syntax (eg "0.1f32").

The status quo is that even Excel defaults to floats and wrong calculations with dollars and cents are widespread.

</comment><comment><USER22> The thing that surprised me the most (because I never learned any of this in school) was not just the lack of precision to represent some numbers, but that precision falls off a cliff for very large numbers.

</comment><comment><USER70> TL;DR - 0.1 in Base 2 (binary) is the equivalent of 1/3 in Base 10 meaning, it's a repeating decimal that causes rounding issues (0.333333 repeating)

This is why you should never do "does X == 0.1" because it might not evaluate accurately

</comment><comment><USER71> Whoo go Ada, one of the few to get it right. Must be the goto for secure programming for a reason.

Take that Rust and C ; )

</comment><comment><USER72> Happy to see ColdFusion doing it right. Also, good for Julia for having the support for fractions.

</comment><comment><USER73> I love how Awk, bc, and dc all DTRT. I wonder what postscript(/Ghostscript?) does.

</comment><comment><USER74> for Smalltalk, the list is not complete, it has scalled decimals and fractions too: 0.1s + 0.2s = 0.3s . (1/10) + (2/10) = (3/10)

</comment><comment><USER75> Those Babylonians were ahead of their time.

</comment><comment><USER76> Use Int types for programming logic.

<comment><USER77> Except when, you know, you can't.

<comment><USER3> Curious, when can't you?

My mental model of floating-point types is that they are useful for scientific/numeric computations where values are sampled from a probability distribution and there is inherently noise, and not really useful for discrete/exact logic.

<comment><USER77> Right; for the former integer arithmetic won't do.

<comment><USER3> Yep, absolutely (and increasingly often people are using 16-bit floats on GPUs to go even faster).

But the person you replied to said programming _logic_ , not programming anything.

Honestly I think if you care about the difference between `<` and `<=`, or if you use `==` ever, it's a red flag that floating-point numbers might be the wrong data type.

</comment></comment></comment></comment></comment><comment><USER78> Why is D different than the rest?!

</comment><comment><USER79> bc actually computes this correctly, and returns 0.3 for 0.1 + 0.2

</comment><comment><USER80> This has been posted here many times before. It even got mocked on n-gate in 2017 [http://n-gate.com/hackernews/2017/04/07/](http://n-gate.com/hackernews/2017/04/07/)

</comment><comment><USER81> lol what

</comment><comment><USER82> Please check some of the online papers on Posit numbers and Unum computing, especially by John Gustafson. In general, Unums can represent more numbers, with less rounding, and fewer exceptions than floating points. Many software and hardware vendors are starting to do interesting work with Posits.

<comment><USER83> Probably one of the more in depth technical discussions of the pros and cons of the various proposals that John Gustafson has made over the years:

[https://discourse.julialang.org/t/posits-a-new-approach- coul...](https://discourse.julialang.org/t/posits-a-new-approach-could-sink- floating-point-computation/26176)

</comment></comment><comment><USER84> IEEE floating-point is disgusting. The non-determinism and illusion of accuracy is just wrong.

I use integer or fixed-point decimal if at all possible. If the algorithm needs floats, I convert it to work with integer or fixed-point decimal instead. (Or if possible, I see the decimal point as a "rendering concern" and just do the math in integers and leave the view to put the decimal by whatever my selected precision is.)

<comment><USER77> IEEE is deterministic and (IMO) quite well thought-out. What specifically do you not like about it?

<comment><USER84> The fact that the most trivial floating-point addition of 0.1 + 0.2 = 0.300000000000004 was insufficient to make this seem HUMAN-nondeterministic to you? (I mean sure, if you thoroughly understood the entire spec, you might not be surprised by this result, but many people would be! Otherwise the original post and website would not exist, no?)

It's kind of a hallmark of bad design when you have to go into a long-winded explanation of why even trivial use-case examples have "surprising" results.

<comment><USER13> ⅓ to 3 decimal places is 0.333. 0.333 + 0.333 = 0.666, which is not ⅔ (to 3 decimal places, that is 0.667). That is all that is happening with the 0.1 + 0.2.

The word you're looking for is "surprising," which is a far cry from non- deterministic. IEEE 754 is so thoroughly deterministic that there exists compiler flags whose sole purpose is to say "I don't care that my result is going to be off by a couple of bits from IEEE 754."</comment></comment></comment></comment>


Western Spy Agencies secretly rely on hackers for intel and expertise - <AUTHOR> https://firstlook.org/theintercept/2015/02/04/demonize-prosecute-hackers-nsa-gchq-rely-intel-expertise/

<comment><USER2> wow, i bet some of the other prolific 'security researchers' are pissed they arent mentioned. On the other hand, those mentioned are probably rethinking their disclosure and OPSEC practices, and its interesting that at least one of the names on the list isnt even legitimately a researcher. This article, if true, I feel represents a by-product of the collusion between spooks and hackers at conferences, gladhanding, coin-swapping, etc. As a former fighter, I learned that out of battle its respectful etiquette to befriend an opponent, or someone once seen as an adversary. But now I think weve seen that sometimes its better to avoid the neutral ground altogether. I now feel dirty that I originally applauded when gen alexander swapped coins with Dave Kennedy (who is a great human being). I was sucking up to the new order, and I can come clean and admit it.</comment>


Google: America's third political party - <AUTHOR> http://news.cnet.com/8301-17852_3-57585208-71/google-americas-third-political-party/ <comment><USER2> I find the comparison suggesting, but otherwise useless. All human endeavors have a political factor, but "political party" evokes something else. Many ads promise to make our lives better, yet they aren't call political.</comment>


Global warming is speeding up Earth's ocean currents - <AUTHOR> https://www.sciencemag.org/news/2020/02/global-warming-speeding-earth-s-massive-ocean-currents <comment><USER2> Fair theory, but climate models haven't included particle forcing in their data sets. The IPCC just began including this data in their releases for climate modelers to work with.

</comment><comment><USER3> See, I would think that it will do the opposite. As the poles heat up faster, it would lower the gradient between the hot waters in the equator and the cold waters at the poles that drive the thermo haline cycle.

At least thats what is happening in the atmosphere. Its why the polar jet stream is weaker, and rossby waves are more prone to break off and send a polar vortex to the US, because the gradient across the boundary is lessened by the quickly warming poles.

</comment><comment><USER4> Duh. Nearly everything they've predicted has come to pass, including this. This is just confirmation of a hypothesis.</comment>


The development of solar + storage - <AUTHOR> http://www.vox.com/2016/2/5/10919082/solar-storage-economics <comment><USER2> Individual solar storage seems like a neat idea, but limited to single-family houses. Major piece of feedback SolarCity received after the first wave of Tesla Powerwall installations was how customers hated giving up space in their garage [http://www.greentechmedia.com/articles/read/Lesson- Learned-F...](http://www.greentechmedia.com/articles/read/Lesson-Learned-From- SolarCitys-First-Home-Energy-Storage-Installs)

So with SolarCity at least, the focus seems to be on community storage, where the grid of replaceable batteries is shared by neighbors on the same cul-de- sac/block/street/community/town. Long-term this is also insurance against net metering going away (which utilities will probably lobby through successfully at some point). Via community storage neighbors can start selling to neighbors.

However, someone still needs to maintain the neighborhood storage system, manage billing for produced/consumed kWh, run the actual wires to/fro storage, at which point we've re-invented the utility company, just at much smaller scale.

<comment><USER3> I'm not sure what the subsidies story looks like for rooftop solar vs grid solar, so I'm not sure if that is distorting the distribution atm.

In some places the cost of maintaining a distribution network may be higher than maintaining a distributed installation/maintenance network (eg Hawaii seems about right), but I don't see that being the case in urban areas.

So I feel like it's more likely that (well run) utilities will pick up solar+storage when it becomes cost competitive, and prices should fall, causing the "savings" obtained from rooftop solar to disappear.

Particularly, if storage tech does fall as this article predicts, I don't see why it wouldn't just replace peaker plants directly.

<comment><USER4> > Particularly, if storage tech does fall as this article predicts, I don't > see why it wouldn't just replace peaker plants directly.

This is the first target market of stationary utility scale storage (MW class), as current peakers are usually inefficient natural gas combustion facilities that only run ~100 hours/year (versus an efficient, more expensive combined cycle generator).</comment></comment></comment>


Taxing Short, Cheap Flights Makes a Lot of Sense - <AUTHOR> https://www.bloomberg.com/opinion/articles/2019-08-31/taxing-short-cheap-flights-will-cut-carbon-emissions <comment><USER2> Why would one levy a tax on flights if one's goal is make travel more efficient? Why not simply tax transportation fuel based on energy or carbon content? The latter would incentivize industry to develop more efficient modes of transportation.

<comment><USER3> The article mentions that taxing fuel was found to be legally problematic.

One assumes the laws preventing taxing fuels were lobbied for by oil companies.

<comment><USER4> I looked at the linked report and it seemed to just assume that, did not even make it explicit, and definitely didn't explain it :(. I then searched for airline fuel tax information on Google and found a page from the FAA about existing fuel taxes and some information on how the funds are restricted, so I am left not understand why this is not an option.

[https://www.faa.gov/airports/airport_compliance/aviation_fue...](https://www.faa.gov/airports/airport_compliance/aviation_fuel_tax/)

</comment></comment></comment><comment><USER5> The basic issue in Europe is that train and bus travel is insanely expensive relative to air travel. That's why Ryanairs of the world even exist and are able to offer fares comparable to bus and train.

If governments are serious about making travel less environmentally stressful, they should consider underwriting rail infrastructure investment and just have the companies recoup operational expenses from fares.

<comment><USER6> Even more expensive full-service carriers are sometimes less expensive and rarely more expensive than (in particular) high-speed train services. To make intercity trains worth it I typically have to buy the ticket well in advance when cheaper fares are available and I assume the flight prices won't decrease significantly due to their unpredictability.

</comment></comment><comment><USER7> Yes. It keeps the unwashed masses from traveling, and is no problem at all for the rich.

How about we charge flights based on net worth, so they're proportionally expensive?

One way for this is for each flight to contribute X% tax points and increase your income and property taxes...

<comment> GuB-42 I have no problem with letting the rich fly as long as they pay. Just use that money to let the poor access eco-friendly transportation more easily.

Here is the problem: a short flight may cost $50, the bus is $40 for the same trip, it is less desirable but much greener. In that situation, people with a $50+ budget will take the plane, people with a $50-$40 budget will take the bus and the others will stay at home. Now bump the plane ticket to $100 and use the money to subsidize the bus down to $30. The result may inconvenience the $50-$100 class as they are now taking the bus but it will open travel for the $30-$40 class, and help the planet at the same time.

It is even better when we take high speed trains into account. The train is often faster and more comfortable overall than the plane on short distances, and cleaner. But it is also often more expensive. Bumping the price of plane tickets and use a bit of that money to help the train will put things back the way they should.

<comment><USER8> The flight is only cheaper because of the massive subsidies the airlines receive.

<comment><USER9> What subsidies do airlines receive?

<comment><USER8> Linked is a starter article with a detailed example:

Custormer pays €81.65 State pays €86.24 in subsidies, €43 in state debt and a trade war

and then we didn't even begin to price in the negative externalities, where airports and planes cause not just huge environmental but also severe health impacts, and the economic value destruction of land near airfields.

[https://www.transportenvironment.org/news/cheap-airline- tick...](https://www.transportenvironment.org/news/cheap-airline-ticket- doesn%E2%80%99t-fall-sky)

</comment></comment></comment></comment></comment><comment><USER10> Doing this in the US will lower the amount of flights to rural areas

<comment><USER11> Those flights are already very heavily subsidized, I doubt they wouldn't provide even more subsidy if this was done

</comment></comment><comment><USER12> And while we're at it, why not have a nation wide $0.25/gal tax on auto gas that directly cross subsidizes buyers of electric cars? Kill two birds with one stone.

</comment><comment><USER13> Short haul flights are the ones which are most likely to electrify first, helping fund the technology for further ranges. Taxing the fuel makes sense.

<comment><USER14> There's already an electrified mode of transport for these short trips: the train.

<comment><USER13> $20.4B and rising... for connecting just two metros in California.

<comment><USER11> What is the cost of equivalent capacity construction on I-5, at SFO and at LAX?

In some markets (NE Corridor) airspace is so busy that you can't even really slot too many more planes in to meet travel demand.

</comment></comment></comment></comment><comment><USER15> Trying to engineer society doesn't make any sense, much less by levying discriminatory taxes.

<comment><USER6> Where is the divide for you between governing and engineering society? I assume this answer varies among people as it frames the same situation in two differently opinionated ways.</comment></comment>


Grand Prize Awarded To BellKor Pragmatic Chaos - <AUTHOR> http://www.netflixprize.com//community/viewtopic.php?id=1537

<comment><USER2> Papers here:

[http://www.netflixprize.com/assets/GrandPrize2009_BPC_BellKo...](http://www.netflixprize.com/assets/GrandPrize2009_BPC_BellKor.pdf)

[http://www.netflixprize.com/assets/GrandPrize2009_BPC_BigCha...](http://www.netflixprize.com/assets/GrandPrize2009_BPC_BigChaos.pdf)

[http://www.netflixprize.com/assets/GrandPrize2009_BPC_Pragma...](http://www.netflixprize.com/assets/GrandPrize2009_BPC_PragmaticTheory.pdf)

</comment><comment><USER3> Did they figure out the Napoleon Dynamite conundrum?

<comment><AUTHOR> I did a search on Google, but couldn't find the reference. Care to share?

<comment><USER4> From what I remember reading about the contest, the problem was that when comparing movies and using movies that were similar to others as a way to suggest movies people either hated or loved Napoleon Dynamite and it didn't match up at all with their previous movie ratings. Thats how I understood it at a basic level.

</comment></comment></comment><comment><USER5> I'll repost from another thread about the NYTime article, <http://news.ycombinator.com/item?id=834681> : The next contest info surprised me:

"The data set of more than 100 million entries will include information about renters' ages, gender, ZIP codes..." It's enough to identify 87% of the people, apparently: [http://www.freedom-to-tinker.com/blog/paul/netflixs- impendin...](http://www.freedom-to-tinker.com/blog/paul/netflixs-impending- still-avoidable-multi-million-dollar-privacy-blunder)

I hadn't realized that someone identified some of the raters in the previous contest with their imdb ratings: <http://www.cs.utexas.edu/~shmat/netflix- faq.html>

Also, why this matters, page 44 of a research paper (PDF): <http://papers.ssrn.com/sol3/papers.cfm?abstract_id=1450006>

</comment><comment><USER6> so why didn't the ensemble win? in the rules it says, 'At the end of this period, qualifying submissions will be judged (see Judging below) in order of the largest improvement over the qualifying RMSE on the test subset.' this isn't the icfp. i assume the person on the leaderboard is in fact the leader.

the difference between the 'test' and 'quiz' sounds b.s. to me. at this rate, i know i won't even be contemplating netflix prize 2. at the very least, netflix owes it to the community to explain why they (at this point, seemingly corruptly) made the decision they did.

i suppose they say they'll post the final 'test subset' scores on the leaderboard. it doesn't appear to be a very 'open' contest if they don't actually publish exactly what the test subset is and how the rankings are determined. when it's that close, everything should be shown to be exactly done within the rules. otherwise, they really risk delegitimizing the whole contest, imho.

<comment><USER7> I'm not familiar with the details in any way, but it sounds like you could probably produce an algorithm that produces a perfect result for the rest if you give it a subset of the publicly available data. That doesn't mean your algorithm is good in general, that just means you essentially encoded the test data set into your algorithm. The real test is always going to be how well you do on data that you've never seen.

As for why the intermittent leaderboard shows results based on the public dataset: using a specially crafted algorithm, you could leak information about hidden data through any feedback you get from the test environment. Such information could be used to create an aforementioned algorithm that does well in this specific case.

see also: <http://en.wikipedia.org/wiki/Overfitting>

<comment><USER6> i see. but if they don't make their private dataset public, what's to prevent netflix from fitting it to a particular entry?

<comment><USER8> Why would Netflix corrupt their own competition in this way? I don't understand what you're concerned about.

<comment><USER6> maybe because 'bellkor's pragmatic chaos' won the progress prize and they have some special relationship with them. could be any number of reasons. this is i think the danger of a private corporation running a programming contest.

i'm not saying netflix is unfair or corrupt. i hope not. but i think they owe it to the contestants to explain exactly why they chose the #2 entry on the leaderboard over the #1 entry. and not just (though i appreciate people's comments) because of overfitting. how is the private 'test' subset chosen? i have no affiliation with this contest or any of the teams in any way, but how would you like to work on this for years only to be told that even though you beat everyone in the quiz, in the private 'test', you lost.

<comment><USER9> Did you read the contest rules and FAQ? (<http://www.netflixprize.com/rules>, <http://www.netflixprize.com/faq> \-- and if you didn't, what are you doing throwing around words like "corrupt"?) It looks to me like they're being very transparent and explained things very clearly. You have a bunch of data in the test set; these are partitioned randomly into two equally-sized subsets, "quiz" and "test". You submit predictions for both subsets, but are only told how you did on "quiz". Netflix provides an MD5 checksum for the judging file that defines what the partition is; this file will be made available "at the end of the Contest". So this will be verifiable by anyone soon.

Also, don't brush overfitting aside. In the (paraphrased) words of one machine learning researcher, "life is a battle against entropy. In the same way, machine learning research is a battle against overfitting." Any data whose test results you use to select your algorithm or to adjust your algorithm's parameters is not properly considered part of the test set; after your optimization that data will provide an optimistically-biased estimate of your true error. Since competitors could get regular updates on their performance on the quiz dataset, one must assume that they were attempting to optimize this performance, and so quiz set performance was not be a good estimate of their true error. You can only get a good estimate of the true error of a method by testing against data that has played no part in its development.

<comment><USER6> > So this will be verifiable by anyone soon

Yeah....what happens when the checksums don't check out?

</comment></comment></comment></comment></comment></comment></comment><comment><USER10> This is great and all, but it still thinks I'll love Once: <http://www.imdb.com/title/tt0907657/></comment>