How to properly use UTF-8 in Perl - <AUTHOR> http://stackoverflow.com/questions/6162484/why-does-modern-perl-avoid-utf-8-by-default/#6163129

<comment><USER2> This is way too much to remember.

The key to handling Unicode correctly is to think about the data that comes into your program and the data that leaves your program. If it's text, it has some sort of encoding, and you need to decode those encoded characters before you can use them as text in your program. Similarly, you can't just leak Perl's internal representation of characters to the world: you have to explicitly encode the characters to octets.

The problem that people run into is not knowing that you have to do this (in every programming language), and then deciding which encoding to use. In some cases, it's easy: HTTP includes the encoding in the headers, so all you have to do is get the request, decode according to the headers, and you're set. Similar, when you send an HTTP response to someone, just set the headers, encode the characters, and everything ends up 100% correct.

The problem comes when poor design causes you to assume the character encoding. What encoding are file names on a random removable disk? What encoding is this text file in? What encoding are those database rows? If you don't know, you simply can't process that data correctly as text. But most people assume "something will magically decide for me and it will all work out". Nope, it won't. Don't rely on magic: be explicit.

The reason why things work most of the time is because you treat the data as binary: opaque, meaningless octets, that don't have semantics like "make the first letter capital". This will work if your world is entirely UTF-8-encoded: UTF-8 filenames, UTF-8 files, UTF-8 source code, UTF-8 database results, etc.

The reason why people run into trouble with Perl and not with other languages, is because Perl assumes that when you treat binary data as text, you really have Latin-1 text. It then says, "hey, this is text", upgrades it to Unicode, and then transforms it as such. When your text was Latin-1 (the backcompat case), it all works. When it's UTF-8, though, then you get double encoding: the dreaded "æ¥æ¬èª" instead of "日本語". But if you just tell perl, via Encode::decode_utf8, it will know that you actually have unicode text and not latin-1 text, and everything will work!

Anyway, the solution is to always specify the encoding via Encode::decode_utf8($str) or Encode::decode('my-encoding', $str) when reading data, and to Encode::encode_utf8 or Encode::encode when outputting data (even to the terminal!). The hardest part is making sure your libraries do this (DBD::SQLite does, Catalyst does, LWP does), and making sure that your libraries have enough information to do it right. HTTP is easy, there's a header. But that blob in your database is not easy, and you may have to handle it yourself... because the information about the encoding exists only in your brain, not anywhere the computer can find it.

(Oh, and this only gets you to the "I'm not trashing any information" stage. If you want your Japanese text to sort あ い う え お instead of in codepoint order, that's going to require a module. Even simpler cases will require you to learn about collation and normalization.)

Edit: distilled the key points I raise here into an SO answer: [http://stackoverflow.com/questions/6162484/why-does- modern-p...](http://stackoverflow.com/questions/6162484/why-does-modern-perl- avoid-utf-8-by-default/6192088#6192088)

</comment><comment><USER3> See "The Absolute Minimum Every Software Developer Absolutely, Positively Must Know About Unicode and Character Sets (No Excuses!)"

<http://www.joelonsoftware.com/articles/Unicode.html>

</comment><comment><USER4> I think I'll hold out for the ... simple way. There should be more than one way to do it, right?

<comment><USER5> Indeed... Simple, and probably wrong.

I won't pretend that Perl's unicode support sucks; Working with Unicode in Perl is far more complicated than it ought to be and rife with bugs and corner-cases. However, working with Unicode _properly_ in just about _any_ language requires a whole lot of careful thought and consideration.

Maybe Java (and by extension JVM languages) is the best you can get right now, I dunno...

<comment><USER5> whoops, I meant that it _does_ suck, and I won't pretend otherwise.

</comment></comment></comment><comment><USER6> After reading this lengthy how-to, I am tempted to say that PHP _may_ have upper hand when it comes to _reliably_ dealing with UTF-encoded data from different sources (including source code) and sinks. At least as long as I default to UTF-8 and convert only when explicitly requested by user or remote system to do so.

PHP has no separate Unicode type, the built-in string type works pretty well for UTF-8. I'd liken its use to the concept of duck-typing (as seen, for example, in Python): if it walks like a duck^W^W UTF-8, swims like a duck^W^W UTF-8, quacks like a duck^W^W UTF-8, it can be handled (consumed and produced) like a duck^W^W UTF-8.

Note in Python Unicode is a separate data type, so the above comparison is related to duck-typing only.

<comment><USER7> It's not pretty in PHP either. Many of the Perl issues have their PHP counterparts.

• You need to be aware that most built-in string functions won't work and use mb_ equivalents where available (and not all are available).

• You need to set internal encoding to UTF-8 for some extensions in php.ini.

• You need to normalize all input yourself (thankfully PHP5.3 has Normalizer class. In 5.2 it was a nightmare)

• PCRE library doesn't support all uncode ranges (e.g. {InFoo} is unsupported). All gotchas of regexes apply to PHP/PCRE as well.

The list goes on…

"Binary-safe" strings and UTF-8 cleverness will let you roundtrip characters safely in most cases, but full, correct Unicode support is really hard.

<comment><USER8> > You need to be aware that most built-in string functions won't work

Actually the non-mb string functions will work perfectly fine for almost everything. For example splitting on space or comma, or joining strings will work just fine with UTF-8. The only things that don't work are those that split by character position. Even search works fine.

> You need to set internal encoding to UTF-8 for some extensions in php.ini.

      ini_set('default_charset', 'UTF-8');
      mb_internal_encoding("UTF-8");

> You need to normalize all input yourself

Only if you need to compare against it (for example to see if the username was taken). Most of the time you just store the input as is.

For PCRE make sure to add the /u option.

<comment><USER6> I'd just stick to mb_ereg_...() functions. They follow the standard setlocale(LC_...). They support the standard POSIX regex syntax. They support Unicode always when the platform does (depends only on standard C library) -- and I've seen PHP's PCRE distributed _without_ Unicode support. Last but not least, PCRE suffers from <http://swtch.com/~rsc/regexp/regexp1.html>

      setlocale(LC_ALL, 'pl_PL.utf-8'); # if not set explicitly defaults to system's $LC_ALL, thus fits most of the time
      mb_internal_encoding('UTF-8');

      mb_ereg_match('[[:alpha:]]', 'Ł') -> true
      mb_ereg_match('[[:alpha:]]', '♙') -> false # that's a white chess pawn character, U+2659.
      mb_ereg_match('[[:graph:]]', '♙') -> true

<comment><USER8> I don't like setlocale because usually I'm writing a general application, usable for any language.

<comment><USER6> I like setlocale() as a central knob that other functions obey. Works a-OK for me on several different websites. Normally my website's code starts with platform defaults, reads configuration early on, and, if need be, issues setlocale(). Should an error be triggered before the config is applied, it would be handled with default locales; I believe that's the only noticeable downside. Hopefully a rare one ;-)

setlocale() can be switched at run-time, even multiple times if you want. You could imagine running most of the code with user's preferred locale to output UI or report any errors, switching to another locale to format and send message to a foreign customer, and then back to normal settings to continue with outputing UI, all in one pass of PHP.</comment></comment></comment></comment></comment></comment>


Creator of Web spots a flaw in Internet Explorer - <AUTHOR> http://www.msnbc.msn.com/id/26646919#.UE32s6T9Fv0

<comment><AUTHOR> Old article, but worth reading.</comment>


How may times links shared on soc networking - <AUTHOR> http://sharedcount.com

<comment><USER2> This is pretty cool. It would be absolutely amazing if I could see the shared count for my _site_ , not just for a _url_. In fact, I could probably submit a list of URLs through the API to make things easier for you (although with the way you're looking up the shares, I don't know if you'd end up hitting their API limits trying to look up 1,000 URLs for a site). But that's something I would find tremendously useful - just being able to see the shares on one particular URL isn't that useful to me. Nonetheless, cool :)

</comment><comment><USER3> This is pretty cool, I always pretty much ignored facebook and used twitter search for twitter

so how do I find out where the comments are for my link? [http://sharedcount.com/?url=http%3A%2F%2Farandomurl.com%2F20...](http://sharedcount.com/?url=http%3A%2F%2Farandomurl.com%2F2010%2F07%2F25%2Fhtml5-pacman.html)

</comment><comment><USER4> Every single URL: "This is not a valid URL. Please try again."

Also, don't clear the input box each time I focus it. I might simply want to correct a typo.

Other than that, it looks interesting.

<comment><USER5> Worked for me with with full address - <http://www.etc.com>

I did have to play around a bit to avoid those errors however

</comment></comment><comment><USER6> Very nice !

I'd pay for this applied to a domain with a date filter :-)

<comment><USER7> You should check out BackType then. They offer analytics like this and much, much more. Check it out <http://www.backtype.com>

<comment><USER6> This looks great ! Thanks</comment></comment></comment>


Ask HN: Is AMP Dead? - <AUTHOR> Just from checking the Google News or the discover feature on Android it seems like lots of publishers are included without an amp version of their websites. Which in the beginning was not the case.<p>I know that as a web standard it received a lot of criticism from HN and other communities. But is there any major benefit of basically maintaining two templates for websites? Could Google drop it at some point in the future? <comment><USER2> It surely will and it should.

Here is link my friend texted me about a news: `[https://www.google.co.uk/amp/s/www.nytimes.com/2019/10/12/sp...](https://www.google.co.uk/amp/s/www.nytimes.com/2019/10/12/sports/eliud- kipchoge-marathon-record.amp.html`)

Notice, she was sending me a link of NYTimes.com not Google.com. AMP has highjacked the content. I did fair bit of research and learning on AMP before deciding not to offer it to my consulting clients and also made company wide policy to refuse the service.

Not just web, AMP, and Google for that matter is also trying hard to push into Email domain. Aside from monopoly - this technology brings intrusive mechanism for many unintended parties.

</comment><comment><USER3> As a web developer and content publisher, i don't see much value for AMP pages, Why would someone put additional effort to create an AMP version of the same content, most of the web pages are very well compatible mobile device, developers are already spending time to make those pages responsive, and most importantly when we have already schema.org tags why do we need again and additional AMP version ? That's too much, not much worthy, so i think AMP will be dead soon

<comment><USER4> There is really only one reason to implement AMP and that is the bump in search results. It's practically extortion by Google.

</comment></comment><comment><USER5> I believe it offers speed improvements, and Google bumps you up a bit in the search results if you enable it on your website. Honestly, I hope it goes away - It makes sharing links on mobile a pain. The speed is nice, but usability always wins, so we'll see.

</comment><comment><USER6> I hope so. It was a transparent attempts to make parts of the web Google proprietary.</comment>


Should Everyone be a Programmer? - <AUTHOR> http://www.philterdesign.com/?p=364

<comment><USER2> At one point in history it was asked if everyone should learn to read and write so that they could more easily communicate with people from far away for the purpose of solving problems with collective action instead of as individuals or small groups.

The answer to that question was yes. The incorrect answer was. "Reading and writing is too hard, the scholars in the temple will do the reading and writing on behalf of the sheep"

Today the question is should the sheep learn to program the computers? The answer is no, it's too hard, we will do it on their behalf. Imagine a nation full of capable coders. Woah. Im glad we made reading and writing mandatory in school.

<comment><AUTHOR> Interesting analogy.

But couldn't that be said of any profession? Everyone should learn to apply medical procedures or represent themselves in a court of law? Ya it would be awesome to have a nation full of great doctors. The problem is that each of these things is a specialization that takes thousands of hours of study and experience to do well.

Yes, the barrier of entry to programming is much lower, but that doesn't mean it doesn't require the same amount of time to be a good coder as it does to be good in other professions.</comment></comment>


Debunking a Computer Chess Scandal - <AUTHOR> http://www.chessbase.com/newsdetail.asp?newsid=7791 <comment><AUTHOR> Simply on the basis of telling a fascinating story, the series is well worth reading. Beyond that, some of what particularly struck me was that:

\- since about 2005, chess programs that run on a regular PC have consistently outperformed the best human grandmasters, and the skill of the computer chess programs continues to improve. 2005 was a quantum shift in the chess programming world, when chess programs first played "brilliant games with deep, beautiful combinations...[and] routinely produce[d] highly artistic masterpieces of chess while avoiding a great many pointless "computer" moves that for many years had been a source of ridicule among strong human players."

\- improvement in algorithms is driven by an active community ecosystem, open source contributions and flagrant copying of the brilliant insights of a few. No matter how brilliant the insight, the community is able to improve upon it. Intellectual property protection would have stopped much of this improvement in its tracks.

It's four separate articles, so I posted the first. All four are:

<http://www.chessbase.com/newsdetail.asp?newsid=7791> (part 1)

<http://www.chessbase.com/newsdetail.asp?newsid=7807> (part 2)

<http://www.chessbase.com/newsdetail.asp?newsid=7811> (part 3)

<http://www.chessbase.com/newsdetail.asp?newsid=7813> (part 4)

<comment><USER2> A quote from Rajlich:

> Yes, the publication of Fruit 2.1 was huge. Look at how many engines took a > massive jump in its wake: Rybka, HIARCS, Fritz, Zappa, Spike, List, and so > on. I went through the Fruit 2.1 source code forwards and backwards and took > many things.

It's not as if he never admitted to studying Fruit.

Also, most of the confusion is there because Rajlich didn't have a repo until Rybka 4, so when the reverse engineered his code they formatted it the same way as Fruit, making it look very similar. If a judge missed the bit saying that the Rybka code is "functionally equivalent code", not the actual code, they might be fooled into thinking the orignal codebases looked very similar. The guy doing this wrote something like 30 posts a day on the discussion board about Rybka, and was clearly not a fan.

They should have made a 3-way comparison - Rybka's PST, Fruit's PST, and some independent program's PST (in the same format as the other two).

</comment></comment><comment><USER3> OT: why do websites find it useful to disable the up/down arrow keys? It is not. It's very annoying. Make navigation a pain in the butt. Make people never want to come back to your site.

<comment><USER4> Maybe it's the fault of the game-history widgets on the page. They use left and right arrow keys for game navigation, so it's possible they steal up and down arrows also.

</comment></comment><comment><USER5> Isn't there something slightly ironic about accusing the best pupil in the class of peeking over the shoulders of the other students during tests? Not that it can't be a correct accusation, but if this guy "cruised to victory in four consecutive WCCC tournaments in 2007, 2008, 2009 and 2010" then he must have done something right (other than copy-paste).

From a copyright perspective I can certainly see an issue, but that would be between Rajlich and the author of this Fruit I would say.

<comment><USER6> Actually, the argument isn't about cut&paste programming. That would be somewhat easy to determine by simply comparing the binary output (after some searching for the right compiler options from the compiler for that era).

Indeed, the accusations clearly say that there were significant changes to the code, which makes a reverse engineered comparison harder to do.

Instead, the complaint is that the underlying algorithms were re-implemented, instead of being original algorithms.

</comment></comment><comment><USER7> For what it's worth, the WCCC's Rule 2 seems to be a lot like a university honor code (if you do borrow code or ideas, make sure you cite it.) I don't believe that's too hard to follow. Maybe someone should fund a kaggle competition to design the best chess-playing program, with the final requirement of making the code open-source after :)

Given that most of what you'd implement a computer chess engine is pretty much available online (at most,behind a paywall), and digging through the source code of GNUChess and Crafty would give a lot of insight into the scoring function, so, yes, the score function would be pretty much similar in most parts unless someone comes up with a radically new way of doing things.

</comment><comment><USER8> From the article, "By definition, plagiarism only happens when credit to sources is not given, which was never the case with Rybka."

I simply do not agree with this. I cannot copy slabs of open source code, incorporate them into my product and pass if off as my own just because I somewhere credited the great performance of my program to stuff I learned from reading open source software.

To me, plagiarism is copying some portion of someone else's work and including it in a work which I (explicitly or implicitly) claim to be my own.

For example, a student handing in a wikipedia article for a homework assignment still commits plagiarism even if they credit wikipedia as a _source_ but don't explicitly say the thing is a wikipedia article. They hand it in as their work when they did not write it. So at this point I do not feel I agree with the article. It's as though the argument is, yes he copied, but it's ok because he gave "credit", there were plenty of new ideas and everyone else was doing it too. That's not the same thing as, "he didn't commit plagiarism". The latter means he did not copy the code.

Having said that, I am absolutely gobsmacked that the committee did not simply ask for the source code to both programs, check that it in fact compiles to the binaries in question, then do a comparison for legally significant quantities of identical code.

It's even more remarkable when you realise that the version of Fruit involved was open source, so they didn't even need to ask for the source code to that!

Note: I edited the above in response to the comments below.

<comment><USER9> I don't have an opinion on what this guy did and whether he should have been stripped, but you're redefining plagiarism pure and simple.

Plagiarism is passing off someone else's ideas or expression of those ideas as if they were your own. And that's it. Plagiarism has _nothing to do_ with whether you _benefit_ from someone else's ideas illegally or unethically.

Let's say that there's a competition to write an original novel. I take Moby Dick and change one sentence. Then I submit it with a big warning on the front cover that says: "This novel is Moby Dick with one sentence changed by me." I have violated the rules of the contest perhaps. If I win I maybe benefitted from it. But I have not committed plagiarism -- I did not pass off these ideas as if they were my own. It was made clear whose ideas they were.

</comment></comment><comment><USER10> Good article. I agree that to slam someone's reputation in public, the ICGA should have had much better evidence. My understanding is that the release of Fruit source code inspired many chess programmers.

</comment><comment><USER11> For UK/Hungary computer nerds, Dr David Levy was also responsible for the Enterprise 64 computer: <https://en.wikipedia.org/wiki/Enterprise_%28computer%29>

</comment><comment><USER12> Although the articles seem to not mention it, it should probably be noted the author is a moderator on the Rybka forums so is most likely not coming from an exactly unbiased viewpoint.

<comment><AUTHOR> Though buried as an aside in the fourth part of the piece, the author does mention that he is the moderator on the Rybka forums.

"In my capacity as Rybka forum moderator I have access to posting statistics."

Indeed, his moderator status leads to one of his core debunking arguments. Some Dr. Hyatt guy, apparently a key individual behind the accusations against Rybka, appeared to be making the attack a full-time job, shifting the arguments as needed to defend a guilty conclusion.

Though he doesn't say it, I suspect that noticing this was impetus for the author to seriously tackle a look at the charges from the defense perspective.

Not that there is anything wrong with either side picking a conclusion (guilty, not guilty) and then working hard to support it. However, part of the credence given the initial charge is that it resulted from an independent search for truth. This does not appear to have been the case. Instead, the charges were the prosecution half of an adversarial process. In these articles, now we see the defense.

Caveat: all I know about this matter is what I read in the four part article posted here.

</comment></comment><comment><USER13> A fine example of terrible writing.

</comment><comment><USER14> It has been discussed (in chess circles) in excruciating and unambiguous detail how Rajlich blatantly ripped off (Google it) open source code, thereby violating tournament rules, open source licenses, etc. This "debunking" series of articles by Chessbase is a pathetic attempt to reinvent the past on behalf of one of their most lucrative products, and perhaps to fool a wider audience who didn't follow the original scandal when it happened.

I used to be a big fan of Rybka, Rajlich and Chessbase, but these continued denials just make a bad thing worse. It's like Floyd Landis, Tyler Hamilton, and the third guy.

<comment><USER15> As far as I can tell, the basic facts are:

1\. There is damning evidence (probably beyond reasonable doubt) that Rybka contains some copy-pasted code from Fruit (the open source program).

2\. The interesting thing about Rybka was that it was dramatically better than any chess engine of its time, due to original improvements.

3\. Many people are bitter that Rybka became an extremely successful commercial product, due to being clearly the best chess engine of its time. It also made several formerly viable commercial products non-viable due to being much better.

4\. A mass of clones obtained by either disassembly or obtaining source code appeared, which are now equal or slightly superior to Rybka, and this has essentially destroyed its commercial viability.

For people who value the letter of the law, #1 is damning, and Rybka cheated, period. This isn't beyond theoretical doubt, but it's beyond a reasonable one.

But, given that Rajlich could have easily and legally re-implemented the same algorithms in the mudane parts of Fruit that are in question without exact code reuse, and that what is significant about Rybka is actually the original and innovative part, there's a pretty good argument that he's well within his rights to profit from his creation.

<comment><USER16> Which did you see as the conclusive evidence that code was copy-pasted? Section 4 of the article made the point that the Rybka source was missing and the accusers' evidence was decompiled code which they had arranged to look as similar as possible.

<comment><USER16> Ed Schröder, the initial accuser who recanted, states in summary:

"4. It is not possible to state categorically from the available data whether:

" 4a. Vas took Fruit source via cut and paste and converted to Rybka;

" 4b. Vas kept a copy of Fruit source open on another screen while he wrote Rybka himself;

" 4c. Vas absorbed ideas from Fruit (and other open sources) and then coded up Rybka hmself."

Source: <http://www.top-5000.nl/evidence.htm> — worth reading, but just as open to accusations of bias as the other links we've seen.</comment></comment></comment></comment>


Ask HN: Have you ever bought software before trying it lately? - <AUTHOR>

My startupt think we have few great ideas how to make accounting software for small business easier. But it would take months to build, and we don't want to risk building it, and then nobody buys it. We are thinking about a Kickstarter-thing, where we ask people to prebuy it, based on high-quality screenshots that we plan to make.
My concern is that small businesses with 0-4 employees (our customer group) in 2013 demand to try the software (not only look at screenshots), before they buy it.
What do you think? Have you bought any software before trying it lately?
Upvote the comments "Yes" and "No", if you want to answer. Only way I could get a poll going. <comment><USER2> No way I would pay for such a critical part of the business without trying it and only see some "supposedly" pre rendered screenshots that will also just set an high level of expectations. I am sorry but I really think this is not going to work.

</comment><comment><USER3> You should have added a poll. My answer is No. I need to try first.

EDIT: I should note my reply is a general No. I am not your accounting market at all.

<comment><AUTHOR> I need 200 karma to do polls. But I've added a yes and no answer, so people can upvote the comment, and use that as a poll.

</comment></comment><comment><AUTHOR> No

</comment><comment><AUTHOR> Yes</comment>


Ask HN: How good is the Bootstrapping Design book? - <AUTHOR>

I saw a posting on the Bootstrapping Design book(http://www.bootstrappingdesign.com/) few weeks on ago on HN and subscribed to their mail list. Now that the book is ready and its beta version is available for purchase for $39, I would like to know if it is worth the money. Also if you can recommend other books on design that is worth looking into. I am not interested in mastering design per se, but at-least develop a minimal level of understanding to help me interact with designers. <comment><USER2> This book seems to go on the same principle of bootstrapping design (maybe a little more in depth?, check the sample chapters as well): <http://designforhackers.com/>

<comment><USER3> The book itself is decent enough but I think it's way too short to justify the price tag.

</comment></comment><comment><USER4> I'm the author. Sorry I'm late in seeing this! If you happen to check back, feel free to email me any questions at email@example.com.

I can also point you to reviews and feedback from customers.</comment>


Facebook can track your browsing even after you've logged out, judge says - <AUTHOR> https://www.theguardian.com/technology/2017/jul/03/facebook-track-browsing-history-california-lawsuit <comment><USER2> > _Judge dismisses lawsuit accusing Facebook of tracking users' activity, > saying responsibility was on plaintiffs to keep browsing history private_

Then there can be no argument against people using ad/tracking script blockers.</comment>


What Happens When an Airplane Hits a Deer - <AUTHOR> https://www.theatlantic.com/technology/archive/2017/02/what-happens-when-an-airplane-hits-a-deer/516951/?single_page=true <comment><USER2> nice. if only these stats were given as much press as anything including the word "drone"</comment>


Thoof "is an attempt to improve on sites like Digg and Reddit" - <AUTHOR> http://www.nytimes.com/2007/06/18/business/media/18thoof.html?ex=1339819200&en=99bf1ddbadf72930&ei=5088&partner=rssnyt&emc=rss <comment><USER2> If this guy thinks the fact that so many Americans believe Saddam Hussein was involved in 9/11 is evidence that a new news site is needed, he's off to a bad start. They believe that because they want to, not because there are no reliable news sources available.

<comment><USER3> The fundamental problem is real though. News sites, including social news like reddit, are riddled with errors. Finding the truth often requires a real effort on the part of the reader, and for the most part I'm just too lazy/busy.

<comment><USER4> There is a traditional solution to this, in theory. The theory goes that a newspaper will meticulously edit and fact check, because at some point, they are bound to write about something I happen to know a lot about, and then if they are wrong, I will no longer trust them on other things. Unfortunately, the theory works a little too well: virtually all newspapers, if you read a few issues cover to cover, contain enough errors about things that you happen to know a lot about that you start mistrusting them. But there are a few publications for which traditional solution seems to work. I have rarely found any mistakes in The Economist, for instance, when they write about things I happen to know a lot about, which leads me to trust them more than most other news sources when it comes to things I don't know.

<comment><USER3> It's funny that you mention the economist, because I used to feel the same way about them, but then a few years ago they published some article about Google that was full of errors.

Clearly the old model isn't working though. I think the solution must be collaborative. I don't know if Thoof is it, but at least it's in the right direction.

<comment><USER2> I think anyone who is a general-purpose journalist rather than a domain expert is going to have this problem. Journalists don't actually understand the stuff they write about; no one could, probably; so instead they have a lot of hacks for sounding like they know what they're talking about.

Come to think of it, though, domain experts are often wrong too. Think about programming books, for example. I've now learned enough about several other fields to be fairly sure this problem is universal, except possibly in math. (Though even mathematicians have often been wrong in the sense of overlooking something that to later generations will seem obvious.)

Basically, you have to be a very critical reader. I doubt there's a "solution" beyond that. This is something the very smartest people struggle all their lives to get better at-- to take in all the stuff they're told, and from that figure out what's really going on.

<comment><USER5> Sometimes the problem has nothing to do with lack of domain knowledge. For example, before I went to MIT I saw a NYT article about students protesting the proposed removal of the pass/no credit system for the following freshman year. The article slanted it as if the students were selfishly interested in not getting grades. It doesn't take a domain expert to realize that if you're an MIT student now, you won't be a freshman next year, and could not be affected by the removal of pass/no credit. Students were objecting because they felt it would have a detrimental effect on others.

</comment></comment></comment></comment></comment></comment><comment><USER6> Thoof (were all the names in the World taken or what?!) was started by this guy: <http://news.ycombinator.com/item?id=21863>

Anyway, I looked at Thoof when it was posted on TC and I got an invite. It's a waste of time. It just requires way too much work from the regular user to be worthwhile and viable. Only redeeming quality was the endless scroll :).

</comment><comment><USER7> Ian Clarke is such an arrogant guy. Back in the day when he was nothing but a developer, he was arrogant as hell, I can only imagine how he is now.

And he's arrogant, and produces nothing but commercial flops. He's a typical close minded geek type, he thinks he knows everything, yet is unable to see the big picture.

Moral of the story: It's not difficult to be polite. There is no excuse for rudeness, but it will come back to bite your ass, even 8 years later!

</comment><comment><USER8> Btw oknotizie ( <http://oknotizie.alice.it> ), a reddit alike system my startup developed for the Italian market, already has the ability for users called "power users" (having enough total rank and enough rank accumulated in the last month) to replace the URL with one better suited (for example not linkjacked).

</comment><comment><USER9> So how does a recommendation algorithm get around the fact that interesting doesn't equate to accurate? Oh wait it doesn't. Although I do enjoy the irony of the first publicity for this site that will find the TRUTH is complete bullshit.

Also the implication "we could win netflix we just don't want to" is not very credible.

</comment><comment><USER10> Headline: The value of paying big $$$ for PR: Yet-another-Digg-copycat gets NYTimes mention.

<comment><USER2> It seems more likely to me that the origin of this article is explained in the last sentence.

<comment><USER11> Indeed. Very funny that _it's the last sentence_. Isn't that kind of important information that belongs at the top of the article?

</comment></comment></comment><comment><USER12> There are maybe one or two genuinely thought provoking articles on the entire web each week. What does it matter how the rest of them are displayed?

</comment><comment><USER13> the fact that this system uses bayes inference means it is doomed for failure. bayes requires large amount of training data. Most users want instant satisfaction. That could only mean failure.</comment>


Go, Marvin Minsky, and the Chasm That AI Hasn't yet Crossed - <AUTHOR> https://medium.com/backchannel/has-deepmind-really-passed-go-adc85e256bec <comment><USER2> > AlphaGo isn't a pure neural net at all — it's a hybrid, melding deep > reinforcement learning with one of the foundational techniques of classical > AI — tree-search

Most board game computer players use some sort of tree search followed by evaluation at the leaves of the tree. What we discovered in the 70s is that you don't need to have human-level evaluation to win at chess; it is enough to count material and piece activity, plus some heuristics (pawn structure, king safety...); computers more than compensate this weakness with their superhuman tree exploration.

This approach never worked so well for Go because evaluation was a mystery: which group is weak or strong? how much territory will their power yield? These are questions that professionals answer intuitively according to their experience. With so many parts of the board that depend on each other, we don't know how to solve the equation.

It looks like AlphaGo is the first one to get this evaluation right. At the end of the game, his groups are still alive and they control more territory. So Go evaluation is yet another task that used to be reserved to human experts and that computers now master. The fact that this is mixed with classical tree search does not make it less impressive.

<comment><USER3> The author of this post (Gary Marcus) is a huge proponent of hybrid systems, in fact he is using that technique for his current stealth startup: [https://www.technologyreview.com/s/544606/can-this-man- make-...](https://www.technologyreview.com/s/544606/can-this-man-make-aimore- human)

<comment><USER4> Yep.

Not to be harsh, but Marcus has been critical of Neural Nets for a while now. His claims that there are issues around the provability of them are well made.

But.. there is a way to make people listen to you. It's called results. Deep Learning is getting them, in an increasing number of diverse fields.

<comment><USER5> _But.. there is a way to make people listen to you._

Hype, right?

Choosing problems for their theatrical effect, rather than utility. Writing articles and research papers as if they're marketing pamphlets. Claiming that incremental improvements are paradigm shifts. Treating arbitrary achievements as if they were commonly agreed upon milestones all along.

All of this is happening right now. Being skeptical in such environment is the only right thing to do.

 _Deep Learning is getting them, in an increasing number of diverse fields._

If you look for practical applications that give tangible benefits to people outside of academia, the achievement of applied deep learning so far aren't nearly as impressive as you make them out to be. This is despite insane levels of hype, huge investments in research and amounts of computing power available.

Heck, if anything, the fact that AlphaGo needs to use a tree search to prop up its ANN components could be seen as a sign that ANNs have some serious practical limitations when it comes to "results". Which is kind of the point of the article.

<comment><USER4> No doubt there is plenty of hype. From where I sit though, a lot of it is justified (Not the general intelligence stuff of course).

 _Choosing problems for their theatrical effect, rather than utility. Writing articles and research papers as if they 're marketing pamphlets. Claiming that incremental improvements are paradigm shifts. Treating arbitrary achievements as if they were commonly agreed upon milestones all along._

I'm not sure what to say to this.

There are no "commonly agreed upon milestones". The closest things are the academic benchmarks/shared tasks that you seem to be critical of.

I guess the closest thing you'll find to a "commonly agreed upon milestone" is something like the Winograd schema[1]? Based on progress like "Teaching Machines to Read and Comprehend" I wouldn't be betting against deep learning on that.

 _If you look for practical applications that give tangible benefits to people outside of academia, the achievement of applied deep learning so far aren 't nearly as impressive as you make them out to be._

Could you explain what you expecting? Deep learning techniques aren't exactly wide spread yet, and outside Google and a few other companies it takes time for things to migrate into products and have tangible benefits.

Nevertheless, Google Search, Pinterest, Facebook image tagging, Android Voice Search etc, etc.. these all are used by billions of people daily. I think it's hard to argue there isn't at least some practical applications.

[1] [https://en.wikipedia.org/wiki/Winograd_Schema_Challenge](https://en.wikipedia.org/wiki/Winograd_Schema_Challenge)

[2] [http://arxiv.org/abs/1506.03340](http://arxiv.org/abs/1506.03340)

</comment></comment></comment></comment></comment><comment><USER6> Interesting article! However, Deep Blue, Watson, and AlphaGo are very different from one another. I don't think anyone deemed beating humans at chess or jeopardy impossible at the time Deep Blue and Watson were built. On AlphaGo the point about generalizing AI is valid, yet I think the author doesn't fully appreciate the novelty of approach described in the AlphaGo paper. Their work advances the field, has more general utility than Deep Blue's chess or Watson's jeopardy programs. AlphaGo paper specifically represents an advance in machine learning algorithms for games in general. As I understand, Watson's new NLP algorithm is PRISMATIC [1]. PRISMATIC is a rule-based NLP system and AlphaGo is more statistical inference/neural networks. Even if AlphaGo's 'policy-network- value-network' framework is not too generalizable, the philosophical implication is that we can build AIs that can mimic 'human intuition'. Jeopardy and chess have lesser components of 'human intuition' than Go. They are apples and oranges. So, I wonder if the author errs in bringing Deep Blue and Watson-Jeopardy into the picture.

In my opinion while Watson-Jeopardy and Deep Blue were 'over-fitting' for Jeopardy and chess respectively, AlphaGo algorithms are more general and over- fits for the larger category of 'games'.

[1] [http://brenocon.com/watson_special_issue/05%20automatic%20kn...](http://brenocon.com/watson_special_issue/05%20automatic%20knowledge%20extration.pdf)

<comment><USER7> I'll get hammered for this statement but.

Traditional AI suffers from a love of solving parlor tricks. Solving tick tack toe, checkers, chess, poker, Jeopardy(tm), are parlor tricks. It seems important because frankly humans just suck at parlor trick type problems and other forms of intelligence like say a cat brain don't even get parlor tricks. SO then we found that computers were really good at parlor tricks and it seemed like we were really onto something here. But nope.

On the other hand playing Go is not really a parlor trick, it's actually _hard_ simple symbolic logic totally fails to grasp the problem at the first level.

<comment><USER8> Can you provide a firm definition of parlor trick?

Not saying I agree or disagree with you, but without a real definition of parlor trick you have a wide open no-true-Scotsman defense.

<comment><USER9> Well that's the crux of the problem - the corollary to your question is "can you provide a firm definition of intelligence?" Nobody can yet, so it's all speculation and subjective opinion.

The reason that I personally consider all these things parlor tricks (including, hypothetically, complete mastery of Go) is that I see no path from these particular types of systems to general intelligence. A human can take in arbitrary sensory data and make all sorts of conclusions and associations with it. Does this particular system have the capability to get to a point where it can see an apple falling and posit a theory of gravity? Will it ever be able to read subtle cues in facial/oral/bodily expression and combine them with all sorts of other data, instantaneously, to achieve compelling real-time social interaction? Will this system ever _invent_ the game of Go, or anything else, because it felt like it? No, it has absolutely no framework to do any of those things, or countless other things humans can do. It's a machine built with a single purpose in mind, and it can only serve that purpose. It's a glorified function call. I don't think this type of machine will just wake up one day after digging deeper and deeper into these "hard" tasks. We need breadth, not depth.

<comment><USER10> You may find this paragraph on wikipedia interesting: [https://en.wikipedia.org/wiki/Great_ape_language#Limitations...](https://en.wikipedia.org/wiki/Great_ape_language#Limitations_of_ape_language)

</comment></comment></comment></comment></comment><comment><USER11> I'm a bit disappointed in this piece. It doesn't say anything surprising to anyone who read a few words into the DeepMind paper, and it serves to settle some of Marcus's academic scores:

> two people ought to be really pleased by this result: Steven Pinker, and > myself. Pinker and I spent the 1990's lobbying — against enormous hostility > from the field — for hybrid systems

Told-you-so's are almost always boring, especially when they are part of a larger campaign. Here, Marcus's campaign is that neural nets are not enough, which isn't really news to DeepMind or most other people working with NNs, and doesn't matter much to anyone not working with them.

His chief critique is their interpretability.

>In 2016 networks have gotten deeper and deeper, but there are still very few provable guarantees about how they work with real-world data.

But for most of the world, including the people using whatever Marcus's startup eventually makes, predictive accuracy trumps interpretability. Let's get to 99% accuracy and worry about why later. And that's what researchers have done for many problems with NNs. Of course it's nice to know why, but it's not a fatal flaw if you don't, most of the time.

IBM's struggles to market Watson are a bit of a straw man. If you had judged the PC market by IBM's moves a few decades ago, you might have reached the same conclusions, and you would have been dead wrong.

<comment><USER12> >But for most of the world, including the people using whatever Marcus's startup eventually makes, predictive accuracy trumps interpretability.

Does it? What if the network has 99% accuracy, but is equally confident about its correct _and_ incorrect predictions? "Deep neural networks are easily fooled", after all.

<comment><USER11> Sure, that's a flaw. It's just not a fatal flaw. For the chief reason that there's probably nothing better. So you take your lumps and remember that it's wrong sometimes. It's something we can work on while still benefitting from these tools.

<comment><USER12> >For the chief reason that there's probably nothing better.

The paper "Deep Neural Networks are Easily Fooled" noted that generative models don't suffer from this flaw.

</comment></comment></comment></comment><comment><USER13> Interesting read & puts the Google Go bot in some needed perspective.

From the article:

> In the real world, the answer to any given question to be just about > anything, and nobody has yet figured out how to scale AI to open-ended > worlds at human levels of sophistication and flexibility.

One doesn't have to shoot for the moon in order to find useful applications for AI or Cognitive technology. If you can restrict the domain of knowledge of an expert system, it doesn't need to be create 'open-ended worlds' in order to provide value. It just has to beat human effort, or be an augmentation to human cognition to enable scale, for it to be useful - or provide business value.

<comment><USER14> But then why call it AI? Perhaps the work that it does is not actually intelligent?

<comment><USER15> Is what people do 'actually intelligent'? If you break down the processes of the brain to a low enough level, all of the 'intelligence' will disappear, just as it does in a computer neural network.

Intelligence is not some kind of aristotelian substance that permeates brain matter. At some level, anything which is intelligent has to be built from parts which are not intelligent.

<comment><USER5> _> Is what people do 'actually intelligent'?_

Yes, although there are plenty of people trying to convince everyone otherwise by example.

 _> If you break down the processes of the brain to a low enough level, all of the 'intelligence' will disappear, just as it does in a computer neural network._

If you break matter down to low enough level, everything is just elementary particles. Now, would you please trade me some gold for equal mass of aluminum?

</comment></comment></comment></comment><comment><USER16> It's fascinating that nature has created human-level intelligence using blind randomness, albeit over a period of 1+ billion years.

My theory is that with renewed global focus on AI, we're going to have a lot of minds looking at this problem from various outside perspectives. I believe a breakthrough in AI will come about not from the computer science sphere, but a very unlikely area that will surprise many.

<comment><USER17> What would your guess be for that area?

<comment><USER16> Biology would be the "obvious" answer, but more specifically, I could see a breakthrough coming from psychedelic research, which is making a huge comeback right now after decades of ridicule. It's amazing how little research has been conducted in this area, a lot of scientists are rediscovering and relearning things that were first explored back in the 60s, and there's already been lots of progress related to human psychology.

<comment><USER18> Deep Mind are working on the biology thing and more specifically study of the human brain. Demis Hassabis, the main guy at Deep Mind did a PhD in cognitive neuroscience and is focused on that stuff. Not so sure about psychedelic research - I don't know how you'd use that to build computer systems even if the Google dream pictures are pretty trippy looking.

([https://www.google.com/search?q=google+dreams&num=20&tbm=isc...](https://www.google.com/search?q=google+dreams&num=20&tbm=isch&tbo=u&source=univ&sa=X&ved=0ahUKEwj41bDriNvKAhXDuo4KHZxNC40QsAQIHw&biw=1093&bih=478))

<comment><USER16> Actually, you just brought up a great point. The Google dream pictures are very similar to what you might see during a psychedelic experience.

I admit I don't know how psychedelic research could help in building AI, I'm just saying that my hunch is that a breakthrough in AI will come about from left field somewhere.

</comment></comment></comment></comment></comment><comment><USER3> An interesting article about the author, Gary Marcus, and his stealth startup[1]: [https://www.technologyreview.com/s/544606/can-this-man- make-...](https://www.technologyreview.com/s/544606/can-this-man-make-aimore- human/)

[1] [http://geometric.ai/](http://geometric.ai/)

</comment><comment><USER19> I went through Stanford in the 1980s, just as it was becoming clear that logic-based AI had hit a wall. That was the beginning of the "AI Winter", which lasted about 15 years. Then came machine learning.

AI used to be a tiny field. In the heyday of McCarthy and Minsky, almost everything was at MIT, Stanford, and CMU, and the groups weren't that big. There were probably less than 100 people doing anything interesting. Also, the total compute power available to the Stanford AI lab in 1982 was about 5 MIPS.

Part of what makes machine learning go is sheer compute power. Training a neural net is an incredibly inefficient process. Many of the basic algorithms date from the 1980s or earlier, but nobody could hammer on them hard enough until recently. Back in the 1980s, John Koza's group at Stanford was trying to build a cluster out of a big pile of desktop PCs. Stanford got a used NCube Hypercube with 64 processors (1 MIPS, 128KB each). The NCube turned out to be useless. There was a suspicion that with a few more orders of magnitude in crunch power, something might work, but with the failure of AI, nobody was going to throw money at the problem.

At last, there are profitable AI applications, and thus the field is huge. Progress is much faster now, just because there's more effort going in. But understanding of why neural nets work is still poor. Things are getting better; the trick of using an image recognition neural net to generate canonical images of what it recognizes finally provided a tool to get some insight into what was going on. At last there was a debug tool for neural nets. Early attempts in that direction determined that the recognizer for "school bus" was recognizing "yellow with black stripe", and that some totally bogus images of noise would be mis-recognized. Now there are somewhat ad-hoc techniques for dealing with that class of problems.

The next big issue is to develop something that has more of an overview than a neural net, but isn't as structured as classic predicate-calculus AI. One school tries to do this by working with natural language; Gary Markus, the author of the parent article, is from that group. There's a long tradition in this area, and it has ties to semantics and classical philosophy.

The Google self-driving car people are working on higher-level understanding out of necessity. They need to not just recognize other road users, but infer their intent and predict their behavior. They need "common sense" at an animal level. This may be more important than language. Most of the mammals have that level of common sense, enough to deal with their environment, and they do it without much language. It makes sense to get that problem solved before dealing with language. At last, there's a "killer app" for this technology and big money is being spent solving it.

<comment><USER20> Some of us went through school in the 90s-2000s and were trained by folks who never let go of the dead-end pure-logic-based systems.

<comment><USER19> Ouch. That was a decade late to be doing that.

I finished a MSCS in 1985. Ed Feigenbaum was still influential then, but it was getting embarrassing. He'd been claiming that expert systems would yield strong AI Real Soon Now. He wrote a book, "The Fifth Generation", [1] which is a call to battle to win in AI. Against Japan, which at the time had a heavily funded effort to develop "fifth generation computers" that would run Prolog. (Anybody remember Prolog? Turbo Prolog?) He'd testified before Congress that the "US would become an agrarian nation" if Congress didn't fund a big AI lab headed by him.

I'd already been doing programming proof of correctness work (I went to Stanford grad school from industry, not right out of college), and so I was already using theorem provers and aware of what you could and couldn't do with inference engines. Some of the Stanford courses were just bad philosophy. (One exam question: "Does a rock have intentions?")

"Expert systems" turned out to just be another way of programming, and not a widely useful one. Today, we'd call it a domain-specific programming language. It's useful for some problems like troubleshooting and how-to guides, but you're mostly just encoding a flowchart. You get out what some human put in, no more.

One idea at the time was that if enough effort went into describing the real world in rules, AI would somehow emerge. The Cyc project[3] started to do this in 1984, struggling to encode common sense in predicate calculus. They're still at it, at some low level of effort.[4] They tried to make it relevant to the "semantic web", but that didn't seem to result in much.

Stanford at one time offered a 5-year "Knowledge Engineering" degree. This was to train people for the coming boom in expert systems, which would need people with both CS and psychology training. They would watch and learn how experts did things, as psychology researchers do, then manually codify that information into rules.[2] I wonder what happened to those people.

[1] [http://www.amazon.com/The-Fifth-Generation-Artificial- Intell...](http://www.amazon.com/The-Fifth-Generation-Artificial- Intelligence/dp/0201115190) [2] [https://saltworks.stanford.edu/assets/gx753nb0607.pdf](https://saltworks.stanford.edu/assets/gx753nb0607.pdf) [3] [https://en.wikipedia.org/wiki/Cyc](https://en.wikipedia.org/wiki/Cyc) [4] [http://www.businessinsider.com/cycorp- ai-2014-7](http://www.businessinsider.com/cycorp-ai-2014-7)

</comment></comment></comment><comment><USER21> Interesting. I studied AI in the 1990s, and although I don't know this author, I've always felt that the real progress in AI would come from combining various techniques. I don't understand why there would be any hostility towards that idea. (Except that in research people can be very protective of their own pet projects.)

</comment><comment><USER22> The challenge is to have grammar-like and pointer-like structures in a connectionist network. To ground such symbolic notions is the entire quest!!

MC tree search converges to minimax.

I'm really fond of Bayesian methods, but I think arguments about optimality should not be overstated. The brain is just an approximation at best.

</comment><comment><USER23> So now that computers are getting good at Go, what's the next logical stepping stone for AI?

<comment><USER24> AI chess that actually thinks like a human: [http://www.popularmechanics.com/technology/robots/a17339/che...](http://www.popularmechanics.com/technology/robots/a17339/chess- engine-plays-against-itself/)

If you could get a chess engine that can prune the tree as efficiently as a human, while calculating as fast as a computer, it would be phenomenal.

<comment><USER25> After reading the DeepMind paper, it does seem like their techniques could also be applied to chess and could possibly improve the state-of-art there as well. It is unclear, however, how much improvement remains in modern chess engines. They are already phenomenal.

</comment></comment></comment><comment><USER26> Beating the world's 663th go player is not mastering the game...

I guess AI has to purge human out from earth to justify the statement "AI masters xxxx"...

<comment><USER27> The superhuman fallacy is the nemesis of all AI research: if it isn't better than the best human who has ever attempted a problem, it is worthless and derisory.

I've done a lot of work on artificial creativity, and am constantly thrilled when code generates something at the level of a creative 3rd grader. But show it to most people and you get "It's hardly a Michaelangelo, is it?"

Frustrating.

<comment><USER28> No, it's rather the idea that you cannot say "AI beats Go master" or "AI masters Go" if it didn't actually attain some sort of high ranking by itself. Beating a low-ranked player while still interesting is not necessarily a proof of proficiency but could be due to just blind luck for example.

<comment><USER29> Winning 5 games to 0 probably isn't blind luck.

There's no clear definition of "mastery." Honinbō Shusaku was a master. Honinbō Shūsai, Go Seigen, and Minoru Kitani would also be considered masters. As time marches on, average player skill increases. While Honinbō Shusaku was among the strongest players of his day, he would be hard pressed to hold his own against professionals in today's era.

I think it's fair to say that anyone who reaches shodan (1 dan professional) plays at the master level. The difference between 1 dan professional (1P) and 9 dan professional (9P) is three stones, or roughly 30 points. In amateur play, for comparison, the difference between a 1 dan and 3 dan is about 2 stones (roughly 20 points).

AlphaGo won 5 straight games against a 2 dan professional player. That puts AlphaGo around 3P, well into the master range.

In Go, the ranks are:

        30 kyu amateur (never played)
        1 kyu amateur (understands the game)
        1 dan amateur (mastered the basics)
        7 dan amateur (nearly professional strength)
        shodan (1 dan professional)
        9 dan (top ranking professional)

[https://en.wikipedia.org/wiki/Go_ranks_and_ratings#Professio...](https://en.wikipedia.org/wiki/Go_ranks_and_ratings#Professional_ranks)

[https://en.wikipedia.org/wiki/Go_professional#Discrepancies_...](https://en.wikipedia.org/wiki/Go_professional#Discrepancies_among_professionals)

"Traditionally it has been uncommon for a low professional dan to beat some of the highest pro dans. But since the late-90s it has slowly become more common."

</comment></comment></comment></comment><comment><USER30> it seems to me that while humans are built out of neural nets, we are capable of logic, and the sort of reasoning that seems to be fall under the category of "tree search" in this conversation. in that sense, it would seem that we can do deductive logical reasoning, and we run that software on neural net hardware (slash firmware/software).

so, why the slavishness among some to neural nets and building everything on top of that? why emulate logic processing on top of associative processing and then mesh the two, when you could just do associative processing and mesh it with more "native" logic processing, since man-made computing hardware already does that easily?

also: this has been brought up here and elsewhere before, but as the article mentions: "Just yesterday, a few hours before the Go paper was made public, I went to a talk where a graduate student of a deep learning expert acknowledged that (a) people in that field still don't really understand why their models work as they well as they do and (b) they still can't really guarantee much of anything if you test them in circumstances that differ significantly from the circumstances on which they were trained. To many neural network people, Minsky represents the evil empire. But almost half a century later they still haven't fully faced up to his challenges."

let's keep in mind that some of the biggest mistakes humans make come from unexamined associative reasoning. let's also keep in mind that this crowd tends to be particularly suspect of the sort of not-strictly-logical associative heuristics that we associate (haha) with more mainstream society (unexamined articles of faith, habits and social customs that have unintended deleterious effects, etc).

one of my favorite failures of loose associative reasoning is racism, so here's a link many of you have probably seen, but it's an important thing to keep in mind, so here it is anyway: [http://www.nytimes.com/2015/07/10/upshot/when-algorithms- dis...](http://www.nytimes.com/2015/07/10/upshot/when-algorithms- discriminate.html?abt=0002&abg=1)

let's not reproduce some of our species' biggest flaws just because it let us expediently reach some narrowly defined (and likely immediately financially desirable) goal.

i realized i've sort of conflated my points here: it is entirely possible to implement all sorts of terrible things (like racism) using logic. there's certainly no shortage of spurious logic to justify all sorts of bad behavior. axioms need to be sound and nuance needs to be considered. but i think my general point is that humans aren't some sort of model of perfection, and should not be copied as such.</comment>


Ask HN: How to purchase a .hu domain name without Hungarian citizenship? - <AUTHOR>

The owner of a .hu (Hungary) domain name has agreed to sell and transfer. The registrar of that domain sent me some forms I need to complete and one of them requires me to prove my Hungarian citizenship. Unfortunately, I am only a United States citizen. Is there any way around this without obtaining citizenship? I really want this domain, please advise! <comment><USER2> I would suggest contacting support at iwantmyname.com I'm not sure if they handle .hu domains, but they do handle most ccTLD's, so if there's a way to do it, they'd know.

<https://iwantmyname.com/>

BIAS: I'm a just a customer of iwantmyname.com for the last two+ years.

</comment><comment><USER3> Dunno about transferring domains, but <http://www.101domain.com/hu.htm> offers proxy registration for a $15 fee.

</comment><comment><USER4> Many domain registrars offer proxy registration, where the name is registered under a Hungarian citizen's name (but owned by you).</comment>


Ask HN: Why do maintainers optimize for small binaries? - <AUTHOR> Just to be clear, I'm not talking about GUI apps or websites, those are separate conversations (cough cough 350mb Electron apps and 10mb Webpack bundles).
I'm talking specifically about CLI tools, webservers, and other tools distributed as static binaries or via package managers.
What's the reasoning for so many package maintainers optimizing for <5mb binaries at the expense of usability?
It seems like when >90% of hosts are running on 2008+ hardware with SSDs or even moderately fast HDDs, loading time and storage space are not major issues below the 30~50mb mark.
A recent example from HN: https://raphlinus.github.io/rust/2019/08/21/rust-bloat.html <comment><USER2> The simple answer is that they don't. It's a false premise.

Developers who are specifically targeting systems with limited memory will try to produce small binaries. If you're talking about a distribution like Ubuntu, though, it's simply not a concern. At all. Applications are built to do whatever it is that they need to do, and whatever size they end up being is how big they are, almost without exception.

The reason CLI binaries are small is that ALL binaries are small. They are compiled, and any resources they need are stored externally in other files. They use shared libraries, making the code even smaller through re-use.

I have 1785 programs in /usr/bin on my Ubuntu server, and all but 10 of them are under 5M. The ones that are larger are only big for unusual reasons (e.g. mysql_embedded).

I'm not sure what you're referring to when you talk about usability. Are you saying that my 1.1MB nginx lacks some utility? And that it lacks it because someone was worried about the size of the binary? That's simply false to the point of being nonsensical.

One of the biggest binaries I use is the Postgres server, at a hefty 6.5MB. Is Postgres missing features that affect its usability?

<comment><AUTHOR> Nginx lacks nginx_http_perl_module, nginx_http_lua_module, and many other modules that I wish it had out of the box. This is why I end up installing openresty instead on most boxes.

Postfix lacks OpenDMARC and OpenDKIM out of the box, which are recommended for almost all non-satellite mail server installs these days. Granted they're not native Postfix modules, but they're so important that I wish it had them out- of-the-box.

Same is true for caddy, certbot, and several other tools I use often.

<comment><USER3> All Linux distros handle these things differently. Some build in every feature under the sun, some try to modularize and build 30 related packages, some strip it down to minimal dependencies. It depends on the package, the maintainer, and the distro.

It is often necessary to build custom packages if you want functionality which isn't core to the original software (perl and lua aren't necessary to run nginx core). If you work with the package maintainer, you may find a way to have the distro build the extra functionality as optional packages.

<comment><AUTHOR> > perl and lua aren't necessary to run nginx core

But that's the crux of my argument, most of the time I don't just want the core, I want to be able to do almost everything and I'm willing to pay the binary price to have an "apt install nginx-everything" available.

Users with limited system resources could always "apt install nginx-minimal".

(If you can show me a distro that includes all the modules for nginx, caddy, postfix, certbot, etc. by default, I'll switch in a heartbeat!)

</comment></comment></comment></comment><comment><USER4> A few thoughts:

(1) I'm not sure that it is at the expense of usability, so I want to question your premise.

(2) Size on disk is a proxy for many other kinds of bloat. That article specifically mentions one in the opening, compile time. Surface area of the codebase is another good example, where people are concerned that the amount of libraries and code that they're pulling in means higher probability of library bugs and maintenance burden.

(3) Bandwidth still has a cost.

<comment><AUTHOR> You don't wish nginx supported using environment variables in config files out of the box?

I'd argue that lacking a way to do that is a significant hit to usability, and it requires recompiling with the nginx_http_perl_module or lua module to get that feature.

</comment></comment><comment><USER5> On my machine I have ~1500 packages. I don't really care if a few are very large, but on the whole, they have to be pretty small. My SSD has limited space and I'd like to use it for things where IO performance is important for the tasks I'm doing. That random package I don't use that's pulled in as a dependency for a feature I don't even have enabled takes up space.

E.g. if you use protobufs for C++, you probably don't want to pull in all of ghc so protoc can support haskell. You would rather use that space for ccache or something.

</comment><comment><USER6> A lot of places still have pretty slow/unreliable internet. Developing countries often end up being basically mobile-only because it requires less infrastructure, but mobile-only doesn't mean 4G in such places. Other times, relatively rich countries will have spotty internet access because of geography: New Zealand famously had very poor internet service for a while because of the complexity around getting cables there, and Australia has such a big difference in population density between urban and rural areas that wiring infrastructure in the sticks is tough. Even in the united states, a surprising number of people are still using dial-up internet because broadband is not available.

Another concern is embedded platforms. If rust can't produce meaningful code that's under a megabyte, then it can't be used to write elevator controllers or microwave oven displays or to program pacemakers -- and as regular computers get more RAM to try to make up for the end of Moore's law, a larger and larger percentage of cases where it makes sense to use a compiled systems language over a JITed scripting language will be embedded ones. To some people, if a compiler can't produce 1k binaries, it's worthless because it forces you to substitute in a beefier machine.

There's also general performance and security concerns: any unnecessary code adds to both vulnerability surface & footprint. If you work your system really hard & you're always a meg or two short of running out of RAM, then saving a meg or two of code size matters a whole lot. (This applies less to desktops & more to applications where you might process lots of data on off-the-shelf hardware: if you're clever & take advantage of parallelism, you can process many TB of data on a machine with less than a gig of ram in a relatively short time, and you can cut it down further if you can run more copies of your application, which you can do better if your memory footprint is smaller.)

Finally, I don't think it's safe to overestimate what percentage of machines are less than eleven years old. Ever since Moore's Law ended, the case for upgrading hardware has been a lot weaker, and even when it was running strong folks often went a decade without doing so.

Bloat doesn't necessarily translate into usability, and usability doesn't necessarily transfer between users. There are a lot of folks for whom a 5mb app is, necessarily, unusable.

<comment><AUTHOR> > "unnecessary code adds to both vulnerability surface & footprint"

I wholeheartedly agree with this. If there is unnecessary code, there's no reason to keep it.

I'm more directing my gripes towards packages that have highly desirable, well-audited add-ons / modules that package maintainers choose not to include in the default distributions (under the reasoning that "users want small binaries").

Take for example nginx, caddy, certbot, or postfix (there are many others too).

All of these require recompiling from source with build flags to enable the inclusion of even their most common add-on modules, e.g. nginx_http_lua_module, nginx_http_perl_module, caddy:http.cache, cerbot dns plugins, etc.

Recompiling from source breaks the ability to use a package manager for install and automated updates, which drastically reduces usability for the majority of users. There are ways around this of course, but for the average user, having to compile a package from source is a major hurdle.

Instead, why not distribute the binary with no add-ons as "apt install packagename-minimal" for the users with bad internet / low-resource requirements, and make the default "apt install packagename" distribution the "batteries included" version?

(If you're interested, here's an old blog post of mine that goes into detail on why I think package manager distributions are worth the effort to maintain in general, even for static binaries or packages with dylibs: [https://docs.sweeting.me/s/against-curl- sh](https://docs.sweeting.me/s/against-curl-sh))

<comment><USER7> Install nginx-extra then?

<comment><AUTHOR> But why not have that default and the non-extras version 'apt install nginx- minimal'?

What about caddy which doesn't offer module support _at all_ in the package manager verserion?

Certbot and postfix also don't include their most commonly used modules out of the box.

<comment><USER8> If apt and others could simply allow us (package authors) to construct a URL from which the binary can be downloaded, and then plop it into the user's PATH for them, this would not be a problem. (What I'm getting at is that all it takes to install a custom build of Caddy -- with any number of plugins you want -- is a GET request.)

</comment></comment></comment></comment></comment><comment><USER9> Instruction cache. If your mega binary doesn't fit into cache, instructions jump around to different and unpredictable address spaces. You are fetching and executing instructions directly from memory. This is why huge applications are so slow, they are both instruction memory bound and data memory bound. Imagine how slow a CPU is with no cache!

<comment><USER10> A ryzen 2300X has 256KB of instruction cache, that's 4x64 KiB 4-way set associative. How can you hope to fill that?

I've personally ran gentoo on a 4.2GHz i3-530 as well as a 4.5GHz FX-4350. From kernel to firefox, I only gained performance going from -O2 to -Ofast. I have no hopes of considerable cache misses on a modern CPU.

<comment><USER11> Remember that your process is not the only one running on the CPU or core. Most of the time, at least, and even if it is via some artifice you get to call kernel code for syscalls and C library too.

</comment></comment></comment><comment><USER12> I'm still not convinced of your claim that "package maintainers optimize for <5mb binaries at the expense of usability". I've never heard anyone talk about that. Do you have more examples?

<comment><USER13> Possibly precipitated by this post which was on the front page yesterday with 400 upvotes:

[https://news.ycombinator.com/item?id=20761449](https://news.ycombinator.com/item?id=20761449)

<comment><USER12> That example was given in the original post. That opinion is Raph Levien's ("One of my hopes for xi-editor is that the core would be lightweight"), so if that's the only example, the OP should have asked him directly.

</comment></comment></comment><comment><USER14> Business logic isn't going to be a large part of a programs size. Most space is used by static resources likes images, lookup-tables, and libraries for handling the former. People aren't cutting out features from their CLI programs to save a couple kilobytes.

<comment><USER15> Hum... They are (specifically, cutting speed), as evidenced by the OP's link.

Not that it matters with the low number involved.

<comment><USER16> I might be missing the point here, but larger binaries are not always slower.

</comment></comment></comment><comment><USER17> The cynical answer is: binary size is, like the price of gasoline, highly legible all out of proportion to its importance, and for that reason, influential. It's easy to see instantly how big a package is because everything reports it as an easy standard metric. You can see binary sizes go up as you add functionality or dependencies. But it's not easy to see actual end-user performance, bug-freeness, or security (binary size being only weakly correlated, at best, to things like those, which the user actually cares about). Setting up a meaningful benchmark or testsuite is much harder than noticing that your binary is no longer <5mb. So, the latter happens more than the former.

<comment><AUTHOR> You can still claim that you have a small core binary of 200kb, while offering the 20 modules in the default package manager distribution that bring it up to 8mb.

Then refer people to "apt install packagename-minimal" if they only want the core.

<comment><USER17> You can but that requires more of an attitude of gaming metrics than most people will have. Lots of metrics are easy to game if you explicitly set out to, but most people aren't dishonest; it's the ones which are easily gameable without explicitly trying to which are the dangerous ones, because most people are honest.

</comment></comment></comment><comment><USER18> I'm the maintainer of ShellCheck. I minimize the size of the binaries because I pay per gigabyte transferred for the hosting, and to avoid playing into Haskell's rep of being bloaty.

<comment><USER19> Shellcheck is huge because of the language runtime, right? It's a couple tens of megabytes IIRC, whereas, had it been a normal C++ program, I would've expected it to be maybe a few hundred KB, maybe a MB or two if pushing it.

</comment></comment><comment><USER20> > Once you accept bloat, it's very hard to claw it back. If your project has > multi-minute compiles, people won't even notice a 10s regression in compile > time. Then these pile up, and it gets harder and harder to motivate the work > to reduce bloat, because each second gained in compile time becomes such a > small fraction of the total. > [https://raphlinus.github.io/rust/2019/08/21/rust- > bloat.html](https://raphlinus.github.io/rust/2019/08/21/rust-bloat.html)

If accepting a feature that causes a 3x compile-time regression were easy, then that 29s build is just 5 easy decisions away from a 2-hour build. Now, nobody cares about adding more time to the builds, since nightly, and developers commit hoping it doesn't fail. This is sadly all too familiar.

> Good programmers write good code. Great programmers write no code. Zen > programmers delete code. [https://www.quora.com/What-are-some-things-that- > only-someone...](https://www.quora.com/What-are-some-things-that-only- > someone-who-has-been-programming-20-50-years-would-know/answer/John-Byrd-2)

</comment><comment><USER3> Well, ask yourself the reverse: what am I gaining by having unnecessarily large binaries?

If you ever have an automated process built around your tool, eventually something will run it lots of times. The more it's downloaded, copied, and run, the more the "bigness" affects performance. It's best to choose smaller whenever it doesn't take away features that you need.

<comment><AUTHOR> I'm gaining all the functionality of nginx_http_perl_module, nginx_http_lua_module, caddy:http-cache, and all the other modules that I wanted to have in the first place.

</comment></comment><comment><USER21> The most accurate answer w/r to the thing you linked is: All binaries used to be much smaller, by a factor of 100x or more. And so on this aspect of performance we have had a kind of regression where most things have a bigger footprint than necessary. Some of it is to be expected since 64-bit code is larger, and Unicode handling is larger, and we support more features even in the smallest CLI tools, but there's also an element of the tooling making an optimization that is locally sensible - expand code size to get some runtime performance benefits - and globally useless, since as the featuresets increase an increasing percentage of all code is "run once/run never" and not a hot loop, and so it's just wasting disk, memory and cache to have it in that configuration.

</comment><comment><USER22> Efficiency.

Here's an analogy using your question:

Why do aircraft builders optimize for weight?

Just to be clear, I'm not talking about cargo planes or helicopters, those are separate conversations.

I'm talking specifically about passenger airliners, fighter jets, and other aircraft that are built in large quantities.

What's the reasoning for so many aircraft builders optimizing for weights <50000 kgs at the expense of usability?

<comment><AUTHOR> Given that most plugin-style functionality can be toggled in runtime config so that it's never loaded into memory when unused. And given that maintainers can provide a separate "apt install package-minimal" distribution with none of the plugins included for people with low, bandwidth.

How does a smaller binary make things more efficient?

</comment></comment><comment><USER23> >loading time and storage space are not major issues below the 30~50mb mark.

Even if you have ample disk space and memory, smaller binaries can be a performance advantage. A smaller binary with less instructions will more easily fit into CPU caches. Binary size can be an indicator of performance.

<comment><AUTHOR> Not if you design it so that optional functionality can be toggled in runtime config. Nginx's optional modules never get loaded into memory if you don't include them in your config.

</comment></comment><comment><USER24> Hi Nick! Happy to answer from my point of view. I just reduced one of my packages to 1/1000 its original size: [https://bundlephobia.com/result?p=drive- email@example.com](https://firstname.lastname@example.com)

There are several reasons that I like small packages, both when making them and when consuming them:

\- Easier to maintain. One packages does a single thing, it's easy to reason and compose with other packages. This is specially true for utility packages. It's easier to document a single thing, to debug a single functionality, etc.

\- Faster installs. It does take several minutes under certain circumstances to install larger packages/projects for me. Not everyone using tech tools live in a world with fast internet. Sure this amounts to ~30 min/week max, but would prefer to use my time differently.

\- You say 30-50mb, but my typical React project is 80-100 MB in node_modules. As an example, my current laptop is ~4 months old, and the "projects" folder has 670k+ files and weights 6+ GB.

\- Copying projects. While you talk about binaries, it's usual that it's either a single minified file with a decent size, or hundreds/thousands of dependencies. While the size itself doesn't matter so much, the amount of files matters for things like backups, searching in files, etc.

\- Signaling. People who care about this, normally won't throw a lot of dependencies on top of it if it can be easily avoided. So you know there aren't many surprises normally or Guy Fieri images: [https://medium.com/s/silicon-satire/i-peeked-into-my-node- mo...](https://medium.com/s/silicon-satire/i-peeked-into-my-node-modules- directory-and-you-wont-believe-what-happened-next-b89f63d21558)

\- Marketing. Many people care about it for these or other reasons. Everything else being the same, a smaller package is better, so there's no disadvantage if you can easily shed some of the library size. I don't care that much, especially because of what you say. But some people seem to do.

\- Tooling is easy! Rollup, webpack, uglify, etc. There are many easy (okay, not Webpack) projects to bundle and minimize a project, which are needed from front-end JS.

Now, I wouldn't optimize at the expense of usability or dev time. For instance, the two features I removed from `drive-db` were removed because they were half-baked. File cache ~> replaced by in-memory cache, which also allows for browser and Cloudflare Worker usage (the main point for refactoring). MongoDB-like queries? ~> JS today is good enough not to need those for a small spreadsheet. They didn't work the same as MongoDB, and not even I, the library author, used them ever.

<comment><AUTHOR> Thanks for the detailed response!

I think I should've added more detail in my original post explaining that I'm very much pro-removing code, my gripes were directed at packages that have optional modules that aren't included by default because maintainers claim "users want small binaries".

I'm of the opinion that most packages should include their optional add-ons in their default distributions, and offer a separate "minimal" version without any add-ons for the users with low-resource requirements.

(See my response to the other comment above too)

<comment><USER24> I see, I totally agree! I want to rewrite [https://serverjs.io/](https://serverjs.io/) at some point, and I'm thinking about what functionality to add. e.g., CORS will probably be a config toggle instead of having to add custom middleware.

But what about things where it is purely an option? There's a slippery slope there IMHO. Database connection? For a server library, no way. Key Value store? Also no way... no, wait, those are used for sessions, now you have to either do no sessions by default, have in-memory sessions which are super tricky (because they "work", until there's tricky production issues) or add Redis/similar connection. Same for rendering templates, I added the 3-4 most common but not others and feels really "meh". It's also A LOT of work to add the top 3-4 options, even without documentation.

<comment><AUTHOR> Those decisions seem pretty reasonable from the perspective of a mostly-Django dev! Do you think >30% of users would want to install those things every time they use serverjs.io? If not it seems fine to leave them out.

I think Django supports db-backed/cache-backed/file-backed sessions out of the box, but I think Flask doesn't have native sessions at all, it's only addable with additional libraries, but I could be wrong.

</comment></comment></comment></comment><comment><USER25> For Linux cli tools the obvious (to me) answer is IoT devices. These devices are typically pretty lacking in terms of hardware capabilities. So for common tools having them as small as possible allows them to be distributed on a wider range of devices.

<comment><AUTHOR> So provide "apt install packagename-minimal" for those with low-resource requirements. Otherwise the tyranny of the minority forces the majority to accept usability reductions.

</comment></comment><comment><USER26> Performance is a major concern. Static binaries help also being much more redistributable and hassle-free compared to dynamic linked ones (imagine all those DLLs, so or jar files around - search for DLL Hell or JAR Hell for instance)

<comment><AUTHOR> Most optional plugin-style functionality can be toggled at runtime, and never has to be loaded into memory, why would including more optional modules in the default distribution affect performance?

</comment></comment><comment><USER27> The example you referenced, Druid, is infact a GUI toolkit.</comment>