Richard Stallman challenges 'misleading' coverage of his comments on Minsky - <AUTHOR> https://news.slashdot.org/story/19/09/14/2033212/richard-stallman-challenges-misleading-coverage-of-his-comments-on-marvin-minsky <comment><USER2> I don't know much about Stallman, but he does seem to be fond of saying "well, _technically_..." in discussions where it's pretty tone deaf and easy to misconstrue, in a similar way to some discussions on HN itself. As someone says in the email thread, "If we're debating the definitions of 'rape' and 'sexual assault', perhaps it's better to accept that this conversation isn't productive. When this email chain inevitably finds its way into the press, the seeming insensitivity of some will reflect poorly on the entire CSAIL community."

But the initial coverage, particularly the Daily Beast article, does seem like a real hatchet job. There's a big difference between what the Daily Beast headline implied he said:

"Victims were 'entirely willing'"

and what he actually said, when discussing how much fault attaches to Minsky:

"The most plausible scenario is that she presented herself to him as entirely willing. Assuming she was being coerced by Epstein, he would have had every reason to tell her to conceal that from most of his associates."

Edit: link to a PDF of the email thread from Vice is [https://assets.documentcloud.org/documents/6405929/091320191...](https://assets.documentcloud.org/documents/6405929/09132019142056-0001.pdf)

<comment><USER3> He seems to be fond of saying "well, technically" on every subject; regardless of if it is tone deaf.

I don't want to pathologize Stallman, but I have a simmilar tendency, so I will project myself a bit here; and probably go a bit of a rant.

This tendency stems from a way of thinking about the world. Inaccuracies build on themselves, so it is generally best to stamp them out when we can. If the technicality is not actually important, then _surely_ everyone involved would have no problem with the correction. They may be annoyed that they had to spend a few minutes on clarifying such an obvious point.

What happens with the kind of discussion that Stallman entered into is just weird. People seem to get deeply offended by the person stating the technicality while claiming that the technicality is not important and without even disagreeing with the technicality.

This reaction (which seems to be consistent for the topic in question) has the almost tautological effect of excluding an entire class of thinkers from being able to participate in the conversation (or rather an entire class of thinking; some people are more able to switch between modes of thinking easier than others). I am obviously biased in this regard, but I happen to think this does a massive disservice to people who actually care about the issue, because I believe that this excluded mode of thinking happens to be extremely effective at problem solving.

> "If we're debating the definitions of 'rape' and 'sexual assault', perhaps > it's better to accept that this conversation isn't productive."

If you are concerned about rape and sexual assault then discussing what those words mean is a vital part of dealing with the issue. It often illuminates the underlying reality as thinking about definitions involves considering the details of what you are trying to talk about, and definantly illuminates the conversation as you do not have different people thinking the same statement means different things. If you catagorically reject these discussions, then I do not see how you can go on to effectivly deal with the underlying issue.

<comment><USER4> Talking about technicalities is an extremely common derailing tactic. So that's the reason why people are upset about technicalities that don't matter.

They derail discussions and it doesn't even matter if it's intentional or not. They are just corrosive and destructive.

</comment></comment></comment><comment><USER5> Might I suggest that this is just the equivalent of gossip under the sheen of hacker news and self-righteousness?

People like Richard Stallman are non neuro-typical geniuses. They focus intensely on some topics, and harmlessly pontificate on many other topics. Historically, before the internet reached its more recent point where everything is so well indexed and networked, that the fixed cost to mob justice has dropped to near zero, we would just ignore these people.

So some genius mathematicians or computer scientists (or whoever) has some weird overly-complicated political views that they poorly communicate, and won't ever do anything about? Who cares, leave that for the faculty gossip.

Now it's so easy for us to share, shame, and discuss. But what value do we offer the world by scrutinizing and shaming some computer scientist who is not seeking any political power? Does anyone have a counter-argument to why this is a valuable and useful thing for us to care about and discuss?

<comment><USER6> Richard is specifically a public figure who influences a lot of people, and some of that influence comes from platforms like his MIT professorship that imply endorsement from other groups.

I agree it's distasteful for people to focus on personal judgements about Stallman's communications when we haven't walked in his shoes, mentally. But public discourse is necessary to determine what to do with the platform given to comments like this - including if they come from "non-neurotypical" individuals.

Separately, because Stallman's area of focus extends into community formation and protection of individual rights, I don't agree that his statements about sexual coercion are separate from the rest of his work.

<comment><USER7> I think the point that RMS was trying to make is that using inherently charged and prejudice language against an individual is sensationalist and (as mentioned) prejudiced.

We should be able to have rational, mature, fact based discussions and entertain suppositions about what may have happened without jumping to conclusions.

<comment><USER6> Stallman's email reacted to the use of "accused of sexual assault" in an event announcement. This phrase matches the legal definition of the current facts, so this is _not_ about someone choosing inflammatory language beyond the norms.

Stallman would prefer that language distinguish between physical coercion and the kind of coercion that can take place because of a difference in knowledge or power (in this case, age). That's a fine point to discuss, but there are very considered reasons people use the same term - it's not just sloppy language.

Also, he'd prefer to assume a scenario where Epstein hid any coercion from his guests - which would make this a "an alleged but possibly unknowing sexual assault". I'm not sure what that's supposed to do to our discourse. Is everyone supposed to imagine the same scenario he does? Or the most expultory possible scenario? I think the uncertainty of the situation and presumption of innocence were already covered pretty well by "alleged".

The facts of the actual case are for the courts to sort out. Stallman's focus on avoiding the factual label for the accusations is a weird priority, and there is nothing immature or irrational about calling it out.

<comment><USER8> > _" Is everyone supposed to imagine the same scenario he does?"_

I think it's fair to expect everybody to consider the possibility (and subsequently discount it for lack of credibility.)

</comment></comment></comment></comment></comment><comment><USER9> I really, really, do not understand why some people's reaction to this is trying to get him fired or to apologize.

Right here we have one human being that will ponder on random issues and offer his honest opinion. He will do it without regard for social pressure. This is an incredible rare thing! This is an amazing thing, go browse your Facebook or LinkedIn feed and see how "authentic" it sounds. What's more, this gentleman is a very rational person and does not shy away from thinking about pedophilia and various paraphilia. What's more he seems truly inquisitive in understanding how to measure what Minsky did, what motivations and perceptions each participant had, etc. This is so much more important than just getting revenge, being constantly angry on the internet or just saying Epstein was evil and calling it a day.

We need much, much more people like Stallman and much, much less people that will cry foul on a Medium post when someone doesn't go along with what they want.

<comment><USER10> People who will jump on every excuse to go "well akshually it's not pedophilia" or "well actually it's not sexual assault" on the internet are not even slightly rare.

<comment><USER9> But they will be anonymous and the level of conversation will probably not be very high.

<comment><USER10> The MIT mailing list thread that provoked this whole discussion was pretty indistinguishable from any of the millions of reddit discussions on the exact same subject.

</comment></comment></comment></comment><comment><USER11> > We can imagine many scenarios, but the most plausible scenario is that she > presented herself to him as entirely willing. Assuming she was being coerced > by Epstein, he would have had every reason to tell her to conceal that from > most of his associates.

This is a totally reasonable statement and it hardly seems like he's defending Epstein or somehow justifying what was alleged to have happened. Stallman's point is that the legal usage of the term "assault" sometimes doesn't match up with the idea of physical violence that comes with the colloquial understanding of "assault".

Also it seems super weird, distasteful, and immature when, instead of discussing with or further debating this person who said something you don't like, you go tattle on them to a sympathetic reporter. Yuck.

<comment><USER12> This is _exactly_ how witch hunts work - if you don't just follow the mob 100%, but rather point out some nuance or call for some restraint, even while generally agreeing with the purported goal of the mob, then you are the next target to be attacked. The true driving force is _not justice_ , but rather _power_.

<comment><USER13> What "power" is gained from this? Also, that was _nothing_ like how a witch hunt works.

<comment><USER14> Care to explain us how a "real" witch hunt works, in your opinion? I'm genuinely curious.

</comment></comment></comment></comment><comment><USER15> I disagree with RMS on many topics, but I don't get the outrage at him. He explained how Minsky was likely not guilty of sexual assault. To use an analogy: If I have a sanctioned boxing match with someone, and it is later discovered that Don King coerced my opponent into taking the match against his will, am I guilty of battery? I don't think so.

RMS is also correct that Epstein was many things, but not a pedophile. He's discussing facts and people are treating it as if he's discussing morality.

<comment><USER16> Epstein's victims being sex slaves is not the unusual outcome but the expectation. His behavior was fairly known.

A better analogy would be participating in a fight organized by somebody who is known to coerce fighters into participating.

<comment><USER15> I don't think that analogy is better. The Minsky allegation is said to have happened in March of 2001. That was years before Epstein was investigated, let alone charged or convicted of anything.

</comment></comment></comment><comment><USER17> I think I've seen several posts on this ongoing debacle.

I keep wondering if there is anything we can do to either elevate the discussion of the matter or more thoroughly stamp the discussions out like the little dumpster fires they tend to be, which naturally threaten to become towering infernos.

I debate whether refusing to comment at all and simply flagging them is the least worst thing I can do in a world that seems to have no constructive path forward on such topics. But that also strikes me as making the topic _unspeakable,_ a practice known to deepen moral problems, not resolve them.

</comment><comment><USER18> Stallman is the worst person to be making these arguments. In a much different context, Matt Damon tried to reasonably add nuance to #metoo and got viciously attacked.

I don't like witch hunts or ruining people's careers for what they say, especially when media exaggerates it to make it sound worse. It's bad enough on it's own.

There's also other billionaires out there, one with an infamous island in the Bahamas, who has dozens of horrific stories about him. A former President and his wife visited this island, and they have lots of photos together. I hope the media and the outrage stays focused on the principals and their enablers, not poorly expressed opinions of people trying to defend deceased friends.

</comment><comment><USER19> From [https://medium.com/@selamie/remove-richard-stallman- fec6ec21...](https://medium.com/@selamie/remove-richard-stallman- fec6ec210794), stallman is quoted saying

> "I disagree with some of what the article says about Epstein. Epstein is > not, apparently, a pedophile, since the people he raped seem to have all > been postpuberal." He preferred to call Epstein a "serial rapist."

yeeeeeesh

<comment><USER20> It is weird how often that seems to happen. When subjects like this come up, people suddenly get _very_ pedantic about the definition of pedophilia versus ephebophilia, and the threads here have been dominated by sprawling, legalistic arguments about the precise definition of "sexual assault" and "rape" and the relevance of ages of consent and whether words even mean anything, really.

And this ... as if the distinction between prepubescent child rape and postpubescent child rape is at all relevant enough to mention.

<comment><USER21> _> as if the distinction between prepubescent child rape and postpubescent child rape is at all relevant enough to mention_

It is if the distinction between prepubescent children and postpubescent children is relevant, which certainly is the case in our current system of laws.

Furthermore, failing to observe such distinctions makes every single term that describes any kind of activity that some people don't like--pedophile, rapist, etc.--all equivalent; they all just mean "bad person". But that makes it impossible to have a useful conversation at all, because "bad person" is not a useful category if you're trying to figure out what to do about such people. You have to look at the specifics. If you refuse to look at the specifics, then you're not helping to do anything about the problem; you're just virtue signaling. "I think all of these people are just bad!" Okay, got it, thanks for sharing.

</comment></comment></comment><comment><USER22> Seeing quite a few of these outrage and holier than thou post lately and I'd like to see even less of them in the future.

Hackernews does a pretty good job on filtering out all the political stuff but all this #metoo, "can you believe he said that?" posts are not things that I want to read about here.

<comment><USER23> It would concern us all, if Stallman were to fall and the ranting author got away with it.

I'd argue there is a higher number of people who'd go "Well, technically" in tech than anywhere else. We're (mostly) not evil, but it may often seem that we're defending those that are, when _technically_ , we're just pointing out factual truths.

</comment></comment><comment><USER24> The thing from that Alumni and Vice are basically hit-pieces, designed to kill the reputation of RMS.

<comment><USER23> I'm hoping he'll legally retaliate. IANAL, but the misleading VICE, Daily Beast and Fox News headlines appear to be defamation/slander, wouldn't he not stand a good chance in court against them?

</comment></comment><comment><USER25> When you find yourself in a hole, stop digging.

</comment><comment><USER26> "I am skeptical of the claim that voluntarily pedophilia harms children." — Richard Stallman

[https://stallman.org/archives/2006-may- aug.html#05%20June%20...](https://stallman.org/archives/2006-may- aug.html#05%20June%202006%20\(Dutch%20paedophiles%20form%20political%20party\))

<comment><USER27> My question is why hasn't the open source community shunned him for these sorts of comments? Someone like the ex-Mozilla CEO was shunned and booted out of his position for much less.

<comment><USER28> Well technically it would be the free software community who would shun him, since the open source community have already rejected his principles and leadership.

<comment><USER29> I see what you did there! Well played sir!</comment></comment></comment></comment>


CIA, FBI launch manhunt for leaker who gave top-secret documents to WikiLeaks - <AUTHOR> http://www.cbsnews.com/news/cia-fbi-on-manhunt-for-leaker-who-gave-top-secret-documents-to-wikileaks/ <comment><USER2> Not just in "spy cases", but in all areas -- e.g. a missing "fugitive" the other week in a nearby state -- I read and hear about the incredible amounts of manpower (gender-neutral) and related resources being devoted to these "hunts."

And I can't help thinking, 'What if you (random gov. agency, or whatever -- and the politics behind it) put those resources _into addressing the underlying problem?_

In my neighboring state problem, several _man-years_ of effort and wages -- including, I'm betting, a significant chunk of overtime wages -- went into tracking down one fellow who went off-grid and was found -- in a non- confrontational manner -- by a local farmer.

That would buy a lot of assistance and prevention programs. A lot of pro- active healthcare and remediation -- areas where a relatively little amount of money goes a long way.

In these "leak" cases, nobody seems to work up-front on systems design and mitigation. And, in my opinion and wish, full-book audit trails; I'm not particularly interested in controlling these leakers, but I would like an inviolable record of every use of these systems that also documents uses to circumvent citizens' rights and due process.

</comment><comment><USER3> Think about what this does to the morale of the people left at the cia or the nsa working on software. You face an endless series of investigations about leaking stuff. It doesn't end even if you stop working there probably. That's one reason I have always hesitated about a government job, there's never an end to them bugging you. You work at Microsoft for a few years, you get a new job, no one comes to arrest you or sift through your possessions or surveils you online trying to see if you did something wrong. At least there's a lot less chance that mikeysoft would do that, compared to the govt.</comment>


When Patents Attack - <AUTHOR> http://www.npr.org/blogs/money/2011/07/26/138576167/when-patents-attack <comment><USER3> I get anxious every time I read another patent troll story wondering when my weekend creations will end up costing me $ when they accidentally infringe on a patent because I roll out some seemingly ubiquitous feature.

I'm deeply saddened by this mess and doubt it will ever be resolved. There is far to much money at stake now to revert things - money = lobbying.

<comment><USER4> If it's any consolation, hearing from a patent troll means you've built a successful product.

<comment><USER5> Except you don't need to make a profit to be subject to a lawsuit. Hobby or weekend projects are just as liable.

<comment><USER4> Well sure. But from a practical standpoint, a patent troll is far less likely to bother with someone who doesn't have reasonably deep pockets.

<comment><USER6> While this is generally right, it is also fairly common for patent trolls to go after some small fish they claim are violating a patent first to set a precedent of wins (or settlements) against opponents with weak legal war- chests.

A string of these small victories makes it much easier for them to shake down the big targets later... So while you PROBABLY won't be the target of a lawsuit unless you're very successful, there's a bit of a lottery that happens pretty often with these cases where some small percent of small targets _do_ get run over by the patent holders on the road to the bigger guys. And if you happen to draw the short straw on that, it sucks to be you.

</comment></comment></comment></comment></comment><comment><USER7> There is a simple solution to this problem. Make maintenance fees payable yearly, and make a requirement of getting the patent renewed proof that you're actually using it. People filing patents only to sit and wait until someone else infringes is a cancer on American innovation.

<comment><USER8> Definitely like the second part. It would seem that if you aren't actually using your patent, the patent's purpose is lost.

I also think there should be oversight on the patents that are granted. Right now you can patent something that has 20 years of prior art with relative ease. That's silly.

<comment><USER9> There _is_ the doctrine of laches, a defense in the case of suits brought after a long period of non-enforcement. As with most defenses, this entails lawyering up and going to court, part of what makes this process so expensive.

There were also reforms in the 1990s which extended the life of a patent (from 17 years to 20), but started the clock at filing time, not grant. This was due to a slew of patents issued to Jerome Lemelson which he kept appealing to the patent office. These were eventually granted, a practice termed "submarine patents". Lemelson (and his heirs) were awarded over $1.3 billion in royalties (he's also got a wing of the Smithsonian named after him).

But the system's still pretty messed up.

</comment></comment></comment><comment><USER10> Throwing an idea out there: Software patents should last no more than 1 year.

If you come up with something worth patenting, you get a year lead on competitors. That's it.

<comment><USER11> I think a better solution is a mandatory licensing process. The threat of absolute exclusion is too damaging to the industry.

AND, the patent office needs to start DOING THEIR JOB and stop granting patents for obvious inventions.

<comment><USER12> Honestly, if the patent office would just do their job and not grant bogus patents based on obvious inventions or inventions with prior art, that would probably be enough.

Software patents aren't the problem, bad software patents are the problem.

<comment><USER13> Can you name an example of a good software patent?

<comment><USER12> As long as it meets the non-obvious and no-prior-art tests, I don't see a problem with the concept.

Something like mouse gestures feel like a patentable innovation worthy of a temporary government-enforced monopoly.

<comment><USER14> Sure, but not for a decade or more. The timescale of patents is simply incompatible with the timescale of software.

<comment><USER12> I totally agree on that front. Five years seems like a reasonable upper limit to how long software patents should last.

</comment></comment></comment></comment></comment></comment></comment><comment><USER15> As the article only touched on, this is precisely why the next computing giant will not come from Silicon Valley or anywhere in the US. The current software patent situation has ensured it.

<comment><USER16> I doubt it. The next computing giant will want to sell in the US, too. Patent laws don't cover where you're from, they cover where you sell (or where you give away free stuff but make money from ads). You'd have to block all US visitors from your site to avoid the US patent system.

I've been thinking about workarounds. Could a "I confirm that I'm from Malaysia" button do the trick?

<comment><USER17> May be the future companies will price products higher in US markets.

</comment></comment></comment><comment><USER18> As a co-founder of a startup, my most worrying thoughts are not finding customers or improving the product. They're "when will I get a scary legal letter that asks for millions over a patent I've never heard of".

<comment><USER19> Really? I never ever think about patents. Ever (except when reading HN). It's just not a real day to day issue.

<comment><USER20> The biggest part of the problem is that companies (and founders) are forbidden from speaking up after licensing deals are made. Perhaps you could post anonymously, but you'd still be violating the NDA in spirit and you'd be punished if discovered.

In other words, if three companies down the block signed millions away to Intellectual Ventures, you would have _no_ idea. Maybe the risk _is_ overblown. Unfortunately we have no idea of knowing.

They try to hammer this point home in the podcast.

It occurred to me there is simple legislation that might chip away at all this shadiness: Pass a law that forbids NDAs in patent licensing deals.

<comment><USER19> You'd know it if it happpened. Management may not do a PR, but people talk. Remember the lawsuit and NDA agreement are proceeded by talks. At the very least you'd know they were approached and then suddenly went silent on the matter.

If it did work it would be the greatest conspiracy ever pulled off. A company extracting a billion hostile dollars and people not even posting on anonymous sites? So not likely.

The genius of IV's NDAs isn't that they have millions of companies paying them, but that they can create the mythology that they might. And you might be next.

I'm literally about 100x more worried that my architect gets hit by a bus than I am of being sued over patents.

</comment></comment></comment></comment><comment><USER21> Did anyone else click the "someone patented toast" link and actually read the claims? It's for toasting bread at 2,500-4,500 degrees, with specialized infrared ovens.

<comment><USER22> <http://home.howstuffworks.com/toaster1.htm>

</comment></comment><comment><USER23> Intellectual Ventures recently posted a rebuttal [http://intven.com/newsroom/insights/11-07-25/Disruption_Invi...](http://intven.com/newsroom/insights/11-07-25/Disruption_Invites_Controversy.aspx)

They like to say things like "ideas have value" and that they are "disruptive."

Are there examples where the patents they own and monetize actually represent valuable "ideas" and not after the fact claims of invention? I'm guessing not, but open to being proven wrong.

And who, or what, do they think they are disrupting? Isn't IV the incarnation of the status quo?

Reading their website feels like reading a politicians...

<comment><USER24> That entire article rests on the assumption that software patents have some value outside of the ability to sue someone else with them or to ward of lawsuits by someone else. They want you to think of a software patent like it's a piece of music or a motion picture. They're just like Netflix or Spotify for patents! I don't know how anyone working for that company can sleep at night.

<comment><USER25> In movies you can't have a patent for action cop movies with a surprise plot twist. That would be absurd. Just like generic software patents.

<comment><USER26> Don't look now but...

<http://www.plotpatents.com/>

They've actually been trying for some time now.

</comment></comment></comment></comment><comment><USER27> One of the best TAL episodes in recent memory. The story mentions that modern patents (specifically software patents) lack the novelty that the patent system was originally designed to encourage. Did patent laws change at some point in history to allow this to happen?

<comment><USER28> The audio version of the story mentions that it was a decision by the patent office, not specific legislation. Sometime in the 1980s they went from copyrighting software (i.e., treating it like literature or art) to patenting it.

<comment><USER29> No, the audio version mentions that the patent office used to operate on the principle that software could be copyrighted but not patented, but sometime in the 80s the judiciary indicated in several instances that was not the case and the patent office complied.

</comment></comment></comment><comment><USER30> <http://news.ycombinator.com/item?id=2795743>

<http://news.ycombinator.com/item?id=2796906>

<comment><USER31> Yup, it's a repost, but I'm okay with it because the MP3 is now available (came out Sunday @ 7pm). Take note, it's only available for free this week. After that, I think you can find it on iTunes or Amazon for a buck or three.

<comment><AUTHOR> Yikes, sorry about that. My search for "npr patents" only brought up a mention in a comment.

<comment><USER32> I'm glad you re-posted, actually, since the original links (when posted) didn't include audio.

</comment></comment></comment></comment><comment><USER33> I'm curious as who are the VC's who invested in IV? Are these typical tech VC's, or are they more of the private equity type that relish in making money off these so called opportunities or arbitrages that don't require actual productive work.

</comment><comment><USER34> Is this an issue for companies owned by HNers? Poll: <http://news.ycombinator.com/item?id=2809951>

</comment><comment><USER35> Two very interesting Econtalk podcasts on IP issues and economics, both of which argue that innovative industries can thrive in legal regimes which offer very limited IP protections:

Boldrin on Intellectual Property [http://www.econtalk.org/archives/2009/05/boldrin_on_inte.htm...](http://www.econtalk.org/archives/2009/05/boldrin_on_inte.html)

Blakley on Fashion and Intellectual Property [http://www.econtalk.org/archives/2010/06/blakely_on_fash.htm...](http://www.econtalk.org/archives/2010/06/blakely_on_fash.html)

pg groupies may also want to check out the Econtalk interview with Paul Graham. Finding it is left as an excercise for the reader, primarily to encourage people to check out this amazing resource.

</comment><comment> daniel-cussen It's interesting to see more mainstream media outlets (npr and This American Life (public radio?)) talk more about patent trolling lately. As of a few weeks ago lay people I talked to had no idea programmers generally disliked patents.

</comment><comment><USER36> For some reason, every time I read the name Intellectual Ventures I sound it in my mind as Intellectual Vultures.

</comment><comment><USER37> In the audio version, they mention several times that patent law is founded in the constitution. I wish they would actually read it though, as it's not very complicated:

 _The Congress shall have Power To [...] promote the Progress of Science and useful Arts, by securing for limited Times to Authors and Inventors the exclusive Right to their respective Writings and Discoveries._

<comment> loup-vaillant Oddly enough, that is not enough to make patent law mandatory by the constitution: the text you cite has the structure _"The congress shall have Power To […] promote X by using the mean Y"_.

First, it is not written that the congress _has_ to exercise that power.

Second, what if Y does not promote X? What if it _hinders_ X? It is written that the congress is allowed to _promote_ X (through Y), not doing nothing about X (through Y) and certainly not hinder X (through Y).

I understand how one might assume that a patent system is a good (best? only?) way to promote the Progress of Science and useful Arts, especially back then. Nevertheless, I think this was a mistake: they wrote the _mean_ in the same stone they did the _goal_. (Of course, goals themselves can turn out to be means to higher purposes, but in this case it is quite obvious from the beginning.)

</comment></comment><comment><USER38> Is this sort of thing an issue in other countries. I'm a tech founder in Australia. I don't think we have the same level of anxiety about being sued for unknowingly breaching a software patent. Can anyone comment on this? Is our patent system different or can we expect to see our industry similarly affected in the future?

<comment><USER39> Patents are territorial. That is, unless someone has filed an application (or designated Australia as part of a national phase of a Patent Cooperation Treaty (PCT) patent application, nothing will issue in Australia. Generally, companies try to figure out where their likely markets are, and where the expense can be justified. Australia is becoming more and more relevant, but it was not nearly as common to file there a few years ago. (Full disclosure, I'm a U.S. patent attorney (but I'm actually a good guy!).)

<comment><USER40> That's a bit reassuring to me. I'm not anyone special, but I've got some ideas I want to code up that likely won't get anyones attention. After seeing patent topics popping up here on HN so often, I'm seriously terrified of even being a programmer anymore. This stuff makes me want to give up and find a job digging ditches or something, Office Space style.

But, I'm Canadian. Your comment makes me feel a bit safer, but I have the feeling that little things like hosting the site in the US or collecting money from Google AdSense or an Amazon Affiliates account could make me horribly liable.

I know that rationally this practically a non-issue, since I'm such a low value target, but the fear is still there, lurking beneath the surface, scaring the hell out of me.

</comment></comment></comment><comment><USER41> it's time this issue got more exposure than just the tech world... reporters aren't worried about their company being sued out from under them.

</comment><comment><USER42> The solution is very simple. Create a simple how to patent template website, then have every developer apply for patents on any business and software process that they can dream of. Within two years you'll choke the entire system into change. Make it impossible for anyone to do anything.

1st idea, the collection, aggregation, and transmittal of electronic messaging data during a waste evacuation process. (i.e. checking your email while on the toilet)

<comment><USER43> To the extent that the system is choked it means that the Patent Office grants patents without having the time to look at them, not that a backlog develops. You might notice that this has already happened.

<comment><USER42> of course that is happening. Even worse is that the people examining the patents are not best and under qualified to review software patents.

</comment></comment></comment><comment><USER44> Can someone please tell me what Myrvolhd actually DID when he was leading MS Research? I remember no innovations from MS in that era.

<comment><USER45> He wrote long memos and famously pooh-poohed the internet.

<http://www.kenauletta.com/themicrosoftprovocateur.html>

Correction: this was before he was sent (banished, the article makes it seem) to Microsoft Research.

</comment></comment><comment><USER46> I actually just heard this story over NPR Radio. It's concerning to think about how companies like this can strong-arm smaller startups. But at the same time that doesn't mean that we should hide in fear. We've got to keep on doing what we do best: developing innovative products and solutions for users everywhere. Torpedoes be damned.

</comment><comment><USER47> Serious question: can/do patent trolls go after open source projects? I'm not thinking of businesses who sell/support open source software (e.g. Red Hat), but would it be a viable "defense" for an indie dev to simply release their project as open source? I've got an idea for a "weekend project" or two that I could conceivably make a few bucks with, or I could release them as open source so that I can at least benefit from the idea without worrying (as much?) about being sued for violating a patent. On the one hand if I'm not making any money from it one would think that the trolls would go elsewhere, but then again...

Just looking for a general idea here, I'm not looking for official legal advice here... :-)

<comment><USER48> The Debian patent FAQ is pretty good for this: <http://www.debian.org/reports/patent-faq>

<comment><USER47> Very useful - thanks for the link!

</comment></comment></comment><comment><USER9> If you're looking for an organization which is actively working to fix the software patent problem, you can look to End Software Patents: <http://endsoftpatents.org/>

</comment><comment><USER49> What if you could pay for some kind of insurance product to protect against patent lawsuits? Or perhaps some kind of co-operative that owned a bunch of bullshit patents to be used for defensive purposes in such a scenario?

<comment><USER50> >What if you could pay for some kind of insurance product to protect against patent lawsuits?

Heh. Programmers' malpractice insurance. The premiums would be ridiculous, though.

>Or perhaps some kind of co-operative that owned a bunch of bullshit patents to be used for defensive purposes in such a scenario?

That's more or less the shitck Intellectual Ventures uses to justify that they're not patent trolls (they point it out in TFA). In the podcast they point out that their need for a good ROI doesn't allow them to sit idle.

How do you defend against patent trolls, though? It's not like they make anything that can be infringed on.

Maybe here's a valid, easy-to-swallow change to the patent system - <USER2> a patent changes hands you halve the amount of time it is valid for.

<comment><USER51> > Maybe here's a valid, easy-to-swallow change to the patent system - > everytime a patent changes hands you halve the amount of time it is valid > for.

Nice idea!

> It's not like they make anything that can be infringed on.

Or, require the patent holder to prove 'damages'. Since they don't make anything and don't actually _use_ their patents there are no damages lost.

</comment></comment></comment><comment><USER52> Well, this was timely. <http://news.ycombinator.com/item?id=2808110>

</comment><comment><USER53> Is it fair to call Intellectual Ventures an arms dealer.

</comment><comment><USER54> I wonder why people waste their time and money on defending themselves in court, why don't they hire Tony Soprano to fix their problem instead? It would be a pretty appropriate response.

</comment><comment><USER55> Poll: What's you take on Intellectual Ventures with regard to innovation? <http://wepolls.com/1510009>

</comment><comment><USER56> Admittedly I stopped reading the article when I got to "IV isn't a patent troll, we're simply just a big patent market connecting suppliers to consumers"

No, you're not simply a 'market'. A market doesn't come and sue you for buying your milk somewhere else.

<comment><USER57> You might want to finish the article.

<comment><USER56> I get that the article is against the patent process, and the downmodders don't seem to get that I was referring to IV's characterisation of itself, not npr's characterisation of IV - I thought it was clear where I was directing it due to the phrase " _you're_ not a market".

I wasn't using 'patent troll' or any of the other terminology. I was pointing out a flaw in this guy's reasoning. Yes, I had only read 500 words into a 4000 word article. I'm now about halfway through and have scanned the rest of the article... and I don't see anything that invalidates my claim above. IV is _not_ the 'market' that it says it is. If anything, I am _supporting_ what npr is saying.

HN moderation is _weird_.

<comment><USER58> The subject of article is more or less how IV's reasoning is flawed. Picking a quote from the beginning and saying "that is flawed" is just echoing the bulk of the article. I would guess your downmodders think you're not adding to the discussion.

<comment><USER56> I don't buy that argument - why aren't many of the other short comments in this page getting the same treatment when they're not adding to the discussion either?

<comment><USER59> If you had cut out the "Admittedly I stopped reading the article when I got to" part of your comment, you probably wouldn't have been downvoted. I think the general perception is that it's difficult to add to the discussion of an article if you didn't even bother to read that article, especially when your comment merely echoed the article.

<comment><USER56> I, too, think it is that particular phrase, and I disagree with the sentiment of downmodding this kind of admission instead of actual content.

I've seen this elsewhere - if someone says they're being sarcastic, they're downmodded to hell... but people in the same thread saying the same thing in the same sarcastic manner, but without the admission, are left alone.

I find this inconsistency annoying - it appears that downmod-friendly HNers are more interested in making a pretense to politeness than actually moderating content. It only takes a single dissenter and your words are already diminished on the page.</comment></comment></comment></comment></comment></comment></comment>


Single line of code cause of London air traffic chaos - <AUTHOR> http://www.bbc.co.uk/news/uk-30463551

<comment><USER2> it usually is the smaller things that get overlooked. doesn't surprise me

</comment><comment><USER3> abort(); // TODO: make sure this doesn't get into production

</comment><comment><USER4> while(planes); { ... }</comment>


Meow Hash: A high-speed non-cryptographic hash function - <AUTHOR> https://mollyrocket.com/meowhash <comment><USER2> This is a good looking hash, just from the construction I can see that it should be extremely fast for bulk hashing of large objects and with strong statistical properties.

I do want to comment for the wider audience why the details of this construction does not lend itself to small keys, and how you would fix that. I have an (unpublished, I'm lazy) AES-based hash function that is very fast for both bulk hashing and small keys, which required a mostly performance neutral (likely fits in an unused ALU execution port) tweak to the bulk hashing loop.

Large key performance is entirely in the bulk loop algorithm (an AES round in this case) and small key performance is entirely in the mixer/finalizer algorithm. If you have a very wide bulk loops with no mixing of the individual lanes then it usually requires a deep mixing stage that makes small keys very expensive since that is a fixed overhead. The question then becomes, how do you cheaply "bank" dispersal of bits across lanes in the bulk loop to shorten the mixing stage _without_ creating an operation dependency between lanes that will massively reduce throughput.

As an important observation, vectorizing the AES lanes makes it difficult to disperse bits because there are no good, cheap operations that move bits between lanes. However, the CPU will happily run multiple AES operations and other 128-bit operations concurrently across ALU ports if they are in independent registers. This allows you to trivially create a "mixing" lane that solely exists to aid bit dispersal because you no longer are trying to do sideways operations on a vector. So what does a mixing lane look like that doesn't create dependencies on your hashing lanes? It is so simple it is embarrassing: XOR the input data lanes. The mixing lane is worthless as a hash but it only exists to represent all the bits from all the lanes.

At the mixing stage, you simply fold the mixing lane into each of the hashing lanes by running it through a round of AES -- one concurrent operation. This allows you to avoid the deep mixing stage altogether, which means small keys can be hashed very quickly since almost all of the computation is in the finalizer. Using this technique, I've been able to hash small keys in <30 cycles with concurrent AES lanes.

<comment><USER2> As one more comment, because I've made the same mistake in the past, it is often a bad idea to use the length in the initialization vector of a general purpose hash because it means you can't make an incremental version of the hashing algorithm e.g. a stream of data of unknown size. This is a trivial change but it has side effects in how you pad partial blocks.

If the length is not part of the initialization vector, then the padding vector for partial blocks should vary as a function of partial data length. Otherwise it is trivial to create collisions by having partial blocks of different sizes that mimic the padding vector. Fixing this can be as easy as using _mm_set1_epi8(partial_length) as your padding vector.

<comment><USER3> I'm a fan of your work; I've personally mangled your Metrohash64 in some of my own work for incremental updates.

Any chance you're thinking about releasing the program for generating new (good) Metrohash constants + shifts?

<comment><USER2> I have a completely new algorithm construction I have been using for a while that I am releasing any day now -- I was literally working on the documentation this weekend. It is a substantial improvement on the original algorithms in almost every regard and also easier to customize.

<comment><USER3> I look forward to the release, and I know the feeling (writing docs, cleaning up comments, etc.). I've been perpetually ~2 weeks from commercial release of my own product since July or so.

</comment></comment></comment></comment></comment><comment><USER4> _To our surprise, we found a lack of published, well-optimized, large-data hash functions. Most hash work seems to focus on small input sizes (for things like dictionary lookup) or on cryptographic quality. We wanted the fastest possible hash that would be collision-free in practice (like SHA-1 was), and we didn't need any cryptograhic [sic] security._

This is an important passage that could be explained in more detail. I can look up a list of non-cryptographic hash functions, nine of which support 128-bit output:

[https://en.wikipedia.org/wiki/List_of_hash_functions#Non- cry...](https://en.wikipedia.org/wiki/List_of_hash_functions#Non- cryptographic_hash_functions)

Given that the claim is that Meow Hash is fast, where's the benchmarking to prove it's faster than the alternatives?

<comment><USER5> 16 bytes per cycle is what they quote. Imo, that's pretty impressive.

For comparison, xxhash is the closest thing to this that I'm aware of and clocks in at around 1.7 bytes per cycle using the official implementation.

<comment><USER6> There are variants of CityHash that use the x86 CRC intrinsics (CityHashCrc128 and CityHashCrc256). They quote 4.3-5.5 bytes/cycle.[0]

(Better than xxhash on x86, worse than Meowhash -- assuming the bytes/cycle numbers can be compared apples-to-apples, which is a big if.)

[0]: [https://github.com/google/cityhash](https://github.com/google/cityhash)

<comment><USER5> I think the big issue with bytes/cycle comparisons right now is that once you get to multiple bytes per cycle, most people are going to be limited on I/O, even to RAM probably. I assume Meow Hash was designed to run on high end workstations, but if you are developing software for end users my guess is that Meow Hash will end up being not significantly faster than xxHash.

Of course, if you are running a lot of hashes in parallel, I bet things get more interesting. (Example: would AES-NI instructions scale in parallel as fast as ordinary bitwise/vector instructions?)

<comment><USER7> ram throughput isn't that different on a cheap to high end pc.

<comment><USER5> When I say high end workstation I really do mean workstation and not gaming PC. Most people do not have quad channel memory on their home PCs but it is in fact a pretty good step up in throughput.

</comment></comment></comment></comment></comment></comment><comment><USER3> Late to the party, some actual benchmarks using an older CPU with AES-NI: Intel(R) Xeon(R) CPU E5-2670 0 @ 2.60GHz bugs : cpu_meltdown spectre_v1 spectre_v2 spec_store_bypass l1tf Kernel 4.18.14 GCC 7.3.0

Short version, it's ~2x as fast as anything else on my machine for data > 4k, maybe down to 1k, all the way up to > cache (where various other algorithms also max out memory bandwidth). Other non-cryptographic hashes (for comparison) on my local machine: xxHash64 -> ~13G/s, Metrohash128 -> ~13G/s, Metrohash128_crc -> ~18G/s, t1ha_crc -> ~17.5G/s, etc., but Meow doesn't break the 1G/s barrier until 64-96 bytes (the others mentioned reach that speed at 8 bytes).

So yeah, faster for longer keys due to the fast aes mixing + parallel pipeline usage, but slower for shorter keys due to the final mixing dependency at the end.

Using a hacked-up version of Smhasher for wider range of hashing sizes (cache on this CPU is 20 megs):

        --- Testing Meow1_64 "parallel aes internals"

        [[[ Speed Tests ]]]

        Bulk speed test - 67108864-byte keys
        Average      -  4.556 bytes/cycle - 14337.09 MiB/sec @ 3 ghz
        Bulk speed test - 16777216-byte keys
        Average      -  7.882 bytes/cycle - 24805.52 MiB/sec @ 3 ghz
        Bulk speed test - 4194304-byte keys
        Average      -  9.813 bytes/cycle - 30884.30 MiB/sec @ 3 ghz
        Bulk speed test - 1048576-byte keys
        Average      -  9.657 bytes/cycle - 30390.48 MiB/sec @ 3 ghz
        Bulk speed test - 262144-byte keys
        Average      - 10.642 bytes/cycle - 33490.19 MiB/sec @ 3 ghz
        Bulk speed test - 65536-byte keys
        Average      - 11.871 bytes/cycle - 37360.76 MiB/sec @ 3 ghz
        Bulk speed test - 16384-byte keys
        Average      - 11.336 bytes/cycle - 35675.98 MiB/sec @ 3 ghz
        Bulk speed test - 4096-byte keys
        Average      -  8.241 bytes/cycle - 25936.55 MiB/sec @ 3 ghz
        Bulk speed test - 1024-byte keys
        Average      -  4.056 bytes/cycle - 12765.26 MiB/sec @ 3 ghz

</comment><comment><USER8> I just ran a simple collision test against it with real life, prod data. It resulted in 13 collisions on 116275 files in 512b mode.

[https://github.com/cmuratori/meow_hash/issues/7#issuecomment...](https://github.com/cmuratori/meow_hash/issues/7#issuecomment-431657191)

<comment><USER9> Given the insanely high rate of collisions for a 512 bit hash, the "we probably won't update the main loop" response is quite something.

</comment></comment><comment><USER10> How does it compare to:

FarmHash

CityHash

SMHasher

MetroHash

FNV1a_YT

Note that if you're just using it for a hash table, where you'll compare the full strings to resolve hash collisions, you don't actually need good randomness. FNV1A_VT is faster than most, has worse collision properties, yet provides a faster overall hash table, because the faster hash (every operation) outweighs the extra full string checks (rare).

[https://github.com/rurban/smhasher](https://github.com/rurban/smhasher)

<comment><USER6> FNV has terrible hash properties (i.e. it fails many smhasher tests as mentioned in TFA), so it isn't really comparable.

SMHasher isn't a hash function? It's a test suite for hash functions.

CityHash's CRC modes are nominally 4.3-5.5 bytes/cycle.

The others are slower.

This article quotes 16 b/c for Meowhash.

<comment><USER11> Yup FNV is terrible from my experience using it in a custom hashmap implementation. The lower bits of the FNV hash have a lot of bias.

</comment></comment></comment><comment><USER12> Yep, unrolled AES instructions are the way to go if you need to hash gigs of stuff and you're on a CPU that supports them.

-Austin, SMHasher author.

</comment><comment><USER13> Never heard of these guys, but they got several hundred hours of video footage of building a game from scratch: [https://www.youtube.com/user/handmadeheroarchive/videos](https://www.youtube.com/user/handmadeheroarchive/videos) looks interesting.

</comment><comment><USER14> I wonder if speed will still be a reason for doing something like this going forward now that Intel is starting to include SHA acceleration instructions in its processors.

<comment><USER6> It will, for a couple reasons. The first is, Intel doesn't include the SHA intrinsics in their high end consumer processors. Only the low end Atom stuff so far.

The second is that even with the intrinsics, this is (supposedly) faster. The SHA instructions only get you a few gigabytes a second (single digits). This purportedly gets tens of GB/s.

<comment><USER15> As far as I'm aware, that isn't true. Skylake added support for 7 different SHA extensions:

[https://www.phoronix.com/scan.php?page=news_item&px=Linux-4....](https://www.phoronix.com/scan.php?page=news_item&px=Linux-4.4-Crypto)

<comment><USER16> Phoronix is, once again, wrong. Skylake does not support the SHA instructions. The only currently shipping x86 CPUs which support them are Intel Goldmont (Atom) and AMD Zen (Ryzen, Threadripper).

Mainstream support for SHA in Intel CPUs will come with Cannon Lake, or whatever the next microarchitecture revision they release ends up being.

* Yes, Cannon Lake is technically shipping as the i3-8121U, but that isn't really relevant to anyone here.

<comment><USER15> Ok, it seems you're correct about Cannon Lake:

[https://software.intel.com/sites/default/files/managed/c5/15...](https://software.intel.com/sites/default/files/managed/c5/15/architecture- instruction-set-extensions-programming-reference.pdf)

Cannon Lake and later according to Intel.

However, I question "mainstream" since many Celeron and Pentium branded parts featured SHA-NI, and those are arguably mainstream, low-cost parts.

Also, in fairness to Phoronix, it looks like SHA-NI support may have been planned for Skylake, but didn't make it based on various posts I see online as early as 2014 and Intel's first postings about SHA-NI on Goldmont, etc. in 2013.

<comment><USER6> > many Celeron and Pentium branded parts featured SHA-NI

Do you have a source for this claim?

> Also, in fairness to Phoronix, it looks like SHA-NI support may have been > planned for Skylake

Yeah, but on the other hand that article was published 4 months _after_ the general availability of Skylake. It was possible for Larabel to verify the facts before publishing and he just didn't. Nor did he correct the mistake in any of the past three years.

<comment><USER15> Sure, quite a few processors were based on Goldmont:

[https://en.m.wikipedia.org/wiki/Goldmont](https://en.m.wikipedia.org/wiki/Goldmont)

</comment></comment></comment></comment></comment></comment></comment><comment><USER17> [https://www.reddit.com/r/programming/comments/9pr5ir/meow_ha...](https://www.reddit.com/r/programming/comments/9pr5ir/meow_hash_a_noncryptographic_hash_capable_of_16/)

</comment><comment><USER18> I just happen to need a fast, non-cryptographic checksum for large amount of data. I assumed using CRC32C would be the fastest thing available on x86_64 since SSE 4.2 includes an intrinsic for it.

Can I expect Meow Hash being faster than that on x86_64? What about on some other platforms such as common ARM chips? Which one is more collision resistant?

Also: the writer seems to be Casey Muratori, who is also the author of the Handmade Hero series! Kudos.

<comment><USER18> OK, after looking into it for just a few minutes, I found out: 1) CRC32C is not nearly as collision-resistant as many common fast non-cryptographical hashes. 2) there are many portable hash functions that are faster (and I think this is due to their bit size.) See: [https://github.com/Cyan4973/xxHash](https://github.com/Cyan4973/xxHash) and [https://github.com/leo-yuriev/t1ha](https://github.com/leo-yuriev/t1ha)

So, my initial assumption was wrong. I'll be trying Meow Hash and some others out!

<comment><USER6> Meowhash claims to be faster (16B/cycle) than CityHashCrc (4.3-5.5 B/cycle), which is a variant of CityHash (ab)using the x86 CRC32C intrinsics (but with much better hash properties than plain CRC32).

Both are faster than xxHash on machines with AESNI and CRC intrinsics (~1.7B/cycle). The advantage of xxHash is that it is very fast on machines _without_ these instrinsics.

Other hashes tend to be slower although I'm sure there are some other intrinsics-based hashes I don't have at the tip of my tongue.

</comment></comment></comment><comment><USER19> Will this hash be useful to check if the content of files of all sizes has changed or not? Thinking about storing the hash of a file in a database to evaluate when changes has taken place.

</comment><comment> man-and-laptop [https://github.com/cmuratori/meow_hash/blob/master/meow_hash...](https://github.com/cmuratori/meow_hash/blob/master/meow_hash.h)

What do AESRotate(foo,bar) and AESMerge(foo,bar) do?

[edit]

So it's a simple cyclic rotation preceded by A.Q0 = _mm512_aesdec_epi128(A.Q0, B.Q0)

From Googling, _mm512_aesdec_epi128 looks like an Intel processor built-in.

</comment><comment><USER20> > Don't use the Meow hash for tiny inputs. The minimum block size for the Meow > hash is 256 bytes, so if you're feeding it 8 byte data, 16 byte data, etc., > it's going to waste all of its time padding the input

This seems like an important caveat that means it will only be used in specialized applications.

<comment><USER21> It's meant for file hashing and it's rare in the real world that files are less than 256 bytes.

</comment></comment><comment><USER22> I think releasing it on yet-another-open-source-license instead of some of well-known ones severely degrades the possibility for this code to be used by companies.

<comment><USER23> It's the zlib license, which is used by GLFW, SDL, and SFML as well. Companies seem to have no problem using things like SDL, so I think this is fine.

Still, it would be nice is to state that is is the zlib license, and to include the license in the repository.

<comment><USER6> And it looks more or less fundamentally identical to the 3-clause BSD license.

<comment><USER24> The zlib license predates 3-clause BSD (zlib was released under the zlib license in 1995, BSD started releasing code under 3-clause BSD in 1999).

I will admit that when I first saw the zlib license, I was similarly confused but it's actually a license that is about the same age as most other common licenses (and quite a large body of software is licensed under it).

<comment><USER6> The 3-clause BSDL is just the 4-clause with clause #3 removed. Per Wikipedia, the 4-clause dates to 1990: [https://en.wikipedia.org/wiki/BSD_licenses#4-clause_license_...](https://en.wikipedia.org/wiki/BSD_licenses#4-clause_license_\(original_%22BSD_License%22\)) . (I.e., all of the text in the 3-clause BSD license was written in 1990.)

<comment><USER24> No software was licensed under the 3-clause BSD license (because it didn't exist) until 1999. Just because the text is similar to an earlier license doesn't change that software licensed under zlib precedes software licensed under 3-clause BSD.

But that's not really the point -- GGGP implied that the license was written from scratch ("yet-another-open-source-license") and that's obviously not true.

</comment></comment></comment></comment></comment></comment><comment><USER25> I wonder if a cryptographer could take a look at Meow Hash. I'm curious how it would fail first when subjected to modern cryptanalytic methods.

</comment><comment><USER26> How fast is it on other platforms, e.g. ARM?

<comment><USER27> Currently, only an x64 implementation is provided. "While it can be implemented on ARM processors which have AES instructions, we have not yet created a reference implementation for that platform."

</comment></comment><comment><USER28> Not even a mention of MD5?

<comment><USER29> MD5 is a (bad) cryptographic hash and therefore has entirely different goals and properties as this hash function.

<comment><USER28> MeowHash is also a bad cryptographic hash. Can we compare properties instead of intentions? MD5 is faster than SHA. Is MeowHash faster than MD5 or not?

<comment><USER30> > MeowHash is also a bad cryptographic hash.

MeowHash is not cryptographic at all.

> Is MeowHash faster than MD5 or not?

Meowhash is advertised as 16 bytes per cycle. MD5 is around 5 cycles per byte[0] or _0.2 bytes per cycle_.

Pre-existing fast non-cryptographic hashes apparently top out around 5.5 bytes per cycle (CityHashCRC256).

[0] [https://keccak.team/sw_performance.html](https://keccak.team/sw_performance.html)

<comment><USER28> Thank you. The comparison of MeowHash to SHA without mentioning MD5 or any actual performance numbers didn't make sense.</comment></comment></comment></comment></comment>


LG's new e-book reader is solar-powered - <AUTHOR> http://www.guardian.co.uk/environment/2009/oct/12/kindle-lg-ebook-solar

<comment><USER2> Looks like a great idea for the beach!

This kind of low-power application is GREAT for solar panels. I'm sick of people trying to put them on everything (like electric cars... do you know how impractical it would be to put them on a Tesla based on their current efficiency and cost?)

</comment><comment><USER3> Awesome. This makes a lot of sense. An e-book reader should always have lots of surface area (since you don't want to read on a tiny screen) and requires very little power. I'm almost surprised e-book readers weren't built this way in the first place.</comment>


Machine Learning Guides - <AUTHOR> https://developers.google.com/machine-learning/guides/ <comment><USER2> Unlike most guides I've seen about ML, this one does a good job of focusing on developing and deploying a simple model first, then iterating. There are also lot of practical tips here, especially around feature engineering.

> the second phase of machine learning involves pulling in as many features as > possible and combining them in intuitive ways. During this phase, all of the > metrics should still be rising

As Google points out, after you build an initial model, the next step to increase accuracy is to perform feature engineering. They explain that this can be done manually or automatically using something like deep learning. Another option that people here might consider is using a library like Featuretools ([https://github.com/featuretools/featuretools](https://github.com/featuretools/featuretools)) for "automated feature engineering". Note: I am one of the developers.

Our goal is to help you increase the performance of your models without sacrificing the interoperability of your features. We have a post up about how our algorithm works here: [https://www.featurelabs.com/blog/deep-feature- synthesis/](https://www.featurelabs.com/blog/deep-feature-synthesis/). There are also plenty of real world demos on our website: [https://www.featuretools.com/demos](https://www.featuretools.com/demos)

</comment><comment><USER3> Your article is cool. I have some things about information of machine learning. just check it out [http://tricks4321.blogspot.com/2018/07/machine- language-for-...](http://tricks4321.blogspot.com/2018/07/machine-language-for- music.html)

</comment><comment><USER4> One of my random question is that what does Google gain by spending resources on developing course like this? Do they want more people to do machine learning as there is a short age of developer with this skill in the market or is there something else involved in the mix?

Secondly, for some reason data science just doesn't excite me as much as typical software development goes. Like, why am I not excited enough to go down the path of specializing in data science in field of machine learning? Even if there is more money in it, I'm still not extremely motivated to learn it.

What i do particularly enjoy is good ol' back end web development. I don't have a degree in computer science but working on a information system degree with focus on "programming", I dream/working my ass to become cult of "software engineer" type II, a sophisticated software developer/programmer. I love building layers, optimizing code, learning new tools, algorithms data structure (without knowing math), creating unit tests, following programming paradigm. It excites me so much. And my core skills to dive into is block chain.. I love studying that topic too and all the algorithms it comes with it.

But when I see data science, no excitement. All I imagine is image manipulation and fancy charts. I know I sound a bit ignorant but, that's how it is.

<comment><USER5> Great and honest points.

>Secondly, for some reason data science just doesn't excite me as much as typical software development goes

Fair enough. Part of the reason is "data science" has been so jammed pack of nonsense and people who don't do the actual work of building things, as you describe below.

> What i do particularly enjoy is good ol' back end web development. I don't > have a degree in computer science but working on a information system degree > with focus on "programming", I dream/working my ass to become cult of > "software engineer" type II, a sophisticated software developer/programmer. > I love building layers, optimizing code, learning new tools, algorithms data > structure (without knowing math), creating unit tests, following programming > paradigm. It excites me so much. And my core skills to dive into is block > chain..

Ok this makes sense. But I'd be worried about 5 years from now. When all the little gears and things that go on in backend becomes a commodity (or abstracted away in the "cloud"), what are you going to do?

> I love studying that topic too and all the algorithms it comes with it.

That spark of interest in the algorithms, (which is just about logic, which is what math is basically about in the end), is basically the essence of what makes "Data science" so attractive.

<comment><USER6> "But I'd be worried about 5 years from now. When all the little gears and things that go on in backend becomes a commodity (or abstracted away in the "cloud"), what are you going to do?"

Well, over the last 8 years or so I started out in a similar kind of place, and have gotten quite good at building CRUD and business logic and glue, and fixing crap on the front end, and configuring servers.

Maybe I can stand in for the OP a few years down the line?

Over the last quarter, I've been splitting my time between things like linux admin automation and a set of pre-calculus core classes.

To answer your question on my personal scale, my whole ability to do this kind of work with my mediocre CS education (my BA is in Philosophy, and my PhD work is in Lit) is premised on leveraging the points in the systems where "all the little gears and things that go on in backend have [become] a commodity"... hence I just integrate ERP systems with WordPress or try and clean up some business's AWS drupal hosting setup some crap like that. That's been a fun and rewarding conjunction of my love for systems and the commodification of parts of IT/ programming work.

My hope is that by the time all the little bits of these data science topics become "abstracted away" over the next couple of years, I will understand the general underlying things well enough to use them. But who knows if that is a good bet or not... certainly not me.

However, it feels perfectly fine to learn things like math... I'm way, way better at it than I was as an undergrad 20 years ago and so it's quite a lot more fun for me. It's not like knowing some math has no application outside of this narrow field.

I dunno if my personal answer (keep learning, and enjoy fixing crap) matches the OP or helps extend your points/ question, but I've been getting a lot of fun (and some money) out of following my answer.

</comment></comment></comment><comment><USER7> Perhaps it's somewhat off-topic, but I've built a spam detector similar to the article's withOUT using "direct" AI, but rather via a key-word or key-phrase "ranker". A simplified example is given below.

The advantage over other techniques is that one can easily trace the exact math of a conclusion, and tune it as needed. The disadvantage is that one probably has to manually tune it all rather than let the machine "learn". However, a hybrid approach could be used whereby "pure" AI suggests words and phrases to encode.

         rule.addList("nigerian, prince", rank=7);
         rule.addPhrase("great opportunity", rank=5);
         rule.addPhrase("lisa smith", rank = -4); // probably good

Here a "list" means that the word order doesn't matter, but with a "phrase" it does matter. A negative value means its less likely to be spam, usually because it's specific to your business or task. Actually I had multiple categories rather than just "spam" versus "non-spam", but that would complicate the example. I also used a database. One could perhaps call it a "weighted" version of MS-Outlook's rule engine. Somebody had a similar idea: [http://dergipark.gov.tr/download/article- file/45302](http://dergipark.gov.tr/download/article-file/45302)

<comment><USER8> You're essentially doing a rough manual version of Bayesian classification on n-grams (which is still very explicable): [http://www.paulgraham.com/spam.html](http://www.paulgraham.com/spam.html)

<comment><USER7> The idea of my approach was that a "power user" could add the rules and scores without having to understand something that may take a while to explain. A scoring sheet can be displayed for a given message that would make sense to just about anybody with an associate degree. Example scoring sheet for a given message:

         Category: Spam
           Rule-ID    Score
           </comment></comment></comment><comment></comment><comment>----
           NgrPrnc1       7
           bPills         5
           knownPeople   -3
             Total:       9 Threshold Exceeded!

         Category: Tech Support
           knownWidgets   3
           offer1        -2
             Total:       1 Insufficient total

         Category: Etc...

One could click on the rule-ID as a hyperlink to see specifics of a given rule (if details don't fit on screen).

<comment><USER9> This is how people did things back in the time. "Expert systems" with hand- crafted rules, built by "experts".

From the past, we learn that these systems are brittle and break continuously. For example, what happens when spammers start using different words, or send legitimate looking emails that are actually spam? Do you think you can build rules to catch 70%, 80%, 90% or 99.99% of spam?

If your goal is simply showing the rules being applied, you can still learn the rules with ML but display them in this way (for example GP suggested looking at Naive Bayes which was the most common method used to fight spam; I'd also point you to decision trees which are easy to visualize).

<comment><USER7> As stated, it wasn't intended for an "expert", but a power user. Somebody has to make the decision anyhow of spam versus non-spam in order to make a training set for "learning" based AI. These days you can purchase spam detection systems/services such that training such systems in-house is usually not worth it. They can use rejected messages from thousands of orgs to train their system.

But what I described had additional purposes such as sub-routing to various departments. It was a multi-purpose email categorizer in the early days of spam. Each approach has trade-offs. I'm not sure how you'd apply a "decision tree" using weights in a way that makes sense to a power user. A non-weighted decision tree seems too blunt an instrument. One generally needs multiple "clues" (factors) voting in tandem.

</comment></comment></comment><comment><USER10> These guides also give good heuristics on how to look at data _before_ throwing a model at it, and deciding what's the most _logical_ model approach/architecture.

A good example is the text preprocessing flowchart (also shared by fchollet on Twitter): [https://developers.google.com/machine- learning/guides/text-c...](https://developers.google.com/machine- learning/guides/text-classification/step-2-5)

<comment><USER11> This is something that's almost always glossed over and I'm glad they included it. It's easy to apply ML algorithms to perfect data that doesn't need cleaning and get great results. Finding a productive model when presented with a nuanced, messy problem is a much more difficult task, however, and something most ML crash courses don't focus enough time on.

I think there's a tendency on Hacker News and other tech websites to diminish the importance of having a PhD in ML fields. The problem solving and communication skills you learn during the course of a PhD program are precisely the skills companies value when they're trying to solve hard problems. It's important to know not just how to apply ML algorithms, but when they're appropriate.

</comment></comment><comment><USER12> Back in 2006, in highschool, I was investigating multilayer feed-forward NNs. I found them magical. I wrote the XOR problem etc. etc.

What always confounded me was the choice of the number and width of hidden layers. This is even now more confusing with the advent of deep and recursive networks. We need empirical work on this, that can be taught in much the same way that gravity is taught as an apple falling from a tree.

We need a determination of the entropy of a network, how to route that entropy and expolit it. Specific scenarios are not adequate.

<comment><USER13> > gravity is taught as an apple falls from a tree.

Is this more advocating for a theory of neural networks rather than empirical evidence?

</comment></comment><comment><USER14> These guides pop up left and right, lately. I can't comment on their quality (I assume it's somewhat decent) but it's kinda ridiculous to try compressing a college degree's worth of knowledge into a bunch of sleek online tutorials.

<comment><USER15> > it's kinda ridiculous to try compressing a college degree's worth of > knowledge into a bunch of sleek online tutorials.

Honest question, why?

We used to give degrees (albeit hundreds of years ago) for material that now is covered, at a high level, in a single course (e.g. physical sciences). The amount of material to cover, and to master, increases dramatically over time. It makes sense to compress the knowledge to be delivered to a compendium so as to simply keep up with progress.

<comment><USER16> Not OP but I suspect they will say something about the math behind it. It's very true you can get quite adept at plug-and-play machine learning models (and indeed be quite successful) but the theoretical statistics, linear algebra and overall mathematical maturity take a long time to develop in my opinion.

<comment><USER17> Yep, the great thing about tensorflow are the canned algorithms. You can focus on your data-sets and problem rather than deep theory.

</comment></comment></comment></comment><comment><USER17> Good guides, just finished the text classification one. The approach is very much grow a good dataset and then find and tune a model the works well for your needs.

</comment><comment><USER18> Rule #0: choose first principles over machine learning.

ML is a last-resort for problems you don't understand. There are lots of these, but understanding the problem is better.

<comment><USER19> Yes and no. When I first started getting into this stuff I was amazed at the difference good feature selection and feature engineering made and I was meticulous about it. But in many real world cases now, an expert human can't do it as well as quickly as a DNN running on multiple K80s. It all boils down to economics in the end.

</comment></comment><comment><USER20> Deep learning and blockchain noise seems to be dying out slowly. Are we going downhill on the hype curve, is it time to short those buzzwords?

<comment><USER21> Deep learning is far from slow down. The research speed has increased a lot in the last 5 years. It amazes me how fast the researchers are able to come up with new things. From one year to the next one there is something shiny and new.

I don't follow very much blockchain but seems like it was 6 years ago

</comment></comment><comment><USER22> why would every ML guide use cats as example

<comment><USER23> I know right! It should be about hotdogs instead.

</comment></comment><comment><USER24> Daily reminder for data scientist and machine learning types: fill your pockets while you can, because machine learning bootcamps are on the horizon!

<comment><USER10> There's been data science/machine learning bootcamps around for awhile (Galvanize/Metis being common examples in San Francisco), but apparently job placement is not in a good place (as with normal bootcamps).

Indeed Machine Learning/Deep Learning has become much more accessible thanks to the number of free guides such as this. But that means data science job placement will become _more difficult_ as competition increases, with more gatekeeping/requirements (e.g. Masters/Ph.Ds)

<comment><USER25> Depends what the company is doing.

Most companies are going to utilise ML to some extent. Once technology and tooling improves they'll need boots on the ground engineers and not labs with R&D teams

<comment><USER10> Masters/PhDs can be boots-on-the-ground engineers too.</comment></comment></comment></comment>