Eric Schmidt: "If you have something [to hide], maybe you shouldn't be doing it" - <AUTHOR> http://www.theregister.co.uk/2009/12/07/schmidt_on_privacy/ <comment><USER5> I'd like Eric Schmidt to publicly release his search history, both his work- related and private searches.

<comment><USER6> What if his statement was not intended as a moral imperative, but as practical advice?

Perhaps he is saying, "The fact of the matter is that we're required to retain and release lots of information. If you don't want that information being passed around, you probably shouldn't give it to us. Sorry, we do the best we can under the circumstances."

"If you have something that you don't want anyone to know, maybe you shouldn't be _[googling]_ it in the first place."

<comment><USER5> Who's requiring them to retain anything? For 180 days?

And retaining for policing/security reasons doesn't mean you should use the data for other privacy-invading targeting.

<comment><USER7> Actually there's no law in the U.S. about data retention (E.U. has one) but companies keep the data for their own engineering and marketing purposes AND to stay in good graces of authorities. (The Patriot Act and ECPA, among other laws, specify what data has to be turned over with a lawful request, but they don't require any entity to actually have that info.)

</comment></comment></comment></comment><comment><USER8> Straight back at Eric Schmidt: "If you have to look at people's personal data to do something, maybe you shouldn't be doing it"

<comment><USER9> How would a search engine that doesn't look at your search string work?

<comment><USER10> Of course it needs your query string, but it could run your search and then forget about it. Arguably Google was better when it didn't take your history into account when calculating the results.

<comment><USER11> You certainly _could_ argue that, however, if it were true in most cases Google, a highly data-driven company, would be still be working that way.

</comment></comment></comment></comment><comment><USER12> I hope you're all pleased you jumped on the bandwagon. Here are his actual words from the video:

 _Judgement matters... If you have something that you don't want anyone to know, maybe you shouldn't be doing it in the first place... If you really need that kind of privacy, the reality is that search engines - including Google - do retain this information for some time and it's important, for example, that we are all subject in the United States to the Patriot Act and it is possible that all that information could be made available to the authorities._

Schmidt's statement is clearly pragmatic. One might even argue there's an implied criticism of laws like the Patriot Act. Could he have phrased it better? Yes. _He shouldn't have said the sentence in question._ But next time, actually read/watch the primary source so you can get the full picture. His closing remarks completely change the tone of the conversation. Are you really going to let The Register do your thinking for you?

</comment><comment><USER13> I'd like to hear Erich Schmidt's half of the story.

This is typical Internet drama. We have seen it a million times. I give the guy the benefit of doubt since this wouldn't be the first occasion some journalist taken a sentence out of context.

<comment><USER14> There is no other half. It's a quote taken completely out of context and blown out of proportion with absolutely moronic commentary. I can't believe this is the kind of thing that makes it to the front page of HN, much less to the #1 spot :(

 _But the bigger news may be that Schmidt has actually admitted there are cases where the search giant is forced to release your personal data._

Oh, you mean Google follows US laws? They don't participate in acts of civil disobedience in a suicidal effort to protect our privacy!? Eeeevil!

<comment><USER15> Try hackerhackernews.com. It's the same site, but uses some Bayesian modeling to filter out the TechCrunch-like internet drama.

<comment><USER16> Unfortunately, not regularly updated:

> Page generated at 17:55 on 15 October 2009 (PDT).

</comment></comment></comment></comment><comment><USER17> cool, so my friends that work at Google don't need to keep the details of their projects secret anymore?

<comment><USER18> or they shouldn't be working on that project...

</comment></comment><comment><USER19> "... we are all subject in the United States to the Patriot Act and it is possible that all that information could be made available to the authorities."

The right to privacy from the government is a logical consequence of the fact that the government is the servant of the people, put in place to protect and promote the people's interests. The Declaration of Independence describes our reason for instituting a government at all:

"We hold these truths to be self-evident, that all men are created equal, that they are endowed by their Creator with certain unalienable Rights, that among these are Life, Liberty and the pursuit of Happiness. — That to secure these rights, Governments are instituted among Men, deriving their just powers from the consent of the governed, ..."

We need no justification for privacy from the government and its proxies. The most upstanding citizen still has the right to keep his affairs private. _Even people with nothing to hide have the right to hide it_.

We explicitly give the government the power of search, but only when we (through a judge) agree that it's necessary. Without that agreement, they're not allowed:

Amendment 4 - Search and Seizure. Ratified 12/15/1791.

"The right of the people to be secure in their persons, houses, papers, and effects, against unreasonable searches and seizures, shall not be violated, and no Warrants shall issue, but upon probable cause, supported by Oath or affirmation, and particularly describing the place to be searched, and the persons or things to be seized."

We explicitly restrict all other powers of government through the 10th Amendment:

Amendment 10 - Powers of the States and People. Ratified 12/15/1791.

"The powers not delegated to the United States by the Constitution, nor prohibited by it to the States, are reserved to the States respectively, or to the people."

We do that because we recognize the tendency of unchecked governments to gather more power to themselves. When a government becomes more powerful than its people, the people are slaves who have lost "the Blessings of Liberty" as listed in the Constitution's Preamble.

Government is a dangerous tool. It's the people who should be watching the government, not the other way around.

<comment><USER20> I would like to be able to invoke the 4th amendment like this, but I'm having trouble thinking through how one would establish - if Google handed data to the feds under the Patriot Act - that the feds would be searching or seizing anything that was on my person, nor in my house, nor my papers, nor my effects.

Rather, it seems more likely that the feds would be knocking on Google's house and seizing a portion of _their_ "papers" that recorded the times that I've come over to visit. I don't think I own their log entries for my searches. Moreover, does a corporation get protection under that amendment, and even if they do, there's already the "warrant" mechanism.

<comment><USER19> Anything on Google would be your papers and effects, I would think. The spirit of the law is to protect your privacy, and it's not the place of government to search your affairs just because new technology makes that possible and practical.

Also, the government is not demanding data from Google because they're interested in Google; they're interested in _you_.

<comment><USER21> No, anything on Google would be their papers and effects... it's on their servers.

</comment></comment></comment></comment><comment><USER22> So much for "don't be evil".

<comment><USER23> It seems like he's actually telling you, the user, not to be evil. Sad.

<comment><USER24> It seems to me that what he thought he was referring to was cases like a subpoena for child pornography. The cases when google is obliged to divulge personal information are reserved for cases in which someone is seriously believed to be evil. When people read his quote I think tend to think of embarrassing things they themselves have searched for, but no one is going to subpoena (or even care) about your search for "nurse + handcuffs".

When I use google I know that there is some probability of the results getting hacked or leaked or subpoenaed or looked at by a google employee trying to improve search. However the expected value of that embarrassment (probability of someone I actually care about seeing it X how embarrassed I would be) is very low, and, for me, doesn't even come close to outweighing the benefit provided by the suite of google products. On the otherhand (and I think this is Erik's point) if you are using google to do something that would legitimately land you in jail the expected value of bad-stuff is much higher. (Actually both the probability of getting subpoenaed and the level of bad go up).

I feel that there is this idea that everything you do everywhere on the internet can and should be totally private and that any chance that it isn't is seen as removing a fundamental right. In my opinion the internet is more like a trail out in the middle of the wilderness: usually no one sees what you do and its probably ok to have sex with your girlfriend without anyone knowing, but if you start burying bodies out there the FBI can and will use that against you.

<comment><USER25> I worked for a company that routinely provided user-uploaded child porn to the FBI along with user details. The EULA gave us the right to do this although I'm not sure how explicitly this was stated. I'm ok with this, but it's a slippery slope downward.

<comment><USER26> Thats interesting. How do you avoid the liability of possessing child pornography? Invariablly, someone needs to see it before it can be passed on, and it has to be stored somewhere + communicated. Is there protections in your local laws?

<comment><USER25> We off-shored the manual review of submitted images! And, it was perceived by management/legal that proactive cooperation with the FBI would alleviate the risk of being charged with distribution of child pornography. As someone peripherally involved, I had to sign a release stating that I would not claim harassment because my job involved viewing sexually explicit images.

Never in my life had I imagined writing an email to a VP stating that I had just burned a CD of kiddie porn for her only to receive a terse "Thanks!" message back. We did this on a dedicated machine which could easily be wiped/re-imaged.

<comment><USER26> Sounds like very shakey ground. I would quite vigourously avoid pissing anyone off who has proof you (or anyone else for that matter) was directly involved with these materials. Literal interpretation of the law could see you registered as a sex offender + jail time. What a rediculous concept :/

</comment></comment></comment></comment></comment></comment></comment><comment><USER27> hey, guys, chill out. He's not saying that you shouldn't do anything you don't want the public to know or anything like that. Google is actually protecting your privacy from the public. Your search history is not available to anyone except for the law-enforcement with a subpoena. He is only saying that under _some_ circumstances Google _has_ to release _some_ information according to the current law. And if you don't want this happening, you should't have this information on Google in the first place.

For example if you plan to murder someone you better not google for "how to hide a corpse" :)

<comment><USER28> On a related note, if you plan to murder someone, you should not post an HN comment suggesting what not to google for when planning a murder.

</comment></comment><comment><USER29> I don't particularly care about the government investigating me.

My concern is about the people in power abusing their power to get information about their main rivals. Think "Watergate". Except without the physical breaking and entering.

There are legitimate reasons why people who have done nothing wrong would want what they have done not revealed to those in power. My concern about our descent into becoming a police state is that the checks and balances that should protect those people are disappearing.

<comment><USER30> _There are legitimate reasons why people who have done nothing wrong would want what they have done not revealed to those in power._

Mr. Braithwaite? We're investigating a co-worker of yours for possible misuse of computers for XXXXXXX. Can you please tell us: have you ever observed him doing XXXXXX? Do you think you could have a look at his computer the next time he takes a bathroom break and tell us if you see any signs of XXXXXX on it?

"That sounds pretty bad but to be perfectly honest, I'm very comfortable with your ability to gather surveillance information using normal means, including obtaining a search warrant for his computer. I'd like to consult a lawyer before answering any questions about anybody, thank you very much. Please excuse me..."

Mr. Braithwaite? Let me see here... Oh yes, you're they guy who likes to look at pictures of XXXXXX. And I see here you purchased the following toys... Do you keep those in the house where your... let me see... two kids visit? Does your ex-wife's lawyer know what kind of movies you like to watch? Would you like her to know? Sit down please, we have much to discuss.

You don't want to get all legal with us and demand a lawyer, do you? You don't want us to have to bother with a lot of troublesome search warrants, do you?

</comment></comment><comment><USER31> Funny, Schmidt is using the same reasoning people commonly use to defend the patriot act (and, for that matter, other government-instilled impediments on privacy and freedom of speech).

<comment><USER32> What is the reasoning?

</comment></comment><comment><USER33> "This seems backward. Almost certainly, there is something wrong with you if you don't think things you don't dare say out loud."

\- <http://www.paulgraham.com/say.html>

</comment><comment><USER34> Thanks for the confession... the scary part is the magnitude of which google has access to everyones personal data. Thanks to analytics they even can track everything outside of their direct control

<comment><USER35> I agree. This is an important thing to realize. It's also not just analytics. Every site that links to advertisements controlled by google or one of the many advertising companies they've acquired can also be used to correlate data about you. I started worrying more when they bought doubleclick.net. I wonder if that same google opt-out cookie works for those sites? What if you're concerned about your privacy enough to clear your cache and cookies every time you close your browser? Google is in a stronger position to track more of what you do on the net than many people may realize.

</comment></comment><comment><USER36> It is funny how everyone blames Schmidt and not the government or the patriot act or the politicians that enacted the patriot act. Schmidt was just being honest.

<comment><USER37> It sounds like Schmidt supports things like the Patriot Act.

If he was against it he would have said something like "if you like your privacy don't put it on the internet". Instead, he said "you shouldn't be doing it in the first place".

This single phrase has a strong connotation that you are doing something bad. Schmidt doesn't strike me as being dumb enough to say this without knowing exactly how the public would interpret it.

</comment></comment><comment><USER38> My corollary: if you have something to hide, and doing it is a good or neutral thing, then you have identified a bug in society. You probably ought to fix that.

<comment><USER39> I would rather hide, thank you. In a fight between me and the world, my bet is on the world.

<comment><USER38> Who else does the good/neutral thing? Make common cause with them.

</comment></comment></comment><comment><USER40> I guess "Don't be evil" now becomes "Don't be paranoid"

From Wikipedia Article about Google Chrome: Chrome sends details about its usage to Google through both optional and non-optional user tracking mechanisms

</comment><comment><USER41> Are you expecting Google to defy the US government? He is merely stating a fact:

"...we are all subject in the United States to the Patriot Act..."

How many of you are actually concerned you'll receive a subpoena from the feds to reveal you porn viewing habits?

<comment><USER42> The concern is that the feds will subpoena Google for everything they've got about everybody in a witchhunt sweep, and the results of that will be used to target individuals for further investigation based on 'suspicion of wrong- doing' rather than actual crimes being committed. Taken out of context, just about anything can be made to look suspicious.

</comment></comment><comment><USER43> This is only the tip of the iceberg, and the future absence of privacy will not really be about Google... Google is just a harbringer.

Universal sousveillance is the future, but since you'll be less and less able to tell whether you're in the future in this sense, acting as though you are might be prudent. :)

</comment><comment><USER44> Wow. I think the more important point (as the article says) is made in the next paragraph, where Eric Schmidt reminds us all that data Google collects is always subject to seizure by governments. So, even if you do trust Google, by using Google you're creating a data trail easily accessible by governments. If you used your own server based (say) in the US, on the other hand, then realistically only the US could seize your data, and even then it'd be much harder than if it were stored on Google.

</comment><comment><USER45> privacy and the argument against nothing to hide on slashdot: <http://yro.slashdot.org/article.pl?sid=07/07/10/2054219>

<comment><USER46> Don't miss the comment starting with "Privacy is only a way to protect you IF THE DATA ISN'T COLLECTED AT ALL." in that thread.

</comment></comment><comment><USER47> I've not really kept up with this story, nor can I rouse myself to feel the outrage shared by a lot of the community on this one... but if I were to throw in my two cents, I'd have to point to Google Flu Trends [1] and the fact that their modeling of influenza outbreak and spread are some of the most accurate and up-to-date on earth, often besting the World Health Organisation. This is sometimes used as justification for extending the length of time that Google retains data.

If this anti-google sentiment is to be believed and they are marching the corporate road to evil, I shudder to think of the information that could be inferred from collective search pattern trends.

I just noticed that the example had not been given yet and I think it's an important one in this debate. It is a seemingly benign, if not entirely altruistic, case study. What is the consensus, smoke and mirrors, or is there a real social value to be gleaned [2] from the harvest of the vast data swarm?

[1: <http://www.google.org/flutrends/>] [2: <http://en.wikipedia.org/wiki/Gleaning>]

</comment><comment><USER48> If you wanted to end Google, all you would have to do is offer a reward for a whistleblower to videotape a privacy violation at Mountain View.

Imagine putting _that_ on Youtube.

(EDIT: yes, I know they own Youtube. Of course they'll take it down. But it would be reposted on Vimeo and the like, and the publicity for the takedown would cause the Streisand effect).

<comment><USER49> Except that Google owns Youtube.

Imagine that getting _taken down_ from Youtube in record time.

</comment></comment><comment><USER50> The video clip is available at [http://gawker.com/5419271/google-ceo-secrets- are-for-filthy-...](http://gawker.com/5419271/google-ceo-secrets-are-for- filthy-people)

</comment><comment><USER51> reminds me of this riff on privacy by Bruce Schneier from a few years back addressing the same notion that Schmidt is quoted here to believe

[http://www.schneier.com/blog/archives/2006/05/the_value_of_p...](http://www.schneier.com/blog/archives/2006/05/the_value_of_pr.html)

<comment><USER38> Privacy protects from abusive power, and privacy protects from irrational judgment: true but incomplete. Privacy allows us to _tolerate_ these problems - rather than being forced to confront them. That may not be a good thing.

<comment><USER52> Society is full of things we compartmentalize, minimize and tolerate.

Mostly, because we don't know _how_ to confront them. I certainly don't know how to patch human nature so that power doesn't corrupt. So, in the meantime, I'll take privacy, thanks.

</comment></comment></comment><comment><USER53> The thing people don't realize with google - You are not the customer. You are the product. The customers are the people who pay google to do their advertising.

</comment><comment><USER54> this thread is now too massive to get any decent visibility for this, oh well.

there is a much deeper, more philosophical reason for instituting a right to privacy than just the fear of corrupt governments.

humans are adapted to signal certain values for the sake of group cohesion. we have a "public persona" and a private life. this isn't something sinister, it's a way of coping with the demands placed on us by our interaction with others. we know in practice that depriving people of the ability to hide things from each other is extremely detrimental to their psyches. most importantly, it is detrimental regardless of the triviality of the things kept secret.

when deprived of the ability to keep secrets people are forced to be "always on". they must maintain the "proper" signals at all times. this is extremely draining (yes, even for extroverts, eventually). the things we like to signal to others aren't necessarily the things that make us most happy.

</comment><comment><USER55> I don't quite understand why this recent trend has started, but things are showing up on HN that were on Reddit a few days ago.

3 days ago in this case.

[http://www.reddit.com/r/technology/comments/ab8jt/googles_ce...](http://www.reddit.com/r/technology/comments/ab8jt/googles_ceo_adopting_the_if_you_arent_guilty_then/)

It's like a very slow echo.

</comment><comment><USER56> We need a privacy contract similar to Richard Stallman's GNU General Public License. Something standard that both parties can quickly understand and agree to.

Idea: A new competing search engine could release a General Privacy Agreement along with their search service.

<comment><USER57> Maybe this would be a good starting point:

<https://www.ixquick.com/eng/protect-privacy.html>

They claim to be _"The only search engine that does not record your IP address."_

<comment><USER19> Thank you for that. Very interesting meta-search engine, clean results. It appears that Google is not among the engines they use.

I like that they don't record your IP, and that they offer https access. Hard to say whether they really do what they say, but they do have the European Privacy Seal, for whatever that's worth.

I'm using it now to see how I like it, so far so good.

</comment></comment></comment><comment><USER58> I think it's a very fair and reasonable advice. You don't _have_ to use Google. This is the service Google offers. Your privacy is your payment. It's a great opportunity for someone to start a competing search service that focuses on privacy.

<comment><USER59> Cuil's shtick is that they don't store any search history <http://www.cuil.com/>

</comment></comment><comment><USER60> The sad thing here really is how obviously limited his perspective is. Maybe this is true in the U.S. (but I'm pretty sure privacy is still something highly valued here) but what about foreign citizens? What about people who have things to hide from the chinese government? What about governments that have things to hide from their people? Oh, wait, Google already answered that:

<http://en.wikipedia.org/wiki/Censorship_by_Google#China>

</comment><comment><USER61> I bet that he wishes he could roll-back that comment!

I think that most non-tech people would be very surprised if they realized how much personal data our corporate overlords have collected.

</comment><comment><USER62> Yeah I disagree with Eric on this point. There are many things I think are fine and acceptable to do, but they don't need to be a part of my public life.

I don't want employers, grandparents, and the government to have access to my private and personal life.

No reason everyone should know movies, books, music, games, sexuality, political beliefs, and everything else about me.

</comment><comment><USER63> the real problem is not google or schmidt, but things like the patriot act and the ridiculous habit of our government of using "matter of national security" as a catchall for abuse.

the more people who realize how little privacy they have left the better. it's the only way the trend will be reversed.

</comment><comment><USER64> Thomas Jefferson would not approve of that opinion. I actually think less of Google for this stance.

</comment><comment><USER65> In twenty years, does privacy still exist? What about in a hundred?

I'm definitely not a fan of exposing data that people wish to keep private. It seems conceivable, though, that privacy is an idea that won't survive time and technology.

<comment><USER66> Exactly. If you didn't see the death of privacy as early as the 80s you either weren't paying attention, or weren't born yet.

What's going on here is a redefining of personal data as "private but without privacy." While the distinction itself is somewhat subtle, the ways this redefining is being carried out are anything but.

In Eric's case, he has never exactly been known for his subtlety of thought, has he…?

</comment></comment><comment><USER67> Cool, so Google will be publishing the search histories of all Google employees by name?

Hypocrite.

</comment><comment><USER68> If you need to justify your non-privacy policies with accusations, maybe you shouldn't be collecting so much identifiable data.

There are plenty of legitimate, even honorable actions which are threatened by poor privacy policies.

</comment><comment><USER69> If you have a cancer, maybe you shouldn't have it.

It is very easy to get caught with this type of argumentation. It is wrong.

Keywords may actually reveal a lot more that one would assume, and prone for des-interpretation as well.

</comment><comment><USER37> Previously submitted here

<http://news.ycombinator.com/item?id=982838>

Though unfortunately my submission didn't get as many upvotes for whatever reason.

</comment><comment><USER70> In other words, if I use Google Checkout, I should be ashamed of myself for wanting my credit card information kept private?

<comment><USER71> The quote doesn't have anything to do with shame. Not sure how you read into that...

The message is simply that any information on the Internet is subject to the risk of exposure. We use credit cards knowing that the risk is very low - <USER2> we all know its _possible_ your credit card number will be intercepted, misused, etc. This is a fact, not a question of morality.

</comment></comment><comment><USER72> "in our case we're building the platform that will allow the content people to do more targeted content."

Telling us what 'personalized' is about. Not doing better searching, doing better selling. Find the people with a little extra money, 24/7, and offer them something they might go for, 24/7, whether they need it or don't. Helping them stay one paycheck away from broke.

</comment><comment><USER73> Go to Scroogle and click HTTPS...

<https://ssl.scroogle.org>

Problem solved!

</comment><comment><USER74> I think that Mr. Schmidt is trying to do two things: (1) Preparing for Google to eventually be "outed" about just how much data it actually hands over to the government on the basis of the Patriot Act; (2) Distracting the general public from the broader range of privacy concerns that Google raises (specifically: private surveillance as contrasted with government surveillance).

He mentions the Patriot Act and I would paraphrase him as throwing up his arms and saying "Hey, we're subject to those laws just like you! We have no choice!" Alas, Google does have the choice and unique opportunity to challenge those laws but it appears that they simply decline. That is why I think he's preparing for Google to be outed vis a vis the extent of its cooperation with government.

Next, asked about privacy concerns, he immediately reframes the question as if it were a question about government surveillance - neatly ducking any issues about private surveillance. Bartalomo fails to follow up.

The issue about private surveillance is significant in mundane (but serious) ways and in more speculative, cultural crisis kind of ways.

A mundane concern is the question of whether or not Google "leaks like a sieve". For example, if someone at Google dislikes me, how hard is it for them to scratch around for dirt in my (nominally) private information? If someone at Google is a friend or (black market) business partner of a recruiter at some other firm to which I apply for employment, is an unauthorized background check possible? Do such things happen? It's hard to imagine that they are _technically_ hard to prevent....

A speculative but serious cultural crisis concern regards Google's deliberate and accidental implementation of what we could dub behavioral tracking and targeted manipulation. For example, let's suppose that one day I get a crazy suspicion from something I read on a blog. I think that there could possibly be a vast conspiracy and that my best chances for survival might be to liquify all of my assets and invest everything I have in personal caches of peppermint candy. I don't fully believe it but I start researching via Google. Google's AIs quickly figure out that I linger on ad-carrying content from backers of this conspiracy theory and, pretty soon, that stuff is at the top of all my searches and predicts 40% of the ads I see. From my perspective: that stuff is "all over the Internet" and I begin to wonder if my neighbors aren't in on the conspiracy since none of them talk about it. Pretty soon, Google has built me a personal, private channel of conspiracy news and peppermint candy ads. I sell my house, leave my family, cash out my retirement plan, by a van to live down by the river, and fill up a rent-a-store spot with boxes of candy.

Now, that same personal bug - my vulnerability to a conspiracy theory - <USER3> have been there all along. The change here is that back in, say, 1975 - <USER4> biggest mistake I might make is to subscribe to a few newsletters and magazines, perhaps hedge by buying a dozen boxes of candy rather than a gross. Meanwhile, the TV, the radio, all these other things are feeding me counter- evidence to the conspiracy and I'd eventually have a much better chance of coming to my senses.

But Google is unprecedented in its comprehensiveness as a source of monitoring and manipulating my attention. Like the Vegas slot machine that convinces me (wrongly) that I have a system to beat the slots because it pays out 49% of the time, so long as I pay it with my recognizable frequent-gambler credit card -- Google's AI can potentially really mess a person up.

 _That_ is a privacy concern. That's a privacy concern directly raised by Google's stated intentions of dabbling in behavioral tracking. And it's a concern that Schmidt deflects from attention by turning the question into a question about government surveillance.

Now, things can get even worse. Suppose that Google's AI doesn't cause large numbers of people to drop out of society and load up on peppermint candy but that, shucks, little discrete tweaks here and there can use the same behavioral science effects to, say, influence the next election for the local dog catcher. And one of the candidates for dog catcher has a slush fund, and a friend at Google.... Or, forget the friend at Google... suppose the dog catcher candidate has a friend who, this year at least, has found some underhanded tricks that currently work really well for SEO....

The ethics of running search and ad placement are unprecedented and difficult and hinge very much on how we update our understanding of "privacy". They hinge very much on the question of privacy vs. behavioral tracking. Schmidt appears uninterested in talking about these issues and very much interested in deflecting attention away from them.

At most only slightly exaggerating: we're all analogous to lab rats, now, in Google's massive-scale human-subject psychological research project with unprecedented amounts of automated surveillance and stimulus production. With money and power hinging on the outcome.

</comment><comment><USER75> 'If you don't want anyone to know, don't do it'

Bullshit argument for invading privacy. I prefer to hide my genitals from public view.

Since he has nothing to hide, let's set up a streaming webcam outside his house, zoomed in on his bedroom. Or better yet, let's implant a nano-camera up his rectum.

</comment><comment><USER76> So when are people going to start poisoning their search histories by collectively performing mass searches for "bad" content? I've been expecting that to be the eventual reaction to Google disclosing search history.

</comment><comment><USER77> So, at what point does Google having all your info (e-mail, DNS, search, Google Voice, social network activity, etc.) become worse than AT&T / NSA tapping all your calls?

</comment><comment><USER78> Like participating in the democratic process?

(see voter privacy)

</comment><comment><USER79> Absolute power corrupts absolutely

<comment><USER80> This is why philosopher kings aren't allowed possessions.

</comment></comment><comment><USER81> Google - Be Evil.

</comment><comment><USER82> The register suggests it is possible to avoid Google products, but the web is literally infested with Google ads and that is a tracking mechanism.

<comment><USER83> you can block cookies based on domain name, or from third parties

<comment><USER84> There is also, unless I'm mistaken, the possibility of Javascript-instantiated and Flash ads which have the possibility of relaying your IP to the mothership.

Having a filter on your DNS client that excludes Google-related IPs might be more thorough.

</comment></comment></comment><comment><USER85> If you have something to hide, don't do it on the Internet. Sure. But Schmidt's careless disregard for privacy? Look, _1984_ was a warning, not a textbook.

</comment><comment><USER86> Does this mean he's going to open source all of Google's infrastructure? And release all the trade secrets?

That would be awesome!

</comment><comment><USER87> "If you have something to hide, maybe you shouldn't be doing it" is valid not only in the context of net privacy but much more so as a general rule.

<comment><USER88> Do you close the door when you go to the bathroom?

<comment><USER43> I do close the door, but I don't search for pinhole cameras in public bathrooms, in spite of the fact that some exist. I think that highlights the difference between expecting privacy and trying to ensure it.

<comment><USER37> But the fact is that you still seek out a "private" bathroom to relieve yourself and don clothing do you not? Or are you as likely to do the same things in plain view?

What do you call that, if not ensuring privacy? The only thing different is that your level of trust is lower than others.

<comment><USER43> Even if I didn't subscribe to the taboos of my culture (which I do, at least in this case), there are lots of excellent reasons to use a facility meant to handle human waste.

Because of that, this is a bad example.

Instead, we should be using an example of something which doesn't require any special facility, we have no cultural taboos against displaying[1], and isn't illegal, but which people want to hide, anyway. I'm not sure I can think of anything.

[1] As someone who was raised fundamentalist Christian, I can assure you that habits learned in childhood can still have lots of force in adulthood, even if you no longer agree with the reasons for them.

<comment><USER88> The only think i can think of is illness or weakness in the presence of someone you don't trust.

I believe it is universal, although I'm willing to accept there may exist cultures that don't have this desire.

In any case, everyone knowing everyone else's state of health sounds ok, until i think about people i don't like. It's a little too enticing to know about how X is sick, and i don't want X to know that i am sick.

Maybe this is to abstract to be useful.</comment></comment></comment></comment></comment></comment>


Google is reportedly trying to buy Fitbit - <AUTHOR> https://www.theverge.com/2019/10/28/20936415/google-fitbit-acquisition-alphabet <comment><USER2> Dupe: [https://news.ycombinator.com/item?id=21378471](https://news.ycombinator.com/item?id=21378471)</comment>


Lojban: A constructed language, grammatically based on Predicate Calculus - <AUTHOR> http://www.lojban.org/tiki/Lojban+Introductory+Brochure <comment><USER2> Lojban: "spelling is phonetic and unambiguous"

They clearly ignored that rule for the name of the language =)

On a more serious note, this is rather exciting. It would make a very good candidate for a universal language. A great deal of time and effort went into making this language. Hopefully this isn't the last time I hear of Lojban.

<comment><USER3> If you like this, you should read In The Land of Invented Languages by Arika Okrent. Aside from the being the best non-fiction book I've ever read, it talks about Lojban and characteristics of invented languages that make them suitable for becoming a universal language. It turns out, Lojban would make a terrible universal language for precisely the reason I pointed out in my other post. It's too complicated. When we speak, we frequently start a sentence not knowing how it's going to end. We pause and insert filler words to give us time to form our thoughts. We use ambiguity because sometimes we just don't know precisely what we're trying to say (Lojban has, if I recall correctly from Okrent's book, over 30 ways to say 'and'). Forcing people to have a complete understanding of what they want to say before saying it won't make them more clear when they talk, it'll make them not talk.

<comment><USER4>
      > When we speak, we frequently start a sentence not knowing how it's going to end.

You can start and continue with a Lojban sentence indefinitely via various means. Metalinguistic markers such as _sei ... se'u_ allows a discursive (on- the-fly) predicate or sentence. You can insert parenthetical notes with _to ... toi_ into anywhere. You can insert the attitudinals (used to express attitudes, emotions, evidentiality, etc.) into anywhere. A construct called _tanru_ allows an endless sequence of predicates (whose form in Lojban do not alter according to the natlang parts of speech such as adjective or adverb, thus imposing less restrictions on the way you keep forming a sentence than natlangs such as English does). With _li'o_ you can omit any quantity of text you don't want in your expressions. With _si/sa/su_ you can 'erase' various mistakes in your utterance. And so forth.

      > We pause and insert filler words to give us time to form our thoughts.

That's what _y_ is for in Lojban.

      > We use ambiguity because sometimes we just don't know precisely what we're trying to say

The unambiguity of Lojban is mostly syntactic, not always semantic. You can be semantically ambiguous in Lojban with e.g. tanru.

      > (Lojban has, if I recall correctly from Okrent's book, over 30 ways to say 'and').

Which includes _ju'e_ , a vague connective for "and".

      > Forcing people to have a complete understanding of what they want to say before saying it

Which is not the case in Lojban.

<comment><USER3> Syntactic ambiguity is half the point. I still have to decide whether or not I want to use the vague/all-purpose connective for "and." If I want to add parenthetical information, I have to realize I'm doing that and indicate it. If I want to "erase" a mistake, I have to recognize that that's what I want to do and use the proper word. I still have to know exactly what I intend to say. When I'm speaking, I don't think to myself, "Ok, this is parenthetical... this is correcting an error... this is vague and all-purpose." I just speak.

In Python, whether or not a method is private is defined basically by whether or not you call it from another class. In Java, you have to be explicit about it. You CAN just declare everything public (be vague and all-purpose), but aside from being frowned upon, you still have to conciously decide to do this. If you don't care about the protection of scope, you can make everything public, but you're still specifying scope. In Python, you don't specify scope at all. Even if you wanted to, you can't.

The same thing shows up in type checking. If you want, you just call everything an object in Java. It's vague and all-purpose, but you still have to specify a type. You still can't write 'a = 4'. You still have to write 'Object a = new Integer(4);', which it's tough to argue is simpler just because it isn't type-checked.

Explicit ambiguity is not much closer than explicit disambiguity to implicit ambiguity.

I realize at this point, I'm getting beyond my own knowledge of Lojban, but I just want to make the general point that being able to be ambiguous does not automatically afford the advantages of the natural ambiguity in "natlangs."

<comment><USER4>
      > I still have to decide whether or not I want to use the vague/all-purpose connective for "and."

If you are undetermined, you use the undetermined option, the vague one. You don't really make a decision for that.

      > If I want to add parenthetical information, I have to realize I'm doing that and indicate it. If I want to "erase" a mistake, I have to recognize that that's what I want to do and use the proper word. I still have to know exactly what I intend to say.

Planning is not required for adding parenthetical information. You use it on the spot where you happen to want it.

If you don't recognize that you want to erase a mistake, you just don't use the erasers.

      > When I'm speaking, I don't think to myself, "Ok, this is parenthetical... this is correcting an error... this is vague and all-purpose." I just speak.

I'm not a native English speaker. When I started learning and speaking in English, I would think to myself, "Ok, the word "which", when used after a slight pause, which corresponds to the comma in writing, it means a non- restrictive relative pronoun, which is what I want now, so I'm going to use it that way". As I kept practicing the language, I internalized the rule, becoming less and less actively conscious of it. Lojban is no exception. I have already internalized some parts of the grammar to which I used to pay much attention in my earlier period of learning.

Also important to note is that, what English expresses with non-verbal properties such as intonation, Lojban can do verbally. When you orally say something in English which you would put in a parenthesis in writing, your speech act are still subject to some phonetic principles such as inserting a pause, changing the rhythm, lowering the pitch, using less breath, and so on. You consciously or unconsciously have to be in command of these properties if you are to successfully deliver your utterance. A particular combination of these non-verbal phonetic properties is what Lojban expresses with _to ... toi_ , the parenthetical markers. Whether or not you are actively conscious of them is a matter of internalization. Just like an experienced English speaker not always actively thinks about each noun's grammatical number in their utterance but still manages to add the plural marker "s" where appropriate, an experienced Lojban speaker would be able to correctly start a parenthetical note with "to" but without always actively checking whether or not that's what they really want to say.

> I realize at this point, I'm getting beyond my own knowledge of Lojban, but > I just want to make the general point that being able to be ambiguous does > not automatically afford the advantages of the natural ambiguity in > "natlangs."

Could you give me an English example with such advantages?

</comment></comment></comment></comment></comment><comment><USER3> What they don't tell you is that Lojban is so complicated no one can speak it. Go to the forums and pick almost any thread that contains Lojban, and you'll find debating about the language. If the supporters can't even speak it right, what hope if there for those who just want to communicate?

<comment><AUTHOR> My experience suggests that there are a handful of people who can read, write and speak the language fairly well, using it, when they choose, as their primary means of communication.

Most of the discussion falls into two types.

The first type is when a newbie tries to transliterate into lojban, and often says something they don't mean. This is common with anyone learning another language, it's just that in lojban one can be much more precise about what was wrong, because there's a formal grammar. Common confusions are mixing "verbs" with "nouns", neither of which have an exact matching concept in lojban, hence for some it's tough to get a grip on what's going on.

The second is debating the semantics. What does this word, explicitly created for lojban, actually mean? What's the difference between a pot and a jar? How, in English, do we decide which word to use? What is the defining characteristic of each?

These are often held up as criticisms of the language, but I think unfairly so. They provoke debates in epistomolgy, semantics and pragmatics, and touch on many of the questions that will need to be answered by the machine translation, semantic web and artifical intelligence challenges.

And what's the point in learning a language that's just a re-coding of your native language?

The other undeniable short-coming is that there is no underlying "culture." Much of the interest in learning, say, Russian, comes from the cultural insights. There is none of that in lojban.

Lojban isn't ready for generic, general purpose communication. It's still growing. If all you want is a tool, don't go there.

If you want mind-expanding concepts and debate, have a look.

<comment><USER3> First of all, let me get something out of the way. There are no languages that are just re-coding of other languages.

I understand the idea behind Lojban but you've already conceded my point. Lojban is not a suitable language for communication.

<comment><AUTHOR>
      > There are no languages that are just re-coding
      > of other languages.

That's true, but it's "less true" of Italian/Spanish or Russian/Bulgarian than it is of, say, Russian/Swedish. I suspect that we are in complete agreement, and might argue only about precision of expression in our opinions.

And I didn't say lojban is unsuitable for communication. There are a few people who use it exclusively when communicating with each other. It is not yet ready for generic, general-purpose communication, just as the newest computer programming language isn't ready for mainstream use by those not interesting in the language so much as getting things done.

But again, I suspect we are/would be in "violent agreement" about most of these points. I contend that lojban is, regardless, mind-expandingly interesting.

For some.</comment></comment></comment></comment>


Radio identification on mobile device - <AUTHOR>

Hello everyone. I am very new to mobile development, so I have one question about radio identification on mobile device. Is there any way to develop mobile software which identifies radio signal (recognizes fm frequency) that plays in the room in which I dwell. Any real case scenarios? Thank you. <comment><USER2> I'm not sure what you're asking here. Do you have an FM transmitter in your room? Or are you talking about detecting what FM radio station you're listening to. If the latter, you're jumping way too far ahead of yourself and you need to bone up on radio theory (eg. get an amateur radio license and the ARRL radio guide).

So I'll answer the question with a random answer that is most likely to be useful to you down the road.

It may be possible to experiment with FM on certain Android devices that have FM receivers in them AND they are not disabled AND the antenna leads are connected AND you have API access to those receivers.

Every chip maker does it different, but most likely you'll be dealing in kernel space. Hope you're comfortable there.

This is a question better off for the more esoteric corners of <http://forum.xda-developers.com> . There was also an official Google Android hardware hackers list somewhere. I'd get you the link, but Google nerfed the Android Developers website recently and I can't find anything.

<comment><AUTHOR> I am talking about detecting FM radio station on my mobile device.I want my mobile device to be able to catch the radio station. What about sound radio detection (does not have to be frequency detection)? Is it more obviously? All I need is an algorithm for radio station detection. Thank you for your answer. I will check Android FM receivers.</comment></comment>


Growing a Language, by Guy Steele [video] - <AUTHOR> http://video.google.com/videoplay?docid=-8860158196198824415

<comment><USER2> I watched this video a week or so ago (it was linked in a Tell Me Which Google Tech Talks Are Good To Watch thread) and I was impressed at how well-reasoned his points were, how poorly Java has lived up to them since 1998 when the talk was recorded (still no operator overloading!) and how many subsequent languages have followed his guidelines for growth, either deliberately or as parallel evolution.

For example, although Python's syntax isn't extensible to the extent that Scheme or Lisp are, Python's Decimal class is still fairly transparently interoperable with built-in numeric classes. Other parts of the interpreter, such as object instances, classes and modules are built out of exactly the same dict objects that normal scripts use, which makes it easy to build new language features that feel like part of the base language.

<comment><USER3> I note that Guy Steele's examples support a language which has a sufficient vocabulary. (Which can be easily expanded.) There is no example which argues for a large _syntax_.

EDIT: I would also need to add that a language that wants to be widely popular needs a _large enough_ syntax. Even though it results in a less elegant design, enough cultural expectations need to be respected so that programmers are immediately comfortable. Python is close to this ideal. (But definitely errs on the side of being too large.) Operator precedence is so deeply ingrained in engineering and technical culture, that it is a non-negotiable for many programmers.

To put it another way, algebras are a _pattern_ most expect to be a feature of a language. (This is written by a Smalltalker, who is perfectly comfortable working in a language with its own, much simpler, rules of precedence.)

EDIT: Languages should be designed so that it is easy to implement Algebras and easy to use Algebras!

</comment></comment><comment><USER4> I loved the paper - it took me a couple pages before I figured out what he was doing. Very clever.</comment>


Ask HN: How do you sync your user/dev environments? - ask-hn What do you do to sync environments across multiple systems? Not just development, but also general user space.
I've considered putting my user directory in Dropbox, a git repo, or a Docker container.
Any other suggestions? Any that offer live-reload upon change? <comment><AUTHOR> Hi, maybe rclone + [https://github.com/rhummelmose/rclonesyncservice](https://github.com/rhummelmose/rclonesyncservice)</comment>


Artificial Intelligence Is Already Weirdly Inhuman (2015) - <AUTHOR> http://nautil.us/issue/27/dark-matter/artificial-intelligence-is-already-weirdly-inhuman <comment><USER2> I don't like when articles make it seem as if neural networks think, and come up with results the way humans or alians would. They don't perceive, they don't reason, they just crunch numbers. Also, it's strange to say neural networks run algorithms written by humans - They don't. You might argue training creates the algorithm run, but even then, humans didn't design saif algorithms. The algorithms humans write are training algorithms, or the neural network itself. These articles are clearly written by people who haven't ever worked on AI research before (as with most internet articles on anything of course)</comment>


'Suspicious package' at Texas A&M was an admission letter with lights attached - <AUTHOR> http://www.kbtx.com/content/news/Code-Maroon-Texas-AM-building-evacuated-due-to-suspicious-package-405462035.html <comment><AUTHOR> I hope the kid gets admitted! That might be iffy, though: The newspaper report said that the suspicious package was "neutralized" by the bomb squad. [1]

[1] [http://www.chron.com/news/houston- texas/houston/article/Bomb...](http://www.chron.com/news/houston- texas/houston/article/Bomb-squad-investigating-package-at-Texas- A-M-10783078.php)

</comment><comment><USER2> Well...the application certainly worked in garnering the attention the applicant had hoped for.</comment>


Dinosaur egg treasure trove found in Japan - <AUTHOR> http://bgr.com/2015/07/14/dinosaur-egg-treasure-trove-found-in-japan/ <comment><USER2> They mention dinosaur eggs in the 1-5oz size range. For comparison, a grocery- store chicken egg is typically 2-2.5oz; an ostrich egg is about 48oz.</comment>


PayPal Targets Identity Ownership with Its First Blockchain Investment - <AUTHOR> https://www.forbes.com/sites/michaeldelcastillo/2019/04/02/paypal-targets-identity-ownership-with-its-first-blockchain-investment/ <comment><AUTHOR> Keybase seems relevant here. Do they have any blockchain projects? Namecoin would seem an obvious partner.</comment>


The GPL, the App Store, and You - <AUTHOR> http://i.tuaw.com/2011/01/09/the-gpl-the-app-store-and-you/

<comment><USER2> What's so funny about all this is that the distribution constraints of the App Store do not make it incompatible with the GPLv2. Having to pay Apple $99/year for keys is irrelevant, as is the signing of binaries that prevents redistribution between devices (and even if those were problems, there's jailbreaking). Hell, it wouldn't violate the GPLv2 even if there was no way for users to get their code on the device at all (see TiVo, et. al.). It's unfortunate that all the early discussion of this was laser-focused on the irrelevant DRM.

Rémi's argument is that the Apple Store Terms & Conditions constrain your usage rights — _"YOU SHALL BE AUTHORIZED TO USE THE PRODUCTS ONLY FOR PERSONAL, NONCOMMERCIAL USE."_ — and downstream authors/distributors are not allowed to _"impose any further restrictions"_ beyond the GPL.

Rémi is technically correct (the best kind of correct!) that the App Store terms are in conflict with the GPL. The clause in question isn't just for apps, its basic purpose is that purchasing a song or movie from Apple doesn't give you public performance rights to it. Apple could pay their lawyers to bloat out the whole usage section so that it'd be difficult for Rémi to pick a fight over it, but they probably won't.

But if people are going to pick fights over minor unenforceable EULA terms, they're going to find them everywhere they look. I mean for fuck's sake you also agree that you will not use iPhone software in the operation of nuclear facilities or for aircraft navigation, like in a great many EULAs. If some ideologue really wants to burn it all down they could probably find a ton of high-profile but ultimately pointless technical GPL violations in this vein.

Wherever a fight is sought it will be found. No copyright license will ever prevent either upstream or downstream from being huge asshats if that's what they want to be.

<comment><USER3> Remi's arguements were an attempt to get Apple to change the 'personal non- commercial part' of it's licence to something more open.

But I suspect that Apple removing VLC has more to do with controlling what formats you can play, and so where you can buy content, rather than a desire to protect VLC's GPL contributions.

Interesting that the super hacker cool Apple have effectively banned GPL from their popular platform and with the Mac iStore may soon be banning it from all their machines. While MSFT is announcing more open source support. Looks like the lightsaber of the dark side has been passed on.

<comment><USER4> Apple didn't ban the GPL from the App Store.

The GPL banned the GPL from the App Store.

<comment><USER5> Their rules are mutually incompatible.

<comment><USER4> Which changes nothing, as it still isn't a conscious act by Apple to explicitly ban the GPL.

The GPL banned itself from the App Store, plain and simple, yet everybody will still make it about big bad Apple censoring libre software from the App Store. You chose the GPL when you started your project, Apple didn't. If the license that YOU chose for YOUR software doesn't allow you to distribute YOUR software under Apple's EULA, that isn't Apple's fault.

There's also no financial incentive for them to modify their EULA to be more GPL-friendly. They're going to say, gee, those hundreds of thousands of apps on the App Store have succeeded with the EULA we have, why get into legal battles over two publicized apps? The publicity is negative for us, so why bother?

<comment><USER3> On the same basis the USA bans GPL software, you aren't allowed to export stuff to naughty countries if you are American. The GPL doesn't allow you to block Cuba so nobody in the US is legally allowed to use GPL.

This doesn't seem to led to IBM, Oracle etc stopping selling Linux. Apple put a line in it's\ EULA about only being able to use app store for non-commercial use, which is ridiculous businessmen do use iPhones and iPads

</comment></comment></comment></comment></comment></comment><comment><USER6> I am still baffled by this term:

(i) You may download and sync a Product for personal, noncommercial use on any Apple-branded products running iOS ("iOS Product") you own or control.

Doesn't that mean that you can't make commercial use of iOS apps? Isn't that ridiculous?

<comment><USER2> 'Products' aren't just iOS apps — the category also includes purchased music, movies, television, audiobooks, and iPod games; as well as rentals of movies & television, iTunes U content, and probably more that I'm forgetting.

For the rest of it they'd never be able to publish it without something substantially similar to the "personal, noncommercial use" clause.

Yes, for apps they should probably totally defer to the application authors, instead of just letting them be more restrictive if they so choose.

<comment><USER6> And until they "defer" anyone who say uses twitter to promote their business is in violation of the ToS?

</comment></comment></comment><comment><USER7> I have to call out the whole referenced article as BS. The situation is that Rene filed the complaint, and after Apple complied with Rene's request, Rene then blogged that Apple was being mean by removing the app, as he had requested.

Sorry but there is no honest interpretation here in which Apple is the bad guy and long winded articles claiming so because they only tell half the story are dishonest.

<comment><USER4> It's a lot easier to blame Apple, and people love doing it.

In the other thread I was absolutely appalled that the leader of the project was shifting all of the blame to Apple, and standing by his choice to do so. Just look at all of the comments on Hacker News (one in this thread) reviling Apple for their supposed "anti-GPL stance".

Clearly, the VideoLAN people got what they wanted: the app removed, and pie in the face for Apple with respect to the GPL. Two wins for the price of one?

Link: <http://news.ycombinator.com/item?id=2083254></comment></comment>


What Is Cloud Hosting? - <AUTHOR> http://www.openhosting.com/2011/12/what-is-cloud-hosting/

<comment><USER2> Sure, I think this is a cogent explanation free of BS.</comment>


The Libor Change Is Coming - <AUTHOR> https://www.bloomberg.com/opinion/articles/2019-08-27/the-libor-change-is-coming <comment><AUTHOR> This is one way of getting management buy-in for code maintenance:

> Imagine if you worked at a bank and someone came to you and said, "you have > to restructure all of your derivatives now, every single one of them, it is > basically the law." On the one hand: Sounds like a lot of work! On the other > hand: Restructuring every derivative! You are going to get so rich!

> [...]

> Yeah, look, I take the point, there's a lot to do and it's complicated and > labor-intensive. If you're the Libor Transition Guy at a big bank, you are > going to be spending so much money on lawyers and computer programmers to > scour and rewrite all your contracts and code to reflect new interest-rate > benchmarks. On the other hand, if you are not also the biggest profit center > at the bank for the next five years, I kind of think you are doing it wrong?</comment>


Ask HN: Good solution for video chat on home TV? - <AUTHOR> I've been searching non-stop and even trying to build my own solution with a raspberry pi but I have not yet found a good solution for doing video chat between my daughter and her grandparents. I have a nice big 4K tv and would love a simple way to connect TV to TV and do a video chat.
1. FaceTime is great but when hooking iphone up to TV it does not rotate into landscape mode. A ridiculous shortcoming that's been there for years.
2.  Raspberry Pi 4 with a webcam. Works ok but mine overheats and lags when steaming 1080p from webcam and displaying content on my 4K tv. Also less friendly for the grandparents.
3. Tried a MacBook to MacBook with hdmi out on both TVs. Works ok but the laptop screen has to be open and you wind up looking at the laptop screen.
4. Haven't tried the Facebook Portal device yet but that may work.
5. Buy a pro video conference solution. Haven't given this serious thought because of cost. Over 1k in most cases.
Anybody have success doing this? <comment><USER2> There's also Solaborate although I haven't tried it. [https://www.solaborate.com/hello2](https://www.solaborate.com/hello2)</comment>


Pastry: A distributed hash table in Go - <AUTHOR> http://secondbit.org/blog/introducing-pastry/ <comment><USER2> >We couldn't seem to find any way to subscribe to events and publish events without a single point of failure.

Really? You think this doesn't exist? In fact, this is nothing but a _deployment_ concern for any mature message broker.

<http://www.rabbitmq.com/ha.html>

<http://activemq.apache.org/masterslave.html>

[http://hornetq.sourceforge.net/docs/hornetq-2.0.0.GA/user- ma...](http://hornetq.sourceforge.net/docs/hornetq-2.0.0.GA/user- manual/en/html/ha.html)

There are lots more options than these, and you can also use heartbeat/lvs to take something like redis and make it HA.

I'm glad you had fun inventing your own distributed hash infrastructure, but please do not attempt to convince other people that there are no other options out there for reliable and highly available messaging.

<comment><AUTHOR> Er. This seems a little overly hostile for something that amounted to "Hey, here's this fun thing I made. Anyone else want to play with it?"

Rather than arguing this point again, I'll just point at the other comment thread that addressed these concerns.

Fun fact: I didn't mention highly available messaging in the post at all. I mentioned single points of failure and bottlenecks. Those aren't quite the same.

<comment><USER2> A highly available system has no single point of failure.

<comment><AUTHOR> It can, however, have bottlenecks.

All of your solutions, based on the conversation below, separate the messaging into its own service. Which isn't really how I want to work with my software. I guess the best way I can describe it is as a different aesthetic. There are certainly ways to accomplish the same thing, technologically, but they all feel wrong to me. Introducing a message queue, with a broker, and needing to implement HA on that simply to remove that single point of failure... That feels like a workaround to me. I prefer working with a decentralised pub/sub framework. Personally. Personally, that difference is worth the development time and lack of maturity. To me.

Which is why I slapped an alpha label on the project. Why I made no attempt to convince people they should switch over, or even that my way was objectively better than any other way.

I had an itch. I scratched it. I released the backscratcher I used as open source so other people with the same itch could scratch it. I'm not sure why some people think this is a contemptible act.

<comment><USER2> Bottlenecks can also be managed with various design and deployment strategies. There has actually been a lot of thought and time put into some of these products and they do have to solve the same problems you are concerned with.

Like I said, I'm glad you had fun with it. But you stated in your article, that there are not existing solutions without a single point of failure, and that simply isn't true. That's the only part I object to.

<comment><AUTHOR> I think we may have to agree to disagree here. You do not like the way I expressed my dissatisfaction with the previously available solutions, and I do not like they way you expressed dissatisfaction with my attempt to explain my predicament.

I stand by what I wrote, as it is the best way I can explain the, frankly, hard to describe issue I take with separating the messaging component into its own servers. If I had to rewrite it, I don't know how I could explain it any differently and get closer to accurately expressing myself. I'm sincerely sorry if you feel I've misled you or am misleading people; it is not my intention in the slightest. I can't imagine what I could conceivably stand to gain from such a deception.

I would, however, suggest that if people are taking their tech advice from me without doing any research of their own (which is what you seem to be suggesting, as you've commented several times that the information that you believe contradicts me is easily available), perhaps my inability to properly express my reservations with the previously available solutions is not the biggest issue at play here?

</comment></comment></comment></comment></comment></comment><comment><USER3> > _We couldn't seem to find any way to subscribe to events and publish events > without a single point of failure. We looked at everything we could find, > and they all seemed to have this bottleneck._

"The Spread toolkit provides a high performance messaging service that is resilient to faults across local and wide area networks. Spread functions as a unified message bus for distributed applications, and provides highly tuned application-level multicast, group communication, and point to point support. Spread services range from reliable messaging to fully ordered messages with delivery guarantees."

<http://www.spread.org/>

We use this for LAN event communication among standalone servers acting together, and WAN event communication among POPs acting together, for billions of events per month. Any server in a group dies, they elect a new master, so no SPoF.

What was it about your use case that made this feel like a single point of failure or bottleneck?

<comment><AUTHOR> This sounds like something I would have loved to come across. It didn't come up in any google searches or any conversations with other software people who I talked to about this problem, so I didn't evaluate it. Quite simply, I had no idea it even existed.

</comment></comment><comment><USER4> Cool idea, but I wonder why they didn't use something like RabbitMQ which already exists and is proven?

<comment><AUTHOR> I looked at it, I promise I did. Really wanted to avoid writing this if I could.

From what I can see in the docs, RabbitMQ is a client/server relationship. Meaning there is a server. Meaning a single point of failure and a bottleneck. I hate those more than I hate writing new software. Not because I think there is less danger of bugs in this than there is danger that the AMQP server will fall over... it just feels wrong to me.

I'm not saying, by any stretch of the imagination, somebody should use this over RabbitMQ. But if a distributed hash table is more suited to their needs, or if, like me, they're allergic to SPoFs and don't mind if the software is a little less mature if it means avoiding them... Well, now they have a package in Go they can use. If they want to.

<comment><USER5> 0mq doesn't have a client/server relationship. <http://www.zeromq.org/>

<comment><AUTHOR> I believe I looked at that as well. It was six months ago, so forgive me if my memory is a bit off.

Looking at the intro, it looks like I have to hardcode the IPs and ports of all of the machines I want to publish or subscribe to. Is that correct? If so, that's a little fragile for my tastes. I like "upgrading servers" by standing up new servers, testing they work, and then changing the floating IP associated with the DNS to point to them instead.

And I can hear you say "Aha! Floating IP! Just use that!" but for most cloud providers (I believe), that is billed as external bandwidth, which is not free like internal requests are. Finally, if I want to stand up more servers to scale horizontally, I'd still have to modify my code and redeploy it to all my servers. I think. Unless I'm missing something about how 0mq works. Which I totally could be.

On the whole, learning what I needed to learn to make Pastry happen made me a better programmer, too. So if for no other reason than that, I'm glad I did it.

<comment><USER5> No more so than with TCP sockets you have to "hard code" the IP connected to.

0mq is low level, it's not going to do routing like Pastry, but you can get away with having one fixed location for a "lookup server" that keeps track of everything else's location (something like the Doozer project, also in Go <https://github.com/ha/doozerd> ).

Also, 0mq only needs to know the IP of the publisher, for pub-sub.

<comment><AUTHOR> Very cool. Where were you six months ago? ;)

I was familiar with doozer, but doozer wasn't(/isn't) being actively maintained and it doesn't compile against the latest version of Go, so I'd have to bring it up to speed before I could use it anyways. Not saying it would be _harder_ , but between that and writing the 0mq library (pretty sure one does not exist in Go yet), I'd estimate the work would be more or less equivalent and yield just about the same stability. Seat of the pants guess, but it makes me feel better, at least.

<comment><USER5> Heh, thanks.

The mailing list suggests <http://github.com/4ad/doozer> and <http://github.com/4ad/doozerd> are being maintained as forks of the original. Or there's Zookeeper. Or brew your own central lookup server using 0mq that does the same job of "set a key, get a key, notify subscribers when it changes". It's a single point of failure, but since the system will run fine without it (only lacking topology updates) it's not a hugely problematic one.

<comment><AUTHOR> This reply link just appeared for me. Not cool -_-

The lookup server is a single point of failure. And while it's not a hugely problematic one, it also is a dedicated machine whose sole purpose is keeping the other machines running. Which feels wrong to me.

It probably wasn't the best business decision to invest time in this, and I won't even argue this is the best technical solution. But it's the only technical solution that didn't make me feel like I was working around limitations; things just worked they way they were supposed to. The API servers received an event, they told the WebSockets servers about it. It felt very conceptually pure to me. I'm a sucker for that.

<comment><USER5> To get the reply on a deep comment, click "link" first to get the comment as a single page, and "reply" will be there.

Doozer is only slightly a SPF, it has high availability by having multi-master replication. It's also not doing a lot of communicating, or a lot of CPU work, and it keeps its data in RAM, so it may be OK for it to live on a non- dedicated machine.

Sorry if I come across as criticizing your admittedly cool creation. It was just that you said there weren't alternatives, and I knew of one.

<comment><AUTHOR> No, I definitely appreciate the conversation, because I looked for forever for a pre-built solution to this, and couldn't figure out how nobody had needed a solution before now. So I'm glad there are solutions, and I'm not just crazy.

</comment></comment></comment></comment></comment></comment></comment></comment></comment></comment><comment><USER6> Should be aware of possible confusion with <http://en.wikipedia.org/wiki/Pastry_(DHT)>

<comment><AUTHOR> Confusion is intentional. This is just an implementation of that algorithm, with a few tweaks (that are explained, along with their reasoning, in the README). I believe I even link to that Wikipedia page in the blog post and the README.

<comment><USER7> I didn't see anything in either the Wikipedia page or the code, but how do you achieve redundancy? Wikipedia says nodes can die with minimal or no loss of data, but I see everywhere that each message only goes to one node. Which is the case, and how is this achieved?

<comment><AUTHOR> They both are the case. Well, sort of. The messages can go to any number of nodes as part of the routing, but they're only considered "delivered" at one node. However, nodes can die with minimal loss of data.

The problem is you're confusing the messages for data. As far as Pastry is concerned, your messages are not data that it should retain. They are one-off communications. The only data that Pastry is concerned with storing is the state of the cluster; that is to say, Pastry only considers "data" the information about enough nodes to reliably be able to route a message to any node in the cluster. Your messages, by default, are lost the moment they are sent, forwarded, or delivered. The node stops caring about them at that point.

However, on top of this framework you can build applications that _do_ care about the data in the messages. Before a message is forwarded, it fires off a callback to any application that cares to listen for it. When a message is delivered, it fires off a callback to any application that cares to listen to it. You can register callbacks that store this information, then build your own redundancy frameworks on top of that to store the data you'd like to retain.

If you're interested in this, I'd recommend paper on PAST. It's a high availability storage system built on Pastry: [http://research.microsoft.com/en- us/um/people/antr/PAST/hoto...](http://research.microsoft.com/en- us/um/people/antr/PAST/hotos.pdf)

<comment><USER7> Ah, I must have confused this for a storage algorithm, rather than a message passing algorithm. Doesn't "distributed hash table" refer to, well, a distributed hash table? One that stores key/value pairs? What good is it if the values are lost right after going in?

<comment><AUTHOR> That is what a distributed has table is. And the values and keys aren't lost-- there's just some confusion over what the keys and values are.

The keys and values, at least in Pastry's case, are the NodeID and the metadata about the Node, respectively. According to my understanding, at least. I had to learn a lot to implement this, so I would not claim to be an expert on distributed hash tables by any stretch of the imagination.

<comment><USER7> Hmm, so is the use case a distributed task queue? If so, why not use a consistent hash ring (<http://www.martinbroadhurst.com/Consistent-Hash- Ring.html>), or just pick a server at random? I'm afraid I'm missing the point entirely :/ Is it about maintaining a consistent list of member workers?

<comment><AUTHOR> A consistent hash ring is a similar concept, I think. We're quickly approaching the out-of-my-depth line, though.

It's about maintaining the property that you can route information from many different places and have that information end in a consistent place, no matter where it's sent from. And, assuming the consistent place doesn't go offline, a large number of servers can fall over without warning before that property is changed. It's about making discovery of nodes in your cluster reliable, and passing information in between them.

The use cases for that are many, and each has their own nuances, which is why this is so confusing and hard to talk about.

<comment><USER7> Right. It sounds like one of those problems that can be solved in many ways, and a DHT is designed to deal with a specific constraint. Thanks for the details!

</comment></comment></comment></comment></comment></comment></comment></comment></comment><comment><USER8> The focal point is discovery. Not other queues (or other libraries that can build queues). This is an interesting way to approach it, thanks for open sourcing.

We chose to solve the discovery problem a bit differently in NSQ (<https://github.com/bitly/nsq>) but I could certainly see some interesting opportunities to experiment with a distribute hash table approach as well.

<comment><AUTHOR> I was very excited when I saw NSQ. I thought it would solve my problem splendidly. It seemed, from my reading of the docs, that it did not support multicast however. :(

I was very impressed with it, though. You guys did an awesome job, and I'm extremely flattered you stopped by to comment.

<comment><USER8> It's a bit more explicit in NSQ but we essentially support multicast-like routing through "channels".

A "channel" receives a copy (at the source) of all the messages for a "topic" and has its own set of clients subscribed.

<comment><AUTHOR> Hm. Very interesting, I'll have to take another look and get a better understanding of it.

</comment></comment></comment></comment><comment><USER9> oh quiet you naysayers. just nod in appreciation of the hard work.

<comment><USER10> Interesting that the same group of people who were proponents and pioneers in NoSQL now dismissing libraries and new implementations to "solved" problems with "you can do that with {blank}"

</comment></comment><comment><USER11> Hijacking the topic on P2P: is there a good library (any language will do) for P2P message passing (as opposed to information storing in a DHT)?

I'd like to experiment a decentralized twitter/reddit-like system using p2p message flooding and machine learning to weed out spam.

<comment><AUTHOR> <http://tent.io/> ?

<comment><USER11> BTW, do you know of any real projects using Tent? It does sound interesting.

Going back to P2P: building a flooding P2P library on top of a DHT seems like a total waste. But maybe the common parts could be extracted.

<comment><USER12> Yup. There's a microblogging app (Twitter clone) called TentStatus, available hosted at <https://tent.is>, or run your own instance of the code: <https://github.com/tent/tent-status>. Several people are working on iOS and Android apps to support the microblogging use case.

There's Essayist, a (very alpha) long-form blogging app, hosted at <http://essayist.mndj.me>, code at <https://github.com/mwanji/essayist>

Meanwhile, the inventors of Tent have 5 people working 60+ hours a week running Tent.is and developing the protocol and apps for it. They haven't revealed what's next, though it sounds like some non-status apps are in the works.

</comment></comment></comment></comment><comment><USER13> They were already using Redis and made this? They could have just partitioned and replicated them for scaling.

<comment><AUTHOR> Yup. Totally could have made a choice I wasn't happy with.

Opted to make a choice I was happy with instead.

</comment></comment><comment><USER14> Next step: a tupplespace in Go?

</comment><comment><USER15> Oh man, I was really, really just hoping for this. I've been avoiding putting one last piece into my server because I didn't want to use redis. I'm going to play with this in a couple hours.

<comment><AUTHOR> Please do let me know how this works out for you.

And I mention it in the blog post, and I mention it in the README, but I feel I should reinforce this:

This. Is. Alpha. Please don't use it in your mission-critical code.

I'd like to run a lot more tests on it. I'd like other people to run tests on it and let me know what they think. This is my first time writing anything this complex, and I want any and all feedback I can get on it.

Thanks!

<comment><USER5> How are you planning to implement pub-sub on top of it?

<comment><AUTHOR> I'll be using the Scribe algorithm, as defined by the authors of the Pastry algorithm: <http://en.wikipedia.org/wiki/Pastry_(DHT)#SCRIBE>

The basic gist of it is similar to a phone tree. You route a message subscribing to a topic, basing the message ID on the topic name. As the message routes through the network, each Node stores the information about who is subscribed. When an event happens, that information is passed back through to the subscribed nodes.

I may have to tweak it a little, as I did Pastry, but I'm pretty sure the same basic concept will work for my needs.</comment></comment></comment></comment>


Harvard student database hacked, posted on BitTorrent - <AUTHOR> http://www.news.com/8301-10789_3-9893174-57.html?tag=cd.blog

<comment><USER2> Harvard (Haaahvuhd): we might leak your personal information, but at least we never use dangling participles in sentences.</comment>


Samsung brought Supreme faker up on stage at a launch event - <AUTHOR> https://arstechnica.com/gadgets/2018/12/samsung-embarrassingly-partners-with-counterfeit-version-of-supreme/ <comment><USER2> Glorious. How could their marketing team missed this? Don't they check credentials?</comment>


Dyn Takes a Public Stand Against SOPA - <AUTHOR> http://dyn.com/sopa-what-you-should-know-why-dyn-opposes-it/ <comment><USER2> In the last couple of weeks Dyn started forcing people to give them credit card information before being able to use the 10+ year old dyndns.org dynamic IP update service.

Seems like a bait and switch monetization scheme to me. Millions of routers have DynDNS support because it has traditionally been a free service.

It's a bit ironic that they are advocating for freedom on their blog but practicing vendor lock in with their product.</comment>