Are you ready? This is all the data Facebook and Google have on you - <AUTHOR> https://www.theguardian.com/commentisfree/2018/mar/28/all-the-data-facebook-google-has-on-you-privacy <comment><USER2> This is nonsensical:

> The data Google has on you can fill millions of Word documents. > Google > offers an option to download all of the data it stores about you. I've > requested to download it and the file is 5.5GB big, which is roughly 3m Word > documents. This link [Google Takeout] includes your bookmarks, emails, > contacts, your Google Drive files, all of the above information, your > YouTube videos, the photos you've taken on your phone, the businesses you've > bought from, the products you've bought through Google …

Well duh, Google stores your gmail emails and Youtube videos and Drive documents and so on! This is the service they're providing you! Google Takeout doesn't give you a 5GB archive of advertising profile or covertly gathered info, it gives you a 5GB archive of the documents you deliberately uploaded to Google services - emails, docs, pics, videos, calendar appointments. The list of Android apps installed on your devices is there so that when you link a new device to your Google account, these apps will be automatically installed on it. And so on.

There's a single link on that list related to surveillance - namely, your Google Ads profile, which only shows very basic data, I'm sure they have vastly more complicated data stored on me.

There's enough genuine reasons to be upset with Google's surveillance, let's not promote this kind of silliness. Google knows which Google Groups you're subscribed to, really? 5GB could fill 3m Word docs - how many is that in Libraries of Congress?

<comment><USER3> This is typical. As a techie, you're triggered by technical detail because you understand the technical details of that example that is intended to illustrate the size of the data collected.

Rest assured that this happens all the time and when people read news about a topic that they are an expert on, they cringe all the time. For example, a paraglider pilot will get triggered if the reporter uses the word "Jumped" in a story about a paraglider pilot who crash-landed and injured two more people. This is because paragliders don't jump but take off from hills etc.

Usually reporting on technical topics is very hard because you need to reach the non-technical audience so you will HAVE TO use illustrative examples and not exactly accurate descriptions so that can be absorbed by the non-technical readers. Unfortunately, this will put you in a position where can be called on by technical readers who are triggered by the "implementation details" of your story.

I do believe that calling out the 5GB illustrative example is missing the point of the article. It's just a way to portray size of the data collected on you, it doesn't have any meaning even if this data was 3Mb or 100Gb. For example, if it was 3Mb Google might have used it to convey how little data they collect on you as if the size that this data takes has any correlation about the impact, then why even bother with such an irrelevant point of the "article implementation"?

<comment><USER4> The word "triggered" should really be reserved for military vets and those who have experienced traumatic events in their past that would cause severe distress, not as a synonym for "annoyed."

A veteran who relives the horrors of war when fireworks go off is not nearly in the same space as a developer who is annoyed that someone misused technical jargon.

<comment><USER5> > The word "triggered" should really be reserved for military vets

Certainly not. While many military personal certainly have faced mind-altering situations, they are far from the only ones who have ever suffered from traumatic events.

Children and Women are the subjects of the most twisted and abusive stories I've ever been able to fathom.

<comment><USER6> The full quote is:

> The word "triggered" should really be reserved for military vets AND those > who have experienced traumatic events in their past that would cause severe > distress

(emphasis mine)

You can't just cut a sentence in half and then try to form an argument against it. That's not fair.

<comment><USER5> I agree, I don't think it said that when I wrote my comment.

<comment><USER4> It likely did not, I edited a few times.

</comment></comment></comment></comment></comment></comment></comment><comment><USER7> NO, NO, NO. This is not _all data Facebook /Google has on me_. It's way less.

What I don't get in all of this debate: People grant FB/Google/any other app access to their data and seriously expect that their data isn't harvested? IMHO, there really is zero surprise that Facebook has your call and SMS history, if you granted access to that. Or every single login timestamp.

What is way scarier is that the dump you can download is probably far from complete. My Facebook dump shows rather few items, because I'm not a heavy Facebook user. But I'm quite sure that they have A LOT more on me, because of friends uploading my phone number, profile pictures, _meta data_ from the WhatsApp acquisition.

The big scary question is: How do you access the data in your shadow profile that they won't give to you? How do you even _prove_ that they have more on you than they admit?

Edit: Same for Google. I don't use a Google account most of the time and I bet they STILL have my entire search history associated to some kind of shadow profile.

<comment><USER8> people don't know they are granting permissions.

when you provision a new android device it will try to link your account to chrone, and its very hard even for a developer to catch its chrome and not android os. after that small mistake (which is an optout in the ui pattern) its game over. google have all your history and every webview ad has your full profile. not to mention your web bank passwords are now conveniently save in google servers in plain text.

and yes, an even worse problem is the hidden data/shadow profile.

<comment><USER9> "passwords are now conveniently save in google servers in plain text"

Do you have a citation for this?

<comment><USER10> Not the parent, but here is where they store your chrome passwords: [https://passwords.google.com](https://passwords.google.com)

</comment></comment></comment></comment><comment><USER11> The section labeled "Google has information you deleted" is a bit misleading.

If files are in your "trash" they will be included in your takeout. If you clear your trash, they will not be included in your takeout (after a reasonable amount of time, if you "empty trash" then instantly go and make a new takeout bundle it might still be included). It's the same way with spam and email trash. It's given to you in your bundle until you manually fully delete it.

Implying they keep everything you ever deleted is wrong.

Edit: I don't want to get into hypothetical arguments about what Google could possibly be doing. I'm simply saying that the reason the author had "deleted" things in their takeout archive was because it was most likely in their "trash" and wasn't fully deleted in the UI. And that "things in your trash folder are included in your takeout bundle" isn't "proof" that they never delete anything.

<comment> wu-ikkyu How do you know they're doing a hard delete server side and not just flipping a delete flag?

<comment><USER11> You don't.

But from what I have seen Google takes infosec very seriously, and will fully delete information when they say they delete information. They even point out that it could "linger" for (IIRC) 90 days after you delete it in backups and other replicated copies, but after that should be fully and completely deleted.

<comment> wu-ikkyu Hopefully they so take infosec as seriously as you assume they do.

However, we have repeatedly seen profits prioritized over security in the corporate world with negligible repercussions when the data is inevitably compromised.

<comment><USER11> I knowingly and happily give my information to Google, and I trust that they do not overstep their bounds when I tell them not to collect something.

If Google ever gives me a reason to not trust them I won't. But thus far they have been okay in my mind and I'm happy with the services I received and the tradeoffs made with regard to my information.

I fully expect more nastygrams and threats from people for saying this, because it happens every time I say I trust a company.

<comment> wu-ikkyu Do you think Google has never been hacked or compromised by an insider?

<comment><USER12> Do you think that major companies don't have systems to mitigate this threat?

<comment> wu-ikkyu None that are invulernable to an APT or an inside attack.

Saying "I trust company X with my data" is the same as saying "I trust that company X cannot be hacked".

"There are two types of companies: those that have been hacked, and those who don't know they have been hacked."

-John Chambers

<comment><USER11> >Saying "I trust company X with my data" is the same as saying "I trust that company X cannot be hacked".

No it is not, and it's also why I didn't reply at first because I had a feeling it would go down this path.

I don't believe for a second that my data will never be "hacked" at Google (for whatever definition you want to think of for "hacked"). But I do believe that Google has done their due diligence in preventing that kind of attack to the fullest extent they can, they give me the tools to remove data that I want removed, they are competent in their architecture to make leaks and hacks have limited scope, and i'm confident that they will be able to uncover evidence of "hacking" and will use the legal system to go after those responsible limiting the damage that a "hack" can do.

I enjoy the benefits I get from Google. I like that they scrape my email for calendar info, flight info, package info, etc... I like that they track my location and create automatic albums for pictures I take (and upload to them) while at a location. I like that they can get location indoors using wifi APs or that they use my voice in ML training to improve the product, or that they offer me relevant ads to pay for those free services and products instead of ads that have little or nothing to do with my interests. I like that I can go back and search my hangouts (then google talk) chats from my friends from over a decade ago, or that they backup my files that i put on google drive, or that they record what apps are on my phone (and some data from those apps) so that if I need a new phone I can quickly set everything back up, or that they store saved passwords on their servers so that I can easily get the same ones on multiple devices, I like that my reviews of apps/places can have my name and face put on them and that reviews from my family and friends show up over random people online, etc... And I especially like that I get most of this for no effort on my part. No servers to maintain, no software to setup and manage, no security needed on my part aside from keeping my username, password, and second factor auth secure. I only have a limited time on this planet, and I don't want to spend it setting up private email, storage, photo backup utilities, and whatever else that I get "for free" through Google services.

I'm not being tricked here, i'm not "missing something" or pretending that these companies are infallible. I'm making a consensus decision to trade information about myself to a company I trust for tangible benefits. If that company becomes untrustworthy, then I will adjust my feelings and behaviors, and I will most likely be hurt by it at some point (because you can't un-give information, especially not to a bad actor), but again that's a risk i'm willing to take.

I don't know why I typed all this out, and I don't mean to target it all at you personally, just at the boogeyman I've built up in my head of "internet person telling me i'm dumb for trusting a company". It's exhausting constantly having to defend against what-if's because I want webmail, or being constantly berated and insulted for making tradeoffs with my own privacy like it's some kind of personal insult to people that I am not as private as they are (there's nothing wrong with privacy, I advocate for it quite a lot, but your whole life doesn't need to be private all the time, i'm happy to share some aspects!).

<comment> wu-ikkyu Fair enough, you seem much more knowledgable of the risks (at least on an individual level) than the vast majority of users.

However, have you considered the threats to national and global security that are posed by the centralized aggregation of billions of the most detailed psychological data profiles the world has ever seen?

</comment></comment></comment></comment></comment></comment></comment></comment></comment></comment><comment><USER13> I feel a disclaimer that The Guardian uses Google Ads, Google Analytics, and Google Tag Manager on the site to track you and help Google further track you should probably be added to a scaremongering article about Google tracking the user.

I feel most technical people are aware of the level of tracking going on. The more privacy concerned do what they can to mitigate it.

<comment><USER14> Yeah, but I find it kinda hard to keep up with their development speed. Every year or so, I find out about Google having introduced a new option which is on by default and that Google has started collecting data from me again.

</comment></comment><comment><USER15> But I willingly give my data to Google - they have (for some reason or another) kept my trust.

Facebook broke my trust with data I didn't know they had collected.

<comment><USER16> What data does Facebook have on you that Google doesn't collect?

Google buys your credit card purchase data. Did you give it that? Why do you think Google wants you to stay logged into Chrome?

Let's be honest here and call a spade a spade.

The world is just realizing how ads based business models work and there's no difference between Google and Facebook.

<comment><USER17> > _Google buys your credit card purchase data._

A more accurate explanation of the scare quote:

[https://news.ycombinator.com/item?id=16698587](https://news.ycombinator.com/item?id=16698587)

> _IIRC, this was announced as part of a program that uses double-blinded > crypto that allows google to perform matching against this data in aggregate > (Set intersection knowledge), but not individually identity of what a given > user purchased, at the same time preventing the merchants from learning the > identities of individuals either._

> _According to what they presented at a crypto conference, they went to great > pains to avoid capturing personally identifiable transactions. The overall > strategy is that Google Knows = Set A = (all ads shown for user X at > Merchant Y) Merchant /Credit Card Proccessor Knows = Set B = (all $$$ paid > by User X at Merchant Y), and that it is possible to cryptographically > compute sum($$$ of A intersect B) without either side learning Set(User X)._

> _Now whether you believe in the security of the algorithm and whether there > are side channel leaks or attacks is another issue._

> _Edit:_ [http://bristolcrypto.blogspot.com/2017/01/rwc-2017-secure- > mp...](http://bristolcrypto.blogspot.com/2017/01/rwc-2017-secure-mpc-at- > google.html)

> _About 10 minutes in_ > [https://www.youtube.com/watch?v=ee7oRsDnNNc](https://www.youtube.com/watch?v=ee7oRsDnNNc)

</comment></comment></comment><comment><USER18> My google activity history, youtube search/watch history, and location history are all completely empty because I've disabled all of it. I would recommend going through the privacy checkup[0] to control what does and doesn't get saved.

[0] [https://myaccount.google.com/privacycheckup](https://myaccount.google.com/privacycheckup)

<comment><USER19> I also did it some time ago, but I have a feeling, with nothing to backup my claim, that they still log, store and analyse all my activity but just don't show it in the dashboard. I don't know how targeted ads are (because of ad block) but youtube doesn't suggest me anything based on watch history so I guess I might be wrong here.

<comment><USER20> One thing I like to do is just leave them all on, but minimize my contact with Google's services. Then I check in every few months just to confirm they haven't collected anything on me.

<comment><USER19> Off topic - I am actually working on a framework to automate the process of self-hosting these services on some VPS. I might open-source it once it is in usable shape. But some things just have no real alternative(like maps and android)

<comment><USER20> Apple Maps is actually acceptable I think (at least on iOS). I'm trying to find an alternative for Google Photos.

</comment></comment></comment></comment></comment><comment><USER21> This person stored their PGP private key on Google Drive, which they use "to encrypt" emails. I'm glad the author is security-conscious enough to know what PGP is, I guess. And that they deleted it from Drive (didn't say why, hopefully they came to their senses about the more-than-usual amount of trust they were placing in Google).

Is there a valid use case for "encrypting" email with a private key? I guess signing a message is technically encryption, but it's not typically called that, right? I'm hoping their use of "encrypt" in this context is just an error, and that tech reporters at the Guardian understand how key pairs work...

<comment><USER22> One of PGP's functions is to encrypt email. Why would you expect someone be savvy enough to understand how PGP works but not be savvy enough to use its main function?

[https://en.wikipedia.org/wiki/Pretty_Good_Privacy](https://en.wikipedia.org/wiki/Pretty_Good_Privacy)

<comment><USER23> When you encrypt an email with PGP, you use the public key of the individual you are writing to, not your own private key. You should only use your private key to read the mail encrypted with your public key, or to sign messages.

</comment></comment></comment><comment><USER24> Is this really surprising to anyone but the laymen? I mean, of course they store everything the user does, why wouldn't they?

<comment><USER25> > Is this really surprising to anyone but the laymen?

That's a pretty huge caveat. "This isn't news, except to the majority of people". It's an article in The Guardian, it's obvious intended for the "laymen", of which there are a pretty big number.

> of course they store everything the user does, why wouldn't they?

That's the crux of it, for me. I agree, they have no incentives not to hoard data. So those incentives, like GDPR, should be created.

<comment><USER26> > That's a pretty huge caveat. "This isn't news, except to the majority of > people"

TL;DR defaults matter to the layman - the 99%, google will say they gave you the options (so many that it's tiresome for most people to bother going through).

Yup, when reading through all of the stuff I was just checking off my mental list of all the things that don't affect me. I don't use google on phones, and for gmail stuff i've already trawled through the massive list of options to stop tracking and logging various stuff. Additionally I only log into a google account in isolation so it can't join history together.

Even after all of this I recognise that google mostly likely has other tricks up it's sleeves like browser fingerprinting to join sessions together and associate their activity with my account in secret, not to mention they are mining all of my email of course, but this is the best I can do while continuing to use a google account (and even without a google account they can still track individuals via other means).

This is only practical for those the most aware of what these companies do and the most interested in preventing - understandably, the layman just wants to use it as intended without doing days of research and digging through piles of options in paranoia trying to turn everything off, these types of services rely on this fact. I am not a normal user and they really don't care that 1e-10% are evading them, they are relying on the fact that almost all of their users wont be going to great lengths to turn all the tracking and logging options off.

</comment></comment></comment><comment><USER27> This is where a legit issue start to turn into clickbait weak journalism.

<comment><USER28> I wouldn't be so fast to dismiss it as clickbait. Do you really think non-tech people are aware of the amount of aggregated data these companies have on them?

<comment><USER29> As pointed out elsewhere in this thread, the article uses scare tactics that misrepresent what google is actually storing.

They are storing your emails, for instance, and your photos.

This isn't google tracking you any more than a storage unit company is stealing from you.

<comment><USER30> What percentage of users do you think fully understand that Google is literally tracking their every footstep? Yes, it's Google tracking you whether you realized it or not. Tracking is not and never has been "opt-in".

</comment></comment></comment></comment><comment><USER31> I'm more interested in what they collect about me that they're not showing on a user-friendly dashboard.

<comment><USER32> \+ what do they collect when you are not logged in.

<comment><USER33> or don't have an account.

</comment></comment></comment><comment><USER34> _" We would never let the government or a corporation put cameras/microphones in our homes or location trackers on us. But we just went ahead and did it ourselves because – to hell with it! – I want to watch cute dog videos."_

You can get people to do anything with the right incentives.

</comment><comment><USER35> Took advantage to go onto Google and give it even more info on what ads they should (and should not) show me. That's probably not what the Guardian expected/wanted me to do.

I may be alone in this, but I expect that everything I do on the Internet is done in public. Nothing new here.

<comment><USER25> > I may be alone in this, but I expect that everything I do on the Internet is > done in public. Nothing new here.

So how do you send private messages? Carrier pigeon? Even Signal uses the internet, after all.

<comment><USER36> I bought an entry level fountain pen, some ink, some lightly cotton blended paper and stamps.

Then I got hooked.

I learned how to fold paper into its own envelope. I bought supple wax and sent a design for a custom wax seal stamp, so now I can mail a letter that is wax sealed domestically or internationally.

My friends love getting letters in the mail that's not spam. And a few even write back. One made his whole envelope out of duct tape.

We still call and text, of course.

And now I am practicing calligraphy in order to craft my own book, binding and all. I got very inspired when I saw the Book of Kells and Chester Beatty collection in Dublin.

Practically this has cost me (the fountain pen setup plus forever stamps) $100 USD. But that's because I bought a nice ink. So I think if people have a little bit of disposable income, and have Netflix to cut out of their life, could pick up old fashioned writing to catch up on life's details in private.

Heck, could even skip the writing and just buy the envelope and stamps, and type up a letter to print and mail!

<comment> at-fates-hands I totally subscribe to this novel idea. However, I hear rumblings about doing away with the USPS for various reasons. It's nice to know I'm not the only one still using some "analog" methods to communicate.

</comment></comment></comment></comment><comment><USER37> How feasible is it for Google/Facebook to infer/build profile of private browser activity?

Since most sites implement google Analytics and Facebook tracking code (ghostery is reporting both on the guardian website). They could use browser fingerprinting, which has already been proven to be extremely accurate, to augment known user profiles and data. Facebook already creates shadow profiles for people without profiles. I don't think there's any US law from making these inferences from "voluntarily" submitted information so it's perfectly plausible they would do this without telling people, no?

A more compelling argument for users to take privacy seriously would be to tell them: Hey guess what Google has the ability to know what porn you search for (yea even in your private window) since you're sending them all the data required to track you, and there's no laws in place to prohibit it.

<comment><USER38> I'm presuming that you're not talking about Firefox's Private Browsing mode, which comes with an actual tracking script blocker.

The biggest factor for tracking without the ability to store things on the client is IP-based tracking. In many cases, the IP address narrows the number of possible persons down to a handful. Then you can use behavioural analysis, fingerprinting or factor in other data to narrow it down those last few meters.

And well, any time you visit a webpage which has a DoubleClick ad on it, uses a font from Google Fonts, incorporates JQuery from Google's server, incorporates Captcha, Maps, YouTube, Custom Search Engine, Google+-Button from Google's server, etc., or in fact, even though this is the least of your worries then, Google Analytics and Google Tag Manager, any time one of these is on a webpage you visit - which is pretty much every time you visit any webpage, your IP address is sent off to Google.

And now comes the bad part: As is information about what webpage you're currently on, as part of the HTTP Referrer.

And I mean, if your URL is not resolved by 220.127.116.11, you can consider yourself lucky, too.

So, Google effectively has your complete browsing history, whether you're in private mode or not.

Facebook is not nearly as bad in this respect, but their Like-Buttons are rather widespread, too, and they do have an analytics framework, as you've already pointed out.

<comment><USER37> I would like to add that not only does Facebook have like buttons everywhere, many websites add Facebook analytics code as well to track performance of their ads.

</comment></comment></comment><comment><USER39> This headline is patently false, Facebook is known to purchase information on people from Data Brokers and none of that is included in their data downloads.

Things like MAC address logging and such is also missing.

The only things included in the download are things you've given them; pictures, messages, likes, contacts, web browsing history, and phone metadata if you gave them permission on your phone.

If they were able to purchase phone metadata from your service provider, it won't show up there.

If all Facebook kept was user-submitted data it would be orders of magnitude less creepy than it is.

</comment><comment><USER40> OMG google has been storing all my email?!

Creepy, that's what they said they would do when I opened my gmail account.

<comment><USER41> However, they have also been capturing your webcam as you type out each email. Please see document #3015698 for the top 3 most likely things you will eat for breakfast over the next week over what is most likely in your fridge right now

</comment></comment><comment><USER42> Now if a malicious actor gains access to your login credentials they also gain access to this rather unique data..

</comment><comment><USER43> The Google timeline has surprisingly missed a log places that I have been. They are either not telling me about it, or they don't store everywhere I go. [https://www.google.com/maps/timeline?pb](https://www.google.com/maps/timeline?pb)

<comment><USER44> If your laptop doesn't connect to anything, it has no idea where you are.

If your phone doesn't connect to anything and enable location services (GPS is costly in terms of battery though), it has no idea where you are. Of course, your phone -is- connected to the cell network most of the time, and so theoretically it could calculate where you are to within a mile or so, but that, 1. Takes battery, and 2. Isn't that accurate. So there's a good chance it wouldn't bother.

</comment></comment><comment><USER45> "The data Google has on you can fill millions of Word documents" What does this even mean?

<comment><USER41> Ordered word documents, one bit stored in each file that is reconstructed in sequences of signed bytes

</comment></comment><comment><USER46> This misses the wifi router data Google collects via android phones. Android is a sieve designed to enable a spyware economy like the rest of Google who are creepily stalking every human unfortunate enough to use their services.

But most here know about this, many are intimately involved in it and are happy to defend it. Outrage targetted at Facebook while making excuses for Google feels inauthentic.

Everyone wants an ethical society but when it comes to paying a price economics always wins out and we are left with the spectacle of the privileged hand wringing, blaming the system, blaming the victims and acting helpless. If you can't be ethical you can't expect ethical behavior from anyone else and you get the society you deserve.

</comment><comment><USER47> I was actually surprised going down the list at how little Google has on me. I mean, I've used Duckduckgo for a long time and never used Android, but forgot I had turned off ad personalization and deleted youtube and search history.

I'm surprised they didn't have anything in search or youtube because I deleted the history long enough ago that I didn't remember doing it, so I've successfully stayed away from using them.

Still waiting for the data download, but I know they have all my gmail messages and some stuff I put on Drive a long time ago.

<comment><USER48> Do you think they actually delete your history when you turn those features off? I assumed they just stopped displaying it to you.

<comment><USER49> They actually completely wipe out. This was discussed before by Google Sres.

</comment></comment></comment><comment><USER50> So I clicked on several of the Google links in the article to see what info they had on me, and all of the resulting pages prompted me to sign in to Google to see the info. (Which I did not then do.) So... they're only gathering this info on people who use Google services while signed in? It's not like you need to be signed in to use search or maps. I thought that this article would be about how Google tracks you _without_ being signed in, and even if you clear your browser's cookies.

</comment><comment><USER47> By making this data so easy to see (which I admit is really nice) Google makes it that easy for other people to see it if they get your account log in. Sort of Scylla and Charybdis here.

</comment><comment><USER51> I'm unable to see my historical Google location data on iOS via

[https://www.google.com/maps/timeline?pb](https://www.google.com/maps/timeline?pb)

I'm getting the following error:

"Location Services permission is not set to always. Timeline cannot function properly if you don't allow Google Maps access to your location while in background. [Skip][Turn On]"

Seems like I should be able to see what historical location data they have without enabling background location.

Disclaimer: I work for Apple

</comment><comment><USER52> Evidently we're at the outraged-explainer-porn stage of the media manufactured panic cycle. They are certainly using every part of that buffalo.

<comment><USER53> And you're at the superior-to-all cynic portion of your comment history.

</comment></comment><comment><USER54> What kind of noob "data consultant" must you be if you didn't know years ago you can opt-out of most of the google stuff? Google allows to deactivate/delete (we dont know if they do but at least its offered) most of the stuff he wrote about. Facebook is its own story, but then again..his own fault for using and pushing data up there

</comment><comment><USER55> ...and once again, an article that could have actually gone in depth on this topic, manages to leave out the data these companies collect and purchase from third parties (e.g. credit card transactions) and public records.

An accurate title for this piece would have been "This is all of the data Facebook and Google are willing to tell you they have on you"

</comment><comment><USER56> >Somewhat pointlessly, they also store all the stickers you've ever sent on Facebook (I have no idea why they do this. It's just a joke at this stage).

It's pretty obvious Facebook needs to store the stickers you sent if they store your chat history. They come in their own directory in the bundle so they can be shown in the chat HTML pages.

</comment><comment><USER57> Both of those services are free of charge. What did people think was going to happen? I really don't get the huge amount of paranoia these stories have been generating recently. You're using an SaaS platform, so yes, they have your data. "Your data" is literally their business model.

</comment><comment><USER58> Kind of shitty that they didn't mention that you can delete the data from your Google account and disable further data collection (at least for the stuff mentioned in the article). I guess that would have made the article more informative and less scary.

<comment><USER48> Can you? Does that actually delete data from Google's database? Or does it just flip a switch so the won't serve you "personalized" ads based on that data?

<comment><USER58> I don't know what really happens when you click the delete button but I assume it does what is says. Why would they offer to delete the data and then not do so? Some intern finding out about that would cause a PR disaster and maybe even legal trouble. Doesn't seem worth it, they could have just not added the feature then.

</comment></comment></comment><comment><USER59> The fact that Google stores deleted documents from my drive makes me kind of uneasy... why would they ever need this? Speaking of which, does anyone know if the contents of your drive is used for your ad profile?

<comment><USER60> Possibly they were in this users Trash folder, otherwise it's worrisome.

</comment></comment><comment><USER61> I just thought of something, all the attention that this is finally getting means you can go dark without raising suspicions.

</comment><comment><USER62> Are you ready? Hold onto your seats folks! This is an article whose aim is to spread fear, uncertainty and doubt.

<comment><USER47> It's funny, I bet Google is glad this article is out there. Look how easy they make it to see "everything" they know about "you". And you can delete it if you want!

When they are thinking about this article, no one will worry that you can't see any profile they have for you when you aren't logged in.

</comment></comment><comment><USER63> "Google has every email you have ever sent using Gmail"

Err ya that's the service right :P

</comment><comment><USER64> Has any novel useful insight on humans ever been deduced from all this data?

</comment><comment><USER65> with all the data they have on us WHY IS IT STILL SO DIFFICULT TO PAY TAXES.

Why can't this be easier ? Why can't google produce a tool for this ?

<comment><USER66> I believe the primary reason is that there's a pretty big industry around tax preparation that's making that difficult.[0] I've also heard arguments that there are anti-tax motivations that want to keep it difficult so people view taxes more negatively. I don't have any info on the latter, so take that with your daily recommended allowance of sodium chloride.

[0]: [https://techcrunch.com/2013/03/27/turbotax-maker-funnels- mil...](https://techcrunch.com/2013/03/27/turbotax-maker-funnels-millions-to- lobby-against-easier-tax-returns/)

</comment></comment><comment><USER67> Privacy concerns are good and should always be raised. But before panicking like anything remember who gave them the data in the first place.

</comment><comment><USER68> Wow, that was an eye opener.

</comment><comment><USER69> It's okay, I only use Incognito mode.

<comment><USER45> That mode doesn't do what you think it does.

<comment><USER69> My joke was poorly received. I'll do better next time.</comment></comment></comment>


Julie Rubicon - <AUTHOR> https://www.facebook.com/notes/robin-sloan/julie-rubicon/985697811525170 <comment><USER2> Post is fiction. Spoilers follow.

.

.

When I started reading this, I didn't realize it was fiction. When I got to the point where the protagonist left the end-date off a query and saw a spike, I thought it was going to be explained by falsified data and lead into a (real) accusation of providing fraudulent metrics to advertisers. It wouldn't be the first such accusation. But it turned out to be a science-fiction story with unexplained time travel in it. Oh well.

<comment><USER3> I read 2-3 paragraphs into this and then thought, hmm, maybe I should check the comments first.

Thank you.

<comment><USER4> For once I'm glad I didn't read the comments before reading the story. jimrandomh comment would have spoiled it.

<comment><USER2> Edited to add a spoiler warning. My apologies to anyone I spoiled the ending for.

<comment><USER5> Not effective. Your spoiler is telling us that it's fiction. The delight would have been in that surprise realization, not the nitty-gritty details of the story.

</comment></comment></comment></comment></comment><comment><USER6> Fiction or not, this is similar to an actual story of (ex)-fraud researchers at Capital One[1], who (ab)used their access to credit card transaction data in order to infer whether target companies' quarterly earnings would be below/above expectations, and then traded on that knowledge.

However, in the actual story, there was no "black box" to query, the ex- employees wrote all the complicated queries themselves.

EDIT: They turned ~$150K into ~$2.8 MM USD over about three years, before being caught, mostly through options trades it seems.

1\. [http://www.bloombergview.com/articles/2015-01-23/capital- one...](http://www.bloombergview.com/articles/2015-01-23/capital-one-fraud- researchers-may-also-have-done-some-fraud)

<comment><USER7> Can someone explain why this is illegal (besides any data/privacy breach within Capital One)?

<comment><USER8> Possibly something to do with privileged information. Keep in mind that even things such as intent may get you on the wrong side of the line*

* example: you post a trade but you don't intend to execute it but instead hope to fool the market, then you cancel the trade and switch to other direction. forgot what that's called exactly

<comment><USER9> [https://en.wikipedia.org/wiki/Pump_and_dump](https://en.wikipedia.org/wiki/Pump_and_dump)

</comment></comment></comment></comment><comment><USER10> Usually I read comments on HN before the article, but there were no comments when I saw the link, so I went straight to reading it.

What a bizarre piece. How long did it take everyone else to realize it was fiction? For me, I did not realize until the very end -- and even then I still wasn't sure. It could just as well have been written by some delusional non- technical employee at Facebook.

<comment><USER11> I started to wonder if it was real around the point that it theorized without explanation the predictive powers of the in house tool and I was pretty convinced it was fiction about the point where they were going to start a company within a company using bitcoin.

Using Facebook internal data to trade in ways no one can match isn't far fetched. Makes me wonder if Facebook should pivot to be a hedge fund based on signals no one else has access too.

<comment><USER12> Given that the posts can be from folks that work in the company Facebook would be trading stocks for, that would count as insider trading.

<comment><USER11> That's if you use insider information. That's only one kind of signal that Facebook has access to. There are signals that aren't based on insider information that FB could legally trade on.

Information that other people don't have access to isn't by definition insider information. It's just stuff others can't or haven't figured out.

</comment></comment></comment></comment><comment><USER13> Worth noting that Robin is also the author of Mr. Penumbra's 24-Hour Bookstore [1] that others around HN would probably find pretty enjoyable. I thought it was quite fun.

1 - [https://en.wikipedia.org/wiki/Mr._Penumbra%27s_24-Hour_Books...](https://en.wikipedia.org/wiki/Mr._Penumbra%27s_24-Hour_Bookstore)

<comment><USER14> Yep! I'd recommend reading the short story, skipping the novel:

[https://www.robinsloan.com/books/penumbra/short- story/](https://www.robinsloan.com/books/penumbra/short-story/)

I felt like the book was mostly filler added after the popularity of the short story.

<comment><USER15> Ha -- the short story was, sorry to say, not _that_ popular ;)

<comment><USER14> Hi!

Popular enough to turn it into a book? :)

Well, maybe it was because I'd read the story before (and really enjoyed it) so I couldn't get the same enjoyment from the book.

Anyway, good work and keep writing.

</comment></comment></comment></comment><comment><USER16> I'm "technical" and "hands-on", yet I was gullible enough to believe the story, IM-ed it to my wife, and was surprised when she hadn't panicked like I did.

I read way too much science fiction. Back to work now!

Kind of relieved though.

<comment><USER17> Sigh I did the same, sent her the link as I was reading the article halfway.

Once I finished, when they repeated the name Julie I scrolled up, came back here and boom, fiction. lol

</comment></comment><comment><USER18> "Published on the day Julius Caesar was murdered 2060 years ago, after crossing the Rubicon. Julie Rubicon. Nice touch."

</comment><comment><USER19> Nice piece. When I saw that first from-future graph, I thought Facebook is faking their data. As they are known for faking page likes.

<comment><USER20> same. when i saw this line:

'I spent the day haunting the roof of MPK20, feeling jangly and nervous, staring out across the bay.'

i thought it was about her confronting the ethical dilemma about whether she should rat out on her employer or not. it feels more likely you would accidentally disclose your fake data generator than your future predictor.

</comment></comment><comment><USER21> I think the HN audience will "see through" this story, but anyone who isn't familiar with the current state of neural nets/has read the many pieces fortelling the future of AI might just find it plausible. I really liked the writing style.

</comment><comment><USER22> The first graph immediately looked too fake. Spike after an event would follow something like a log normal distribution, rather than sudden spike and day after back to normal. What's more, the world is not equally using the internet. The uneventful graph should follow something like a sin wave, with the highest point 2x higher than the lowest.

</comment><comment><USER23> It is a fun story and one that comes at the question of "how much data is too much data and is too hard to resist?" Something that anyone who has worked in a popular Internet facing application has to come terms with. I found 'Manna'[1] a more compelling emergent AI story, but the data as predictor aspects of this story have their own particular flavor.

One of the things that immediately flashed it as fiction for me was that the graphs had all the same shape, which if you've ever looked at trend graphs you will see they might all have a similar outlier quality to them but they build and sustain in different ways.

At Blekko (a search engine company) we did an interesting study on query traffic to see if you could "predict" the "hallmark" holidays based on search queries. The idea was that holidays like Valentines Day come up, people start thinking about plans or gifts before that, could we advise an advertiser when the "peak" planning session was so that they could maximize the impact of their advertising spend by focusing it during the peak? And if so what sorts of queries were people making that indicated they were doing holiday planning? The results were mixed. For things like Valentines it was easy, flowers, chocolates, bed & breakfast reservations sort of rose out of the general query stream, St. Patrick's Day? Not so much. But the data peaks all had different shapes appropriate for different levels of impact (Christmas shopping really starts in August among the back to school traffic for the really prepared). So looking at (and for) "interest spikes" like the ones in the story had a bunch of different shapes, some with slow onset and rapid decline, some with rapid onset and rapid decline, and some which were like soft swells on a breezy afternoon at the beach.

That said, the dataset made possible by Facebook's chat stream would be even better for those sorts of investigations.

[1] [http://marshallbrain.com/manna1.htm](http://marshallbrain.com/manna1.htm)

<comment><USER24> I don't want to contemplate a world where Christmas decorations are being put up in stores in August. Them being installed before the start of December is too early already (for me).

</comment></comment><comment><USER25> The funny thing is that if the implications noted in the story were true and Facebook could accurately forecast events into the far future, building a social network would be the _least_ of their priorities.

<comment><USER26> How do you know that's not true today? Any effort to expand the social network would just be helping the prediction engine in reality.

</comment></comment><comment><USER27> This is some fantastic writing.

And if you are looking for a longer, slightly better, fictionalized account of Facebook, "The Circle" by David Eggers is a quick, engrossing read and quite hard to tear yourself away from once you begin.

[https://en.wikipedia.org/wiki/The_Circle_(Eggers_novel)](https://en.wikipedia.org/wiki/The_Circle_\(Eggers_novel\))

</comment><comment><USER28> Great piece of fiction, it feels like the author works at Facebook or knows someone who does though.

That said, what's described could possibly be done with existing technology . Could Facebook accurately move forward statements like "next Monday is going to be massive for Volkswagen" over time you could weight private messages from people who work for regulators higher. For more public events like an apple launch they could predict a spike easily just by all the media mentions of the date

</comment><comment><USER29> Haha..it's like a modern campfire ghost story. It's written in a believable manner too. It had me going. I like it. Good job!

</comment><comment><USER30> reminds me of the movie primer: [https://en.wikipedia.org/wiki/Primer_%28film%29](https://en.wikipedia.org/wiki/Primer_%28film%29)

<comment><USER31> Yeah it's really similar, not just in premise, the writing style really reminded me of the narration in that awesome film.

</comment></comment><comment><USER32> This piece scared the hell out of me (seriously, I felt panicked) until I discerned that it was fiction. Great read.

</comment><comment><USER33> Very entertaining! How did no one realize this was fake by the first graph? Does xkcd design the graphs for fb premium partners?

<comment><USER25> The aesthetics of the chart are pretty standard for a 2D Stacked Area chart.

The only "mistake" is that the CONFIDENTIAL watermark is a lower z-order than the chart, which defeats the purpose of the "watermark."

<comment><USER34> We circulate internal drafts like that all the time. The purpose of the "Confidential" or "Draft" marking isn't to be in front of the data, it's to clearly indicate to anyone internally who has a copy that it shouldn't be released.

</comment></comment></comment><comment><USER35> Lovecraft does data science?

(the variance on the graphs intuitively felt a bit low, but then again it is facebook)

</comment><comment><USER36> Reminds me of this interpretation of Minority Report, only with AI instead of human precogs.

[http://mjyoung.net/time/minority.html](http://mjyoung.net/time/minority.html)

</comment><comment><USER37> I think April Fools has come early this year

<comment><USER38> Oh thank God they didn't waste this story on that.

</comment></comment><comment><USER39> Was I the only one, whos first impression was, that facebook was somehow involved in insider trading. And that for some reason they were brave enough to also use this information for load balancing...

</comment><comment><USER40> As someone who is non-technical, I pretty much was mind blown until the very end. Had a feeling it was fiction haha. Glad the comments here clarified. Entertaining read.

</comment><comment><USER41> Even when you feel its fiction, it just sounds like something that I could be reading off tomorrow's newspapers. This was a great read!

</comment><comment><USER42> It's funny that I've received the notification for the HN500 while I was reading "Mr. Penumbra's 24h bookstore" and noticed it after having finished the book. Clicked the link and the author was Robin Sloan as in the book :)

Apart from the funny combination I just chime in to recommend the book here, I think most of you here would enjoy that.

</comment><comment><USER43> By seeing the title along with the domain facebook.com I couldn't resist but click. Perfect click bait. Though a good story.

</comment><comment><USER44> Aw. It's in the tradition of "The Endochronic Properties of Resublimated Thiotimoline", by Isaac Asimov.

</comment><comment><USER45> Fiction, but that was entertaining. To the author, you should probably consider writing a book in this style :)

<comment><USER46> The author did write a fiction book, Mr. Penumbra's 24 Hour Book Store. It's decent enough :)

</comment></comment><comment><USER47> Nice piece of fiction

</comment><comment><USER48> Best version of this is still "Vaxen, my children..." [http://www.hactrn.net/sra/vaxen.html](http://www.hactrn.net/sra/vaxen.html) (read to the end, and check the meaning of the date)

</comment><comment><USER49> So I figured this was too epic to be real, but what part of it is real? Is there a team selling statistics from all posts from advertisers? Because that part I believed without a second thought and honestly, it does make sense.

</comment><comment><USER50> It's disturbing to me how many comments on here assert that the piece is fiction with little to no evidence. It occurs to me that perhaps Facebook is trying to make it seem like it's fiction by posting on various forums!

<comment><USER51> The burden of proof that facebook can see months into the future would be on the poster. No evidence is needed to simply disbelieve this.

</comment></comment><comment><USER52> Identified as fiction because VW isn't traded on the US stock market.

</comment><comment><USER53> Facebook has the data to make a solid prediction market. That should be their new biz model - they'll still exploit our data but they won't be degrading our user experience with ads.

</comment><comment><USER54> Did John Titor write this?

</comment><comment><USER55> I figured it was fiction by the end, but I couldn't really tell. I can't tell the difference between fiction/satire and reality anymore. The world is too strange.

</comment><comment><USER56> Hari Seldon and Gaal Dornick as devlopers of Psychohistory would approve. (Isaac Asimov's "Foundation" universe)

"Psychohistory is the name of a fictional science in Isaac Asimov's Foundation universe, which combined history, psychology and mathematical statistics to create a (nearly) exact science of the behavior of very large populations of people" source: Wikia

</comment><comment><USER57> Very enjoyable, and almost plausible...

</comment><comment><USER58> i thought the following into existence yesterday and read the post today. that facebook uses users as a layer of abstraction above one brain. a brain of brains. putting queries (thoughts) into the system (brain) and obtaining a result that is the average of many.

</comment><comment><USER59> I enjoyed the oracle discovery very much and even more its use on themselves. Cute.

</comment><comment><USER60> This piece was written by Robin Sloan, who is an author of fiction. :) Great story.

</comment><comment><USER61> Oh, somehow I've got the Pollyhop breeze from House Of Cards

</comment><comment><USER62> Hmm, fact-fic; I wonder who run with it make it an urban legend.

</comment><comment><USER63> It still makes me excited about the future of neural networks :)

</comment><comment><USER64> This could easily be a Black Mirror episode

</comment><comment><USER65> Not a very useful submission title.

</comment><comment><USER66> The big short, Facebook edition.

Entertaining read.

</comment><comment><USER67> I really got scary reading this

</comment><comment><USER68> I thought this was real, but man that's probably the best creepypasta I've read in a while.

</comment><comment><USER69> Obvious fiction. Nice story, but I don't like fiction masked as real.

</comment><comment><USER70> Enchilada?

<comment><USER71> PrestoDB, I guess.

</comment></comment><comment><USER72> Who could be the real author if it's fiction? This Robin Sloan might certainly have the means and the motive. Either it is fiction, with a plot which is not to far fetched as any company with these kinds of databases can exploit it in the prediction market. "Information is power and currency in the virtual world we inhabit" Billy Idol once said. It's also an important element in Asimov's Foundation series: extrapolating history through psychohistory.

Or it's not entirely fiction or perhaps even factious. I remember Rupert Sheldrake mention in one of his mindboggling talks that he wanted to investigate potential psi effects on a much larger scale and was in talks with Google. If there were 'results' pertaining to precognition (or AI enhanced precognition of the crowd) would the public get to know about it?</comment>


Blocking China IP Address Blocks - <AUTHOR> http://mergy.org/2013/02/blocking-china-ip-address-blocks/ <comment><USER2> We were very seriously considering if we should block China from our game servers recently. The reason is massive account compromises of our users.

There are 300,000 IPs from China that are just trying public leaked email / password databases against our servers. With so many IPs, any kind of normal per IP limiting just doesn't work. Each IP is only trying 10 or so accounts per day.

Blocking China was potentially a very real solution because I just don't think that they would have access to other bot nets of that sheer size outside of China.

The trouble is that the users don't understand that they are pre-compromised before they even arrive.

Anyway, we have now implemented a system whereby accounts get locked if someone attempts to log in from a different country than last time and they have to type in a verification code sent to their email.

All the "My account got hacked" support requests have been replaced by an equivalent number of "Why does my account keep getting locked" support requests. But there you go.

<comment><USER3> You should implement automatic unlocking of the account if the auth attempt is successful and comes from the "usual" country (or from the same "usual" IP).

<comment><USER2> We talked about this for a while and decided that, at least at first, we want to lock out the original owner.

The reasoning behind this was that their actual password _is_ compromised. We want to make sure that the user understands this fact and changes their password. Not only on our service, but on all the others that they may be using the same password for as well.

The email does say that someone else has your password, but if you can just ignore the email then most will not actually get it or assume it's some kind of phishing email.

</comment></comment></comment><comment><USER3> If you resort to blocking IP ranges to prevent attacks, you are missing the point of how to properly respond to an attack. Blocking ranges might by an extra layer of security (philosophy of defense in depth), but in addition to that you should analyze how this email user account was compromised.

Weak password was bruteforced? Start enforcing strong passwords.

Email server vulnerability exploited? Patch your server.

Etc.

<comment><USER4> Password length and complexity aren't all that critical if it's salted and hashed. Locking out a user after a number of incorrect attempts and then requiring a different password during the reset than the one already used is a better alternative.

Reusing passwords on different sites is a much bigger problem IMO since a lot of places still don't store passwords correctly or don't lock out users after failed attempts.

<comment><USER5> _Password length and complexity aren't all that critical if it's salted and hashed._

Unless your users' passwords are something like "password" or their user names. Password length and complexity are important, if overplayed.

<comment><USER4> Ah yes, well... if they're using 'password' for the password, they've got bigger problems ;)

Passwords that can be guessed in 1-3 tries should be excluded, naturally: password, 12345, 11111 etc... But mixed case, special character stuff is a bit redundant.

<comment><AUTHOR> Pretty sure the issue I had to deal with was related to the Java OSX exploit. That being said, it is a total trade-off vs. security or ease of use for the end-users.

</comment></comment></comment></comment></comment><comment><USER6> For my personal servers, that only host private information, I have no reason for them to be accessed from China - though login attempts were extremely common.

I created this iptables script and update it when I notice any new patterns of abuse

<https://gist.github.com/anonymous/1b6f1b08273b92cca890>

</comment><comment><USER7> I'm using fail2ban on one of our linux servers. I have a bunch of fail2ban reports that I can run that lists all ips being blocked, how they were blocked, etc. A lot of times if we get multiple ips being blocked from china (and elsewhere) on the same subnet, I'll just block the entire subnet.

<comment><USER8> Yeah I think an approach like fail2ban is generally better than wholesale banning of IP blocks by country. Block the people who show bad behavior, not everyone.

Of course if you KNOW you have no users in e.g. China, no harm in blocking them, but any skilled attacker in China is not going to _appear_ to be in China, from your vantage point.

</comment></comment><comment><USER9> Many of us have users (actual valid users) who live in and visit China and other countries in the world. So we don't block a IP because we _think_ it is in China.

Use rate limiting and block bad IPs that are brute-forcing services (don't lock accounts) then you'll be able to serve your users while keeping the bad guys out.

<comment><AUTHOR> Absolutely. Gladly, that is not an issue in my case.

</comment></comment><comment><USER10> The ability to do this (and not just for China) is one of the primary reasons I'm looking at new VPN router possibilities to replace the venerable-but- stable RV042s that we've been using for years at clients.

I just haven't found anything yet with a good combination of price, capabilities and hardware VPN support - doubling the price we're currently paying would be feasible, quadrupling it when replacing functioning equipment is harder to justify to non-technical users.

A worthwhile resource for folks with Windows (and with some useful links for others): [http://www.sans.org/windows-security/2011/10/25/windows- fire...](http://www.sans.org/windows-security/2011/10/25/windows-firewall- script-block-addresses-network-ranges)

<comment><USER11> I was also looking to switch out some RV042 (tried the newer RV180 series -- terrible mistake). Finally settled on RouterBoard / MikroTik RB2011L-IN.

The feature base is incredible: <http://routerboard.com/RB2011L-IN>

</comment></comment><comment><USER12> Perhaps this will one day be so widely implemented that the great firewall of China will become obsolete, and the censors' dream will be realized.

</comment><comment><USER13> The Chinese can use a foreign botnet just like anyone else. The fact that so many attacks originate (traceably) from China is probably just down to laziness but that doesn't mean you'll be protected if you block the country outright. Ultimately, individual IP addresses should be blocked for a certain time after your service recognizes anomalies in client behavior (such as multiple login fishing attempts).

That said, it's pretty easy to block countries from accessing web apps at least if you use Cloudflare. The CF proxy passes a special field down to your server containing the country of origin. Works quite well actually.

</comment><comment><USER14> Interesting, most of the cranky search stuff seems to come from the Ukraine or Russia. I've seen some Chinese activity but perhaps because we're a search engine [1] the Chinese government blocks us for their citizens.

Its probably not sustainable to just block the entire country long term though. You have to figure out a different way of figuring out folks who are real from folks who aren't otherwise you end up with really irritated users.

[1] blekko.com

</comment><comment><USER15> There was a massive password leakage happened in the end of 2011 and early 2012. Probably more than 100 millions of passwords from more than ten popular websites including some popular social network websites, were made available on the internet (BT or eDonkey). The size of the password files added up to 10GB after compressed. Some significant amount of the passwords were in plain text when obtained so that with such a huge dictionary rainbow table can be used to decrypt the a large portion of rest.

So the reality is really nasty. Many of the netizen in China are somehow running naked: you can simply query the password after you get the email.

</comment><comment><USER16> This would backfire. The goal, I suppose, is that the Chinese government go after hackers in China more; but even if this happened on a mass scale, the Chinese government would /love/ for more services to be run domestically. They don't need Google or Facebook, what makes you think they won't survive well without any of our sites?

<comment><AUTHOR> True. I am a total "small fish" and China could give a damn, but I think I am just sick of even dishing any bandwidth to known bad address blocks.

</comment></comment><comment><USER17> I looked at the links in the article and its comments, but this one seemed much more "immediately useful" for me: (and not China-specific either, you can pick any ISO country code to add to your iptables)

[http://www.cyberciti.biz/faq/block-entier-country-using- ipta...](http://www.cyberciti.biz/faq/block-entier-country-using-iptables/)

If you have a linux-based router, this can be a 5 minute job.

In fact, I'll save you some time. I modified the script slightly to better suit my needs. Enjoy:

#!/bin/bash

      # License: any/both of the following: public domain or MIT
      #
      ### Block all traffic from AFGHANISTAN (af) - ISO code ###
      #
      # you will need to do the following setup steps manually:
      #
      # iptables -N drop-by-country
      #
      # for a in INPUT FORWARD OUTPUT
      # do iptables -I $a 1 -j drop-by-country
      # done
      #

      ISO="af"

      IPT=iptables
      WGET=wget

      SPAMLIST="drop-by-country"
      DLROOT="http://www.ipdeny.com/ipblocks/data/countries"

      for c in $ISO; do
      	tDB=$c.zone
      	#rm -f $tDB
      	[ -f $tDB ] || $WGET -O $tDB $DLROOT/$c.zone || exit 1
      done

      # convert IP and mask to decimal IP (32-bit value) (and leave mask unchanged)
      BADIPS="`for c in $ISO; do cat $c.zone; done | awk 'BEGIN{FS="."}
      	{
      		if ($0 == "" || $0 ~ "/^#/") next;
      		mask=gensub("^[0-9]*/", "", "", $4)
      		n=gensub("/[0-9]*$", "", "", $4)
      		n=(($1*256 + $2)*256 + $3)*256 + n
      		print n " " mask
      	}' | sort -n`"

      # merge adjacent IP ranges until nothing changes
      N=""
      limit=20
      while [ "$N" != "$BADIPS" ]; do
      	echo "simplifying `echo \"$BADIPS\" | wc -l` rules"
      	N="$BADIPS"
      	BADIPS="`echo \"$N\" | awk 'BEGIN{p1="";p2=""}
      		{
      			n1=\$1
      			n2=\$2
      			if (p1 != "") {
      				e=2 ** (32-p2)
      				if (n2 == p2 && int(p1 / e) % 2 == 0 && int(n1 / e) - int(p1 / e) == 1) {
      					n1=p1
      					n2--
      				} else {
      					print p1 " " p2
      				}
      			}
      			p1=n1
      			p2=n2
      		}
      		END{ if (p1 != "") print p1 " " p2 }'`"
      	limit=$(( $limit - 1 ))
      	[ $limit -eq 0 ] && break
      done

      # convert back to IP format
      echo "$BADIPS" | awk '{
      		o4=$1
      		o1=o4 % 256
      		o4=int(o4 / 256)
      		o2=o4 % 256
      		o4=int(o4 / 256)
      		o3=o4 % 256
      		o4=int(o4 / 256)
      		print o4 "." o3 "." o2 "." o1 "/" $2
      	}' | while read a; do
      		echo "$IPT -A $SPAMLIST -s $a -j DROP"
      		$IPT -A $SPAMLIST -s $a -j DROP || exit 1
      	done

# let me end by just saying that blocking an entire country is the WRONG solution, though it might be considered part of a "layered defense" strategy. On the other hand, if you want to apply this to your home router and play with it, that's a different story.

<comment><AUTHOR> You're right. Outright blocking is not a solution. I understand that. But, there comes a time when you just don't want to tell people to stop knocking on your door. You know?

<comment><USER17> Sure! In fact, I was thinking about the pros/cons of doing it about the time Mandiant posted their APT1 writeup, and I decided to spend some time implementing it.

It's important to me to preserve the open nature of the internet. So I hope the karma bonus of posting some code offsets the karma loss from the code being "racist" ;-)

Great article, thanks!

</comment></comment></comment><comment><USER18> We cut our blog and forum spam massively by blocking China. I'd estimate like 80% or more

Also consider if you can not restrict your ssh and ftp access to very tiny blocks, you can just allow US (or your country) addresses into those ports.

Of course proxies defeat all this but it slows down the generic script use.

Configserver firewall is amazingly powerful and easy (and free) in this regard

<http://www.configserver.com/cp/csf.html>

CSF is also good at noticing distributed attacks across ip ranges.

ps. please consider donating to Chirpy for CSF, I'd hate to see it die someday

</comment><comment><USER19> Blocking china doesn't help anyone, and just hurts people that didn't do anything wrong.

The script kiddies (fake hackers) can't really get into your systems if you apply simple security policies and sanity checks.

The real hackers that can get it aren't blocked by any of your ip filters. They just go through proxies.

</comment><comment><USER20> Be extremely careful using public IP lists, they're not always up to date. My iPhone was reassigned an IP in the 188.8.131.52 block last year, which used to be issued by a Chinese supplier. Caused havoc with geoIP and locked me out of a number of websites for suddenly changing country.

</comment><comment><USER21> Rather than putting multiple deny rules in a chain, why not use xt_geoip? <http://xtables-addons.sourceforge.net/geoip.php>

</comment><comment><USER22> I'm waiting for Dalton to complain about the use of the Svbtle theme...It seems to happen to _every_ post that links to somewhere not svbtle that uses it.

<comment><USER23> <https://github.com/gravityonmars/wp-svbtle>

<comment><AUTHOR> wp-svbtle does comments and some other stuff above and beyond svbtle. I like it. Works for me for now.

</comment></comment></comment><comment><USER24> Blocking an entire country sounds very stupid from the technological and moral point of view (and yes, it may make sense from the financial/time pov).

1) that doesn't make your shitty (lets be rough here) passwords & web apps secure. You didn't care for security yesterday, it's not going to come to you by blocking "china".

2) that doesn't stop anyone from proxying elsewhere

3) the more doing it, the more segmented the internet, the less it actually IS the internet. basically, you're breaking the fucking point of the internet (that justify the swearing.)

<comment><AUTHOR> You're right on all accounts. But, the internet "works" on a common understanding that the various entities involved act with some form responsibility. That is NOT the case right now with China. I can beef-up security all I want, but perhaps people will get to a point when even interacting and rejecting bad connections is a waste?

</comment></comment><comment><USER25> This is essentially racist (well, xenophobic), and doesn't actually solve the problems.

</comment><comment><USER26> you can do this on your SOHO router at home with DDWRT and Optware's asiablock.

[http://www.dd- wrt.com/wiki/index.php/Optware%2C_the_Right_Wa...](http://www.dd- wrt.com/wiki/index.php/Optware%2C_the_Right_Way)

I've had this setup for years. It's simple and effective. I block China and Russia entirely.</comment>


Yunus fired from Grameen Bank - <AUTHOR> http://www.bbc.co.uk/news/world-middle-east-12619580

<comment><USER2> A desperate attempt to get him out definitely. Cannot believe their trying to pin a man who's done more for the country than the government ever will as a villain. I guess I better believe now?

</comment><comment><USER3> Interestingly the government doesn't own 25% of the bank as this and several other articles claim. In fact 95% of the shares is owned by their customers : [http://www.grameen- info.org/index.php?option=com_content&...](http://www.grameen- info.org/index.php?option=com_content&task=view&id=26&Itemid=175)

The government might have special voting right though.

</comment><comment><USER4> It looks like they did it purely out of spite, which makes it even more stupid.

</comment><comment><USER5> What is funny is that their site is hacked as well: [http://blog.sucuri.net/2011/03/grameen-bank-web-site- hacked-...](http://blog.sucuri.net/2011/03/grameen-bank-web-site-hacked- infected-with-spam.html)

</comment><comment><USER6> I had the pleasure of meeting Dr. Yunus in Dhaka and touring a village bank back in 2008. His personal office was simple and sparse (and not air conditioned) and he was very kind and down to earth considering his accomplishments. It is sad to think that a (probably) corrupt government could oust him from his own creation where he has done some much for so many.

</comment><comment><USER7> Politics as usual.</comment>


Bits on the Wire - <AUTHOR> https://www.tbray.org/ongoing/When/201x/2019/11/17/Bits-On-the-Wire <comment><USER2> > ... schemas aren't free. You have to coördinate between sender and receiver, > and you have to worry what happens when someone wants to add a new field to > a message type — but in raw JSON, you don't have to worry, you just toss in > the new field and things don't break. Flexibility is a good thing.

Much like type systems in general, you always _have_ a schema, and you always have to worry about what happens when someone adds a field. You just get to choose whether it's explicit and you have to type it alllll the way out allllll the time, or whether it's implicit, and is defined as "whatever all the usages are everywhere".

Just as gradual typing is taking over the world (TypeScript, MyPy, etc.), I predict success for something that lets you gradually add schemas to JSON.

<comment><USER3> > ... you just toss in the new field and things don't break. Flexibility is a > good thing.

Except when it's not you that are flexible but other parties, and they express that flexibility not only by adding but also removing or modifying fields without notice :/

<comment><USER4> I very rarely see system that distinguish between "this new feature can be safely ignored" and "it's not safe to process this message if you don't support this new feature", but I've always thought it a good idea. Like semantic versioning without a global ordering over schema changes.

</comment></comment></comment><comment><USER5> > _Then XML came along in 1998._

Useful context for the youngsters: Tim invented XML. I mean, techically, there were a few other people who helped write the spec, but Tim was using more-or- less XML in shipped applications before they got involved.

In my book, that means his opinion about the importance of bits on the wire is pretty weighty, but you also shouldn't discount the possibility that he's in some sense constitutionally inclined toward standardized data formats.

About other data formats:

I have some experience with CBOR which I regret; it's more compact than JSON, but it's usually not significantly more compact than gzipped JSON, and it's usually slower than JSON, which is saying something, and is surprising given that no scanning for delimiters is needed. Its only real advantage is that you can safely include binary blobs. It's worth noting that originally CBOR was part of MessagePack until its author decided to fork it off and name it after himself, Carsten Bormann.

Cap'n Proto, FlatBuffers, and SBE (which I guess is too obscure for Tim to mention) get their "deserialization in negative nanoseconds" by virtue of not deserializing; instead, the respective libraries access the desired fields of the serialized data in place, which involves an extra pointer indirection or two and possibly some byte swapping, so it's a little slower than accessing fields of native structs. But it means you don't have to parse the entire data structure, just the parts you're accessing, so you can do things like memory- map a read-only data file instead of parsing it. In many cases, even if you're accessing the whole thing, saving the extra memory copy is actually worth _more_ performance than avoiding the byte-swapping and indirections. This is not a new idea — COBOL was based on it, though without these systems' extensibility and consequently without the cost of the extra indirections — but it's an extremely effective idea in many contexts.

All of FlatBuffers, Cap'n Proto, SBE, Protobufs, Thrift, and I think Avro do have support for adding new fields, contrary to what Tim says about binary formats in his note; but they mostly do presuppose that those new fields are being defined by some central authority, because there are things like sequentially assigned field IDs, so certainly you don't have the level of safe extensibility that you have in JSON (or HTTP or RFC-822 mail or HTML forms or SNMP ASN.1 BER data or, of course, XML and HTML tags and attributes).

<comment><USER6> To be fair, most of those "extensible" formats aren't safely extensible. ASN.1 is because it uses OIDs, and some scenarios in XML and parts of HTTP are safely extensible because they use URIs, but anywhere which relies on just arbitrary text, like JSON, is just shrugging when it comes to global collisions.

What I'm getting at here is, OIDs and URIs are globally namespaced. When I decide to add my own custom foo to a format, OIDs let me go get myself an arc and assign foo an OID from that arc, and URIs let me go get myself an FQDN and then assign foo a URL with that FQDN. In HTML if I just add a foo attribute that collides with everybody else's foo attribute, in RFC-822 mail my foo header can collide with anybody else's foo header. And so on.

<comment><USER5> In theory, I sympathize with this view, but it doesn't really seem to be much of a problem in practice. I don't think I've ever seen a colliding RFC-822 header name or HTML tag name. (There are lots of problems with RFC-822 but colliding header names are not one that I've seen.) I _have_ occasionally had problems in SNMP where the meaning of the same OID changed from one version of a MIB to the next. I don't think I've seen XML namespaces maliciously collided, but there aren't very many of them, maybe a couple dozen in mainstream use.

I think this is sort of like how Wikis don't work in theory but do work in practice.

One nitpick: I don't think the extensibility of SNMP is due to ASN.1 but due to the key-value semantics of SNMP. It's true that the keys are OIDs, and the OIDs are delegated to organizations to avoid collisions (by the ITU I guess), but if you have a field in some arbitrary ASN.1 syntax that is just an OID, that doesn't give you any kind of backward-compatible extensibility. However, although I spent a couple of years struggling with SNMP on a daily basis, I'm not an expert on ASN.1 by any means, so I could be mistaken about this.

<comment><USER6> I get your point that yes, you could use ASN.1 in a way that isn't extensible. To pick a nit in your nit though:

OIDs are arranged in a hierarchy, so anybody who already has an arc can just delegate you an arc from their part of the hierarchy, much the same way foo.example could delegate kragen.foo.example to you, and then you could delegate tialaramex.kragen.foo.example from that to me and so on.

The IETF hijacked the 220.127.116.11 arc, by basically arguing that if hypothetically the US Department of Defence were asked to delegate them the 220.127.116.11 arc from its 1.3.6 arc it would doubtless agree, and so no need to actually ask...

For example 126.96.36.199.188.8.131.52.1 is "serverAuth" the OID used to indicate that an X.509 certificate is intended to identify a TLS Server, that was assigned by the IETF's PKIX many years ago.

But the IETF has an arc under 184.108.40.206 for giving out OID arcs to anybody who needs one, 126.96.36.199.4.1, you can just fill out a form for IANA explaining why you need one (e.g. to make some new SNMP features work) and you'll get an arc back typically within a few days. e.g. 126.96.36.199.4.1.44947 is ISRG, the charity which provides the Let's Encrypt service.

Colliding HTML tag names has been a problem, albeit a small one, even inside "standard" HTML itself because of the conflict between WHATWG and W3C for years. The small number of web browsers with significant market share made this less serious than it might be. As to people redefining MIB OIDs, I can't help idiots, I would regard every instance of this happening as a bug in the later definition.

<comment><USER5> What I mean about ASN.1 is that if you have a field foo that's defined as

        foo SEQUENCE { corge INTEGER, grault OBJECT IDENTIFIER }

you can't compatibly augment it with a new field quux as

        foo SEQUENCE { corge INTEGER, grault OBJECT IDENTIFIER, quux OBJECT IDENTIFIER }

and it doesn't help you at all that grault and quux are both OBJECT IDENTIFIERs.

Which HTML tag names had collisions? I can't think of any at the moment, but admittedly I'm no HTML expert, though I've been reading HTML specs since HTML+ and before.

I'm more interested in problems not happening than on having someone to blame the problems on.

<comment><USER6> <cite> is the canonical HTML tag collision. WHATWG and W3C defined this differently. In practice as so often in the Web it's resolved by a mixture of just accepting everything and so nobody gets what they wanted, and by semantics of the document being ignored anyway.

Yes, you're correct that you can't just shove arbitrary things into ASN.1 in this way and so ASN.1 itself is arguably not where the extensibility is, that's fair.

<comment><USER5> Thanks for the note on <cite>! I don't think that's actually a _collision_ ; <cite> was in HTML 2, so there wasn't much time for people to start using it in conflicting ways. What happened with <cite> was that, for about four years, Hixie published a _changed_ definition in the HTML5 spec, one which didn't allow citing people's names, just the titles of their works. This attempted change was eventually reverted. The URL-based and OID-based extensibility mechanisms we're discussing don't even attempt to prevent such problems.

What I mean by "collision" is something like the following: one person (say, Hixie) defines <c> in their browser or stylesheet to mean "canonical name" while another person (say, the W3C) defines <c> to mean "category", resulting in problems when you try to view documents written for one browser using the other browser. I don't know of anything like this having happened in HTML, ever.

</comment></comment></comment></comment></comment></comment></comment><comment><USER7> "The take-away is that if you're going to start emitting events from a piece of software, put just as much care into it as you would as you do in specifying an API. Because event formats are a contract, too."

</comment><comment><USER8> To me the biggest annoyance of json is there's no standard way to put comments in there. What's the point of a text-based serialization format if it can't carry comments???

It has one advantage over YAML though, with JSON you know when you're not at the end of the file :-)

<comment><USER9> The downside of this is that the comment becomes just another piece of the data model, because anything that consumes that JSON will need to decide: do I preserve comments, or remove them, when re-emitting the JSON?

If I remove them, comments are kind of only useful for things like initial config files or what have you. Which might be fine for your use case, but JSON broadly is not about making configuration files - this is more suited for something like YAML.

If you preserve them - how is that different from a key-value just named "comment": "Here's my comment"? (And if the answer is "because it might not crash poorly coded applications," that's more of a discussion about the application than it is about comments in JSON)

<comment><USER10> Consider the comment as like a header in HTTP in relation to body content: a side channel for information. I despair that anyone was putting processing pragmas in comments, but you can't beat all the human out of a programmer.

The word "Notation" in JSON and the fact that JSON allows an arbitrary quantity of ignored bytes in any object (whitespace) imply that comments would have been just fine. How many JSON processors are required to preserve whitespace? (Ignorant here, I'll take any informed answer and learn from it.)

</comment></comment></comment><comment><USER11> My browser opens up a download dialog on clicking this link. It's probably because the response headers don't include "Content-Type", and so it doesn't know if it will be able to interpret the response.

<comment><USER5> Interesting, what browser?

<comment><USER11> Firefox on Archlinux. Though, now I think the issue is with an extension I have.

<comment><USER5> Weird, I'm not having that problem in Firefox on Debian. But it's sure enough true that there's no Content-Type header, either with LWP or with Firefox. Tim, are you doing that on purpose?

<comment><USER11> I found it's my extension[1]. That's what's interpreting the lack of Content- Type as indication that Firefox is going to force a download (it actually doesn't) and showing me its own download dialog.

[1] [https://github.com/Rob--W/open-in-browser](https://github.com/Rob-- W/open-in-browser)

</comment></comment></comment></comment></comment><comment><USER12> A quick note about json next. I collect new JSON variants, extensions and formats at the Awesome JSON Next page [1]. [1]: [https://github.com/json- next/awesome-json-next](https://github.com/json-next/awesome-json-next)

</comment><comment><USER13> Interesting article, but lots of "probably the next biggest consumer" etc. Was hoping he would have some kind of network traffic analysis report in there that backs up his ranking of the formats. Seems to be anecdotal more than anything. Still good overview of the different formats available.

</comment><comment><USER14> A quick note about protobuf. It's just way too complicated in my opinion. If you need a binary protocol just use TLVs (Tag-Length-Value) - so much less cognitive load and the parsing is easy to implement and read compared to the complicated autogenerated mess that is protobuf.</comment>


3.5% of Android phone clocks are wrong by more than an hour - <AUTHOR> http://opensignal.com/reports/timestamps/

<comment><USER2> My Android phone (a Motorola Atrix) has issues when I cross timezones. For the past few months, I've been travelling between Chicago and Raleigh on a near weekly basis, and I find pretty consistently that when I land in either city, my phone doesn't update the time from the network, to reflect the local time zone. So, if it were showing the time on the East Coast when I left Raleigh, I could land in Chicago and 2 days later my phone will still have East Coast time, unless I physically reboot it. And vice versa going the other way.

I don't know if it's a reflection of the same problem(s) talked about in this article or not, but I know it's incredibly annoying.

<comment><USER3> In the UK most 3G networks do not give out the time. In Belgium they seem to.

Why NTP is not running as standard I have no idea. You have to root your phone to run it.

<comment><USER4> GPS is also a source of time.

<comment><USER5> Although not a great source of time zone info unless you have a big database to back it all up.

</comment></comment></comment></comment><comment><USER6> Title is total link bait. Article says 3.5% of Android phones clocks are off by 15s and that in some cases there are time zone edge cases causing it to be off by an hour. This is not what the title conveys.

<comment><USER7> Actually 3.5% of android clocks ARE out by one hour or more... the 15s bug effects some phones, but the problem of being more than an hour out is very general and does effect 3.5% of phones - sorry if that's not clear!

</comment></comment><comment><USER8> This was an issue for our app. We went SSL-only for our API, which unfortunately fails with a cryptic error message if the clock is off. We eventually messages this to the user ("Sorry, Zite is broken, please reset your clock"), but there's no way to know how many users actually went and fixed it and returned to the app.

<comment><USER9> The Google Play Store app on Android is also affected by this. It'll just show a generic "unable to connect" message if your clock is too far off, which I assume is related to the same SSL problem.

</comment></comment><comment><USER10> >The time displayed to the user will seem correct

This gets into fuzzy definitions of "correct" here. If the phone is showing the right local time, but internally has the wrong UTC time, does it matter?

<comment><USER11> Yes. With many two-factor authenticator applications, your codes won't work if your clock is wrong.

<comment><USER12> Two-factor authenticator applications should be using UTC, and not the display time. I think that's what the parent comment's point was.

<comment><USER10> Right, but these phones have the wrong UTC, if I understand correctly. That's why they are in the write-up. Wrong local/display time is pretty hard to detect, unless you have coordinate data.

Given that, two factor where the second factor is a local authenticator program is not exactly a common thing.

</comment></comment></comment></comment><comment><USER13> It's Candy Crush's fault.

</comment><comment><USER14> My old phone (LG Thrill) had a bug where the clock would simply stop syncing for some unknown reason. It could be hours out of date at any given time. The only way to resolve this was to restart the phone. Of course, it was never fixed before LG abandoned the phone. There were other very serious problems with that phone, and the whole experience put me off Android.

</comment><comment><USER15> In our experience, same applies to also iPhones, but the offsets tend to be time zone granularity. And a lot of users, especially from Asia, seem to use a U.S. VPN or a proxy of some sort. Their devices still have +8 hours or so time zone and an asian carrier name.

</comment><comment><USER16> Does this account for what I saw in a store? The Galaxy Note 8.0's clock widget didn't have the same time as the clock in the upper right corner. Or is this a widget issue?

<comment><USER17> That sounds like a widget issue. Unless you saw the clock widget change times, it most likely froze up.

</comment></comment><comment><USER18> Does this issue affect the iPhone? If not, why not?

<comment><USER19> Simple, its an Android problem.</comment></comment>


Show HN: Send cloudinit output to cloudwatch logs - <AUTHOR> https://github.com/in4it/tee2cloudwatch <comment><AUTHOR> When having a lot of EC2 instances that use UserData, it gets difficult to track the output, as you have to log in to each instance to see the output of the cloud-init logs. I wrote a small golang program that can send the output of your commands straight to cloudwatch logs. That way, you can easily see and track errors in userdata. PR's & feedback welcome!</comment>


How would you implement the Hacker News Ranking Algorithm with Mongodb - <AUTHOR>

I have implemented Hacker News Ranking Algorithm with SQL in the past, but am totally lost on how to do it with MongoDB. Any ideas?<p>preferably not using map reduce. <comment><USER2> I actually did this for exipe.com in node.js/mongodb let me look it up.

<comment><USER2> Store a "score" to rank by in the database and then use some sort of time dependent function. AKA when you search the mongo for posts sort by "score."

Our score was calculated by something like this:

        var gravity = 1.5;
        var score = num_votes / Math.pow(t+2,gravity);

Gravity being how quickly a post would fall. Notice the t+2 ... it added 2 hours to all posts in order to eliminate things that were recently posted with like one vote sitting at the top.

Then we'd just run this function through the whole database periodically (every 5 minutes or something).

<comment><USER3> Recently I was looking into how to implement trees in MongoDB. Apparently there are several ways achieving this depending on querying needs.

Could you share your experience building trees for comments in your project Exipe?

<comment><USER2> We just had a comments model (in mongoose) that had an array of subcomments (MongoDB arrays are awesome). Therefore the "trees" (not really trees as we implemented it) could only have one level of indentation ... but that's all we wanted anyways.</comment></comment></comment></comment>


Viomba: A/B test your website and measure your design with eye tracking - <AUTHOR> http://www.viomba.com

<comment><USER2> Looks interesting, but FAQ requires login???

Also, isn't it a problem that the testers are likely not motivated in any way similar to potential customers?

<comment><AUTHOR> Sorry for so late respond. Somehow missed your message. FAQ should be available now with new name link. Still few more days needed to make everything nice in www.viomba.com About testers:Targeting a right tester group will help. Also eyes are a bit different than e.g mouse. Eyes moves often unintentionally. If you will be interested you can try our demo and see how it works</comment></comment>


Keep on moving: the bizarre dance epidemic of summer 1518 - <AUTHOR> https://www.theguardian.com/stage/2018/jul/05/bizarre-dance-epidemic-of-summer-1518-strasbourg <comment><USER2> Too bad we will probably never know what it really was...

Just like with the celestial phenomena in Nuremberg 1561, and Basel 1566:

[https://en.m.wikipedia.org/wiki/1561_celestial_phenomenon_ov...](https://en.m.wikipedia.org/wiki/1561_celestial_phenomenon_over_Nuremberg)

[https://en.m.wikipedia.org/wiki/1566_celestial_phenomenon_ov...](https://en.m.wikipedia.org/wiki/1566_celestial_phenomenon_over_Basel)

</comment><comment><USER3> If you understand french, I advice you to listen to this podcast by Franck Ferrand which explains lot of aspects !

[https://www.youtube.com/watch?v=bA7K8gfLQsY](https://www.youtube.com/watch?v=bA7K8gfLQsY)

<comment><USER4> I like to listen to his podcasts when commuting : he is right at the edge of pop-history (leaning towards the serious one) and several of the stories were brilliant. Not sure what will happen now that he left the radio he was on.

Another good pop-historian one is of course Stéphane Bern, though the visuals of his TV series are more interesting than the strict historical part.

</comment></comment><comment><USER5> John Waller wrote a book about this 9 years ago which I thought was amazing:

[https://iconbooks.com/ib-title/a-time-to-dance-a-time-to- die...](https://iconbooks.com/ib-title/a-time-to-dance-a-time-to-die/)

I don't know if anything new has come to light though.

A particular touching thing was that they made red shoes for the dancers, a known expensive colour, to show how much they valued them, but it didn't stop the dancing.

The film the Red Shoes was based on HC Anderson's story, which as far as I can tell has no connection with this dancing epidemic, just that his Dad was a shoe maker and had some spare red material...

</comment><comment><USER6> I'm surprised that the poignancy of Martin Luther's proclamations [0] the previous year seem to have been lost in this analysis. It's interesting that the article suggests that the belief-systems driving this hysteria began to diminish in the years following. Was this _peak middle-ages_? Had the papacy driven everybody to breaking point at this stage?

[0] [https://en.wikipedia.org/wiki/Reformation#Martin_Luther_and_...](https://en.wikipedia.org/wiki/Reformation#Martin_Luther_and_the_beginning)

</comment><comment><USER7> It feels like the people from 1518 are trolling us.

<comment><USER8> That or we are seeing some early form of memes.

</comment></comment><comment><USER9> Wasn't the Tarantella dance belived to alleviate the the effect of a tarantula bite by sweating the venom out?

</comment><comment><USER10> I would have guessed that such thing could have been caused by overconsumption of Ephedra [https://en.wikipedia.org/wiki/Ephedra](https://en.wikipedia.org/wiki/Ephedra)

</comment><comment><USER11> This is pretty weird.. who would have thought they're prescribe "more dancing for cooked minds!"

<comment><USER12> I feel some details might have lost in 500 years :) The council probably liked the decadence of the happening and they added some fuel. It was probably unreal to be at a "festival" you never seen before so I can understand.

Just like experiencing a techo party for the first time no one seen before. [https://www.youtube.com/watch?v=hWUiLJnEYJI](https://www.youtube.com/watch?v=hWUiLJnEYJI)

</comment></comment><comment><USER13> A probable explanation is that it was made up to discredit dancing.

</comment><comment><USER14> See also tarantella

</comment><comment><USER15> > what used to be called "mass hysteria".

Now we call it the 'online community'.

</comment><comment><USER12> > Fortunately, the 1518 dance epidemic was the last of its kind in Europe.

No no no... we call them festivals now :)</comment>


Why Are Cryptographers Being Denied Entry into the US? - <AUTHOR> https://www.schneier.com/blog/archives/2019/05/why_are_cryptog.html <comment><USER2> The spooks and have always hated public Crypto and the professional xenophobes known as customs officials and border security are part of that apparatus now. They have long had an absurd anti-cryptography agenda and unchecked power.

Said abuses aren't surprising and won't stop until they get made an example of in the worst way.

</comment><comment><USER3> Those that are denied look to be people who are hard to intimidate and/or control.

Government loves its backdoors in crypto. Perhaps visa denials are revenge by the government for not putting back doors in, or building too secure of a product?</comment>


Why Facebook was down? - <AUTHOR> https://www.facebook.com/notes/facebook-engineering/more-details-on-todays-outage/431441338919

<comment><USER2> That article is from 2010</comment>


Beyond the RDBMS: the Brave New (Old) World of NoSQL - <AUTHOR> http://philip.greenspun.com/teaching/three-day-rdbms/beyond-the-rdbms

<comment><USER2> He misses the point. Do you want your database to scale in units of $20'000 Dell servers + $50'000 Oracle licenses that take a week to deliver and install, or $500/yr EC2 instances you can have up and running in under a minute? Do you want to pay some more money and time to get "Oracle Data Guard" for redundancy, or just boot up another Riak instance? In case of failure would you rather have writes always go through (e.g. purchases are never refused) or have customers wait while "the hot standby machine ... roll[s] forward from the transaction logs and, as soon as it is up to date with the last committed transaction, take the place of the dead server"? There are good reasons for NoSQL databases.

<comment><USER3> That has nothing to do with this article. I can install EC2 instances of PostgreSQL or any other good free database too if I want. Hell there's EC2 instances with built-in SQL Server for minimal extra cost if you want the real high-end stuff. That's not an advantage of NoSQL.

As for your other argument I don't understand either. With SQL servers you can have a hot spare with automatic failover with no 'roll forward from transaction logs' that's just pure FUD.

There are indeed good reasons for NoSQL data stores but those are not it.

<comment><USER2> "Roll forward" is a quote from the article (hence the quote marks). Here is the full quote:

"The simplest approach to redundancy is a "hot standby" server that has access the transaction logs of the production machine. If the production server dies for any reason, the hot standby machine can roll forward from the transaction logs and, as soon as it is up to date with the last committed transaction, take the place of the dead server. As a bonus, the hot standby machine can be used for complex queries that don't need to include up-to-the-second changes. For Oracle 11g, look for "Oracle Data Guard" to learn more about this approach."

If you think that is FUD take it up with the author.

The $20'000 Dell server is (guess what!) a quote from the article:

"It is difficult to find realistic benchmarks for the kind of database activity imposed by an Internet application. The TPC-E benchmark is probably the closest among industry standards. TPC-E is a mixture of fairly complex reads (SELECTs) and writes (INSERTs and UPDATEs) into an indexed database that gets larger with the number of SQL statements that are attempted for processing. The "transactions per second" figures put out by the TPC include both reads and writes. In 2010, a moderately priced Dell 4-CPU (32 core) server hooked up to, literally, more than 1000 hard disk drives, processed about 2,000 transactions per second into an 8-terabyte database.

The authors crowd-sourced the question in this blog posting and for smaller databases that can be stored mostly on solid-state drives, it seems as though a modest $10,000 or $20,000 computer should be capable of 10,000 or more SQL queries and updates per second."

So the argument is that an expensive server is the RDBMS solution for scalability. This is the argument I addressed.</comment></comment></comment>


Tl;dr.js summarizes hn articles, and hn comments in your browser - <AUTHOR> http://devblog.bu.mp/52614084

<comment><AUTHOR> This is just a quick hack that pulls out the "best" or most information dense sentences from an article and shows them to you. Any suggestions or forks are welcome.</comment>