The Lie Behind 1.2B Stolen Passwords - <AUTHOR> http://www.youarenotpayingattention.com/2014/08/08/the-lie-behind-1-2-billion-stolen-passwords/

<comment><USER2> Cross-posting this from another site I posted this on:

I am glad to see that someone has said what I had suspected. I made a quip [1] about the ethics of charging for information regarding whether or not you're affected by a breach--full disclosure, I run a free service that lets you search for data that may or may not have been leaked called Canary [2]. The only thing I care to charge for is if you're a business and want to link this into your SIEM or whatever, I want you to pay up--otherwise feel free to search manually.

1.2 billion compromised credentials is a lot. In fact, it's so much that I found it hard to believe. Adobe's breach was no more than 150 million user accounts, and if you take a look at various sources [3], you'll be able to infer that the number of affected users overall is anywhere between 250 and 400 million depending on where or who you look at (Wikipedia was quick to cite in this case). Needless to say, I call bullshit on this count unless Hold Security is somehow getting its numbers wrong due to an extremely large amount of duplicates.

To put it all into context, Canary has about 476,000 e-mail addresses stored within its databases with about 350,000 potential passwords stored in a hashed format. This has been collected since mid-last year from over 1.1 million unique samples. It's completely automated so I am not sitting on random TOR websites collecting it myself, it's done without me having to take a look.

If you're interested in helping contribute to the project, I'd love to hear from you. I'd like to see us avoid having to rely on products like Hold Security's because leaked data is data that everyone should be able to know without having to pay a broker an enormous sum.

[1] [http://canarypw.wordpress.com/2014/08/06/canary-will-not- cha...](http://canarypw.wordpress.com/2014/08/06/canary-will-not-charge-you- to-find-out-if-youre-affected-by-a-breach-also-we-want-volunteers/)

[2] [https://canary.pw/](https://canary.pw/)

[3] [http://en.wikipedia.org/wiki/Data_breach](http://en.wikipedia.org/wiki/Data_breach)

<comment><USER3> That's assuming that the leak is from a single source (single business or website).

Is it conceivable that if you aggregate all leaks it would equal the 1.X billion number referenced?

The location of hundreds of thousands of WordPress and similar CMS sites are well known and constantly hit by botnets trying to bruteforce passwords and waiting for site admins to leave a site un-updated, I'm sure they get compromised all the time. I operate several honeypots just for fun and have a list of thousands of sites with malware on them and undoubtedly have had their databases stolen. Then everyone who runs a site (or has sufficient permissions) who is in that hacked database can have their site hacked, and the chain continues. If you delete the malware but fail to fix the security hole, the malware will be back on that site the next day.

Why does your Canary site not include larger leaks like the Adobe leak?

I think sites like yours are very important for protecting people's online security. Data mining is fun and being able to offer a service to help people stay safe online is a win-win.

<comment><USER2> Canary does not include data from the Adobe leak because I just haven't bothered. I have considered making it available but because there are many other solutions that are better for such a purpose [1], I'd rather just leave it to them and for the future link to them via Canary's site.

In the future I may consider a solution for this but for now it's just not on the priority list. It's a feature I'd like to have but I just don't have a practical way to do it as of yet.

Regarding the count, if you go through all the leaks, it barely gets above 250,000,000. This is based on many statistics from Verizon and other initiatives. My estimate is that it is probably less than double that but it cannot be much higher. It would take a few Twitter, Adobe, or Facebook-sized leaks to get to 1.2 billion because even if you had 50,000 Wordpress sites breached and they on average only have 5 accounts per site, it's barely going to make a dent in that 250,000,000 I floated.

If 1.2 billion accounts are floating about, someone hasn't spoken up and likely we'd have gotten wind of this by now.

Data mining is awesome. :)

[1] [https://lastpass.com/adobe/](https://lastpass.com/adobe/)

<comment><USER3> Adobe + Target + PlayStation leaks alone total over 300 million credentials.

AOL 2004 = 92 million

AOL 2006 = 20 million

Apple 2012 = 12 million

Blizzard 2012 = 14 million

BNY Mellon 2008 = 12.5 million

Cardsystems Solutions 2005 = 40 million

Evernote 2013 = 50 million

GS Caltex 2008 = 11 million

Heartland 2009 = 130 million

Living Social 2013 = 50 million

RockYou 2009 = 32 million

TMobile 2006 = 17 million

TJ Maxx 2011 = 94 million

US DoD 2009 = 76 million

US Dept of Veteran Affairs 2006 = 26.5 million

Yahoo 2013 = 22 million

Those are only the ones over 10 million before 2014, leaving out some foreign hacks like Auction.co.kr (18 million). (Also UK NHS 8.3 million, LinkedIn 8 million, Ebay, Target, LexisNexis)

I promise you that just over 1 billion is about the number of major company leaks that are publicly available. Yes, lots of duplicates between those leaks, but most contain additional valuable data besides just the credentials.

ATT, BCBS, Citigroup, Facebook, Gap, Gawker, Chase, Medicaid, Monster.com, Network Solutions, Nintendo, Sega, Starbucks, Twitter, Ubisoft, Washington Post, and numerous government and academic organizations all have hundreds of thousands to millions of credentials publicly available.

<comment><USER2> I might be willing to concede on this but you also have to take age into account.

Let's look at this useful spreadsheet:

[https://docs.google.com/spreadsheet/ccc?key=0AmenB57kGPGKdHh...](https://docs.google.com/spreadsheet/ccc?key=0AmenB57kGPGKdHh6eGpTR2lPQl9NZmo3RlVzQ1N2Ymc&single=true&gid=2&range=A1%3AW400#gid=2)

And here are the numbers:

Hacked: 880,575,016

Inside job: 137,714,840

Accidental: 63,322,485

Lost/stolen: 206,237,702

Misc: 1,825,350

Things like government-issued identification numbers and whatnot are the most severe, so how many people have been affected by that since 2004? If we break it apart by category we start to see how these breaches have become:

E-mail addresses: 530,991,405

SSN/PII: 327,471,624

Credit card: 335,772,083

Authentication: 430,756,146

Bank records: 4,270,000

For the first line, it's just a list of e-mail addresses. The latter four are the most severe. Out of that list, what is the most useful? I'd wager the SSN/PII, authentication, and bank records; credit cards are only useful for so long really.

This means we're at over 750,000,000 records that may be usable. However, with the authentication portion, we're looking at that being even more useless as time goes on. Accounts from 2004 may not be usable in 2014 either.

So yes. We have had over 1.2 billion records leaked, but really how much of that is at all useful? None of these take duplicates into account however.

<comment><USER3> Useful to me? None.

Useful to spammers? Half a billion emails is very useful.

Useful to hackers? Hundreds of millions of records.

Useful for fraud? Hundreds of millions of records.

Useful for data-mining and intelligence? All of them.

Many banks don't issue new debit/credit card numbers, they just change the expiration (the 3 digit code is rarely used from my personal experiences). It's easy to brute force the expiration.

SSN numbers and security questions can allow access to many accounts.

Figuring out password hashes (lots of methods) is sometimes easy, sometimes hard.

Bank account numbers rarely change, damage can be done with these.

How many people use their real information online? Most.

How many use secure passwords and change their passwords ever/enough? Few.

The LexisNexis breach alone is a disaster, they're basically data-miners with exclusive access to personal details.

</comment></comment></comment></comment></comment></comment><comment><USER4> Yep, this is all a show for Black Hat, and with many more interesting topics to discuss, it's sad that this bullshit caught the attention of the media as the lead story. At RSA 2014, the lead topic was the role of government, its link with industry, and the ethics of omnipresent spying. All I've heard from BH as an outsider to that conference is "Well, there have been a lot of compromises over the years, and some Russian group gathered them all up to make it a big number."

A tl;dr for the article's tl;dr: This is not a big thing; the list is largely outdated; there's no new breach; if it were a big deal, than more than just some salesman at BH would be screaming about it.

</comment><comment><USER5> That's a lot to read, but the gist of it seems to be that this guy doesn't care for Brian Krebs. Who is _this_ anonymous writer? Does this person have some sort of credibility other than the four posts on this blog?

For what it's worth, one of the other four posts is a complaint about Ryan Block's customer service call with Comcast.

<comment><USER6> The gist is NOT that this guy doesn't care for Brian Krebs. The gist is that Brian Krebs, along with Hold Security, worked together to create a panic frenzy among the uneducated[0], business class by publishing stories and doing interviews with major media outlets all to try to sell a 120/year service that is the equivalent of querying Russian forums and ircs for people mentioning your site in credential sales.

It's a total sham and, personally, I've lost a little respect for Krebs.

edit: [0] By uneducated, I meant only technologically, not anything else. Sorry if anyone was confused.

<comment><USER7> IDK, it seems like a valid business practice to me - like how stories of virusses running rampant boosts (ineffective) virus scanner sales, or pictures of six-pack abs boosts sales of ab crunching devices.

<comment><USER8> One weird tip to a safer computer. Click here!

Would you like to Run or Save NotAVirus.scr?

</comment></comment></comment></comment><comment><USER9> I was fascinated by the amount of people that asked me about this yesterday. People read about it on various websites but I was basically oblivious because none of my normal haunts were reporting it. I guess that means I'm getting my feeds from the right places.

This is the kind of shitshow that occurs when people try to outscoop one another instead of doing actual research. David Christopher Bell on Cracked.com has made a weekly mockery of the Internet chasing stories that turn out to be completely untrue.

<comment><USER10> > David Christopher Bell on Cracked.com has made a weekly mockery of the > Internet chasing stories that turn out to be completely untrue.

Not to be Captain Pedantic, but Mr. Bell seems to write movie stuff. Thinking of a different person, perhaps? I only ask because your comment piqued my interest enough to go look, and all I found was "8 Movie Plots Rendered Completely Implausible Because the Studio Didn't Want to Hire Script Writers That Were Worth a Damn".

<comment><USER9> Here's the most recent one: [http://www.cracked.com/quick- fixes/6-b.s.-stories-that-foole...](http://www.cracked.com/quick- fixes/6-b.s.-stories-that-fooled-everyone-facebook-8514/)

<comment><USER10> Thank you.

</comment></comment></comment></comment><comment><USER11> That entire NYT article read like an advertisement for HOLD Security, just like Chinese Hacker's article was for Madiant.

</comment><comment><USER12> Interesting article, and while I never really believed sensationalist headline I don't doubt that there are huge caches (100's of millions) of stolen U/P's being privately stored and exploited. The value of such lists drastically decreases if they become public.

Also while the story and coverage might be bullcrap it's increasing the awareness of security online. After my parents saw the story on the news they phoned me up and I got them set up with LastPass, so now they can finally stop using the same 6 digit [a-z0-9] password everywhere.

</comment><comment><USER13> Of course this didn't actually happen as it was reported. No where near that many passwords have ever been leaked. And the only thing they were telling us was "scary russian hackers get access to billions of passwords!"

I've found over the years it's easy to tell when a story about security isn't all that true. A story without any actual evidence behind it, that sounds very scary, is almost never true.

<comment><USER3> That's probably pretty close to the aggregate number of credentials leaked, and these leaks are easily downloaded or bought on the internet.

It wouldn't be difficult (and is commonly done) to create a large database of all leaked credentials.

</comment></comment><comment><USER14> Never trust anything you read on the internet unless you there are sufficient independent sources to verify it. In the heady early days of the web we thought it was a liberating experience; todays it's just an amplifier of whatever someone wants you to believe.

</comment><comment><USER15> +1 to the author for the the LoD-versus-MoD reference.

<comment><USER8> Never forget -- and thanks.

</comment></comment><comment><USER16> I didn't pay attention, apparently :(

</comment><comment><USER17> I think the notion of "stolen" passwords should be replaced with _pirated_ passwords, unless the attackers really did something different from just gaining access to and copying them.

<comment><USER18> Semantic arguments about the meaning of stealing are only appropriate in discussions about copying other people's stuff. When it's our stuff getting copied, then it's theft plain and simple.

<comment><USER19> No, just when it requires illegal access to a system, instead of legally buying the stuff and then offering copies to others.

<comment><USER20> Oh, I see. So THAT law, you like.

<comment><USER19> I said nothing about liking any law. I just said one easily equates to theft, while the other one doesn't[1].

[1] [http://en.wikipedia.org/wiki/Dowling_v._United_States_(1985)](http://en.wikipedia.org/wiki/Dowling_v._United_States_\(1985\))</comment></comment></comment></comment></comment>


[NSFW] Vibrrr – Transform your Android phone into a vibrator with just a webpage - <AUTHOR> https://vibrrr.com <comment><USER2> that is liquid damage waiting to happen</comment>


VoidLinux: A Modern Distro Based on the Xbps Packaging System - <AUTHOR> http://www.voidlinux.eu/

<comment><AUTHOR> The reason this distro is so intriguing to me is that the packaging system builds binaries in Linux containers and requires no root access.</comment>


Remember Aaron Swartz (1986 – 2013) - <AUTHOR> http://torrentfreak.com/remember-aaron-swartz-130113/

<comment><USER2> He's born just 5 years earlier than me but had done some real incredible stuff. hats off Aaron.

</comment><comment><USER3> I only knew of him, but I looked up to him. Belated, I know, but: thanks Aaron.

</comment><comment><USER4> Tragedy. One of my heroes for certain.</comment>


PGP: There's Life in the Old Dog Yet - <AUTHOR> https://blog.whiteout.io/2015/02/25/pgp-theres-life-in-the-old-dog-yet/ <comment><USER2> Since Keybase inevitably comes up in these PGP conversations, 2 things in advance:

1\. I've really been very slow in letting people into Keybase. The wait time is still 6 months (last night I was letting in people who asked back in August.) But now that we've added HackerNews key proofs, I'll let people in _today_ who notice this post who have more than 1 karma and write "I'm HN: {theirusername}" in the request form.

2\. We're not just about PGP keys. It looks that way now, and our first goal was to help solve the very on-the-record type things developers do. Which mostly requires PGP - and PGP is very good at. Signing code and commits, releasing software, etc. But sometime soon each of your Keybase devices will have a device-specific Nacl key, for example. And the process for provisioning devices is something we're taking very seriously - we'll soon have a big blog post about how public sibling key announcements will work. Here's the foreshadowing: with a Keybase account, very soon you'll be able to start with something as simple as a Twitter or HN username, and do some very cool, secure things with files and messages.

[EDIT] We are hiring. There's no specific job page because we've mostly been reaching out to people quietly, but we want help building: the Keybase client (Go), iOS and OSX GUIS (Obj-C), and the site (front end development and back end (Node)). Please reach out to me (chris@keybase) if you like the idea of working on usable security software. We are currently distributed in NYC, SF, and CHI, and ideally you'd work with us in one of these offices.

<comment><USER3> I hadn't really heard about keybase yet, but does it solve the only problem worth solving in encryption? Does it allow me to take any email address and give me back a private key that only the recipient can use?

If I wanted to know which public key to use I would just look at the recipients public website or ask that person through a phone.

The problem is that people don't create keys because nobody ever send them anything encrypted and people don't send anything encrypted because nobody ever sends anything encrypted.

But if keybase is just another storage for keys, what is the point? Especially if it is going to add a pointless waiting list that is just going to make the problem of not having a key even worse.

<comment><USER4> Easier PGP programs would increase the use of encryption, especially post- Snowden. I've walked people through using GPG4WIN/Kleopatra and it took a good half hour of explaining how everything works. People care about privacy, but unfortunately not enough to go through the hassle of manually encrypting everything.

<comment><USER5> Not sure I understand what you mean. For some family members I set up Thunderbird+Enigmail, which was very quick including keypair generation.

Now they communicate with me always encrypted, automatically. They don't have to know any details about encryption, except that they occationally have to type in their password.

</comment></comment></comment></comment><comment><USER6> I'm not really positioning TextSecure as a secure mail solution, but this article says that the TextSecure protocol can't support asynchronous operations or federated namespaces. To clarify, that's incorrect, the TextSecure protocol was actually designed from the ground up for asynchronous communication, and is designed to support a federated namespace: [https://whispersystems.org/blog/asynchronous- security/](https://whispersystems.org/blog/asynchronous-security/)

That leaves these points for PGP:

> It's been around for a long time, so it's cryptographically well understood.

Yes, but it's cryptographically well understood not to provide the properties we need.

> There are a variety of implementations and there is a small but vibrant and > highly committed developer community. > A small but active and committed > user community.

Small can't be emphasized enough, and this is the most important part to me. I think people have a tendency to look at PGP and conclude that while it might be painful, we have to put up with it for backwards compatibility. The truth is that it's a protocol with few contemporary redeeming qualities, and an active user base of ~50k. So why bother with backwards compatibility?

<comment><USER7> Still would like to see your source for the "50k active user base" claim.

<comment><USER8> He doesn't have one. Sometimes I wonder how difficult would be to sneak a sabouteur into the crypto community, who advocated for ditching proven and safe crypto tools, wanted his solutions on top of centralized servers owned by others, opposed distribution of signed binaries in FOSS markets, recommended the people to use stock OSs with binary and who-knows-what google libs inside, instead of installing the free counterparts, etc..

Is like people have forgotten that they should trust _only_ the algorithms and the implementations, not the persona behind them. And that persona, should be the first saying "don't trust me! trust my algorithms!".

<comment><USER9> The 50k number comes from the number of people who can reach each other through the web of trust.

If you take every key uploaded to the key servers ever, then it's more like 4 million. However most of those aren't active.

TextSecure, in contrast, has vastly more users than PGP has ever had, despite being a lot newer.

 _Sometimes I wonder how difficult would be to sneak a sabouteur into the crypto community, who advocated for ditching proven and safe crypto tools_

Sometimes I wonder if the best way to sabotage the crypto community would be to fill it with people aggressively advocating for obsolete technologies that have zero chance of any meaningful impact :)

Moxie is unusual in the strong crypto space, in that he is willing to make pragmatic engineering choices in order to ship a working product. That is why he and his team have had a lot more impact than most others have.

<comment><USER7> Having it installed doesn't mean using it, like you said, for the PGP keys being generated but not necessarily in use.

 _On textsecure usage_

In my experience, very very few people actually sign keys, and few people upload them on keyservers. Also, I know very very few people who use textsecure. I used it for a few month, trying to get others to talk to me with it and gave up. And I'm in the bay area working with IT people, most of my friends are IT people mind you, we're exactly the people who try that stuff all the time.

It doesn't mean its bad tech or a bad attempt at all, heck, I would LOVE for it to catch up. But I seriously doubt it has 4 millions active users or more.

 _On PGP usage_

From a key server perspective there are still a large number of keys being downloaded, refreshed and updated daily.

Let's look at Mozilla's stats and they just seem to peer with one low traffic guy and themselves... [http://gpg.mozilla.org/pks/lookup?op=stats](http://gpg.mozilla.org/pks/lookup?op=stats) that's 1k new keys a day, and about 500 updated a day. We're far from the 50k claim since it would reach that in 50 days no matter what.

Of course saying there's only, say a few millions PGP users doesnt mean AT ALL the same as 50k.

 _On misinformation_

Thus I'm calling bullshit. Purposefully misrepresenting data (large order of magnitudes, at that) is a big minus in my book. Being well-known shouldn't mean you can tell people bullshit - if anything, you have a greater responsibility when you do.

<comment><USER9> > It doesn't mean its bad tech or a bad attempt at all, heck, I would LOVE for > it to catch up. But I seriously doubt it has 4 millions active users or > more.

The 4 million figure was total users of GPG ever, not TextSecure. TextSecure for Android has had about 1 million installs.

However, it's also been integrated into the Cyanogen mod which added about 10 million users, and the tech has been integrated into WhatsApp, which got the underlying crypto code (though not the UI) to about 400 million+ users.

So if by "TextSecure" we mean the technology rather than the actual UI widget code, which is generic anyway, then it's in a whole other league by orders of magnitude.

<comment><USER7> You compared the total of PGP users to textsecure saying textsecure has more hence my reply on that (just clarifying)

Or least, that was my understanding of your previous comment You're basically kinda saying there is between 11millions and 400 millions users, and, you know, i dont trust that at all and would compare it to notepad/paint being installed on windows but not actually used.

Anyway, whichever tech really catches up would be a good thing. I just dislike the manipulation of information for marketing of one tech over another for fame, money, power etc. Specially when that means bad mouthing free and open source projects :)

</comment></comment></comment></comment></comment></comment></comment><comment><USER10> Fun facts about GPG's public key infrastructure:

1\. The protocol used to gossip public keys in the GPG keyserver pool is completely undocumented[1].

2\. The protocol for uploading and downloading public keys from the pool only ever made it to draft stage, and is now very outdated[2].

3\. Because the keyserver protocols are not establish, there is only one implementation that is written in OCaml (luckily it is still maintained)[3].

Luckily, these issues are fixable. We just need to spend the resources to fix them. What if GPG had the same resources as WhatsApp?

[1]: [https://lists.nongnu.org/archive/html/sks- devel/2014-02/msg0...](https://lists.nongnu.org/archive/html/sks- devel/2014-02/msg00007.html)

[2]: [http://tools.ietf.org/html/draft-shaw-openpgp- hkp-00](http://tools.ietf.org/html/draft-shaw-openpgp-hkp-00)

[3]: [https://bitbucket.org/skskeyserver/sks- keyserver](https://bitbucket.org/skskeyserver/sks-keyserver)

</comment><comment><USER11> Quoting Dan Geer: "Convenience, freedom, security - choose two."

Solid cryptography concepts were never easy to implement and use.

My main problem lies not with the Gnu/PG or PGP software implementations, but with the actual platforms. Do I trust my iPhone/Android/public internet cafe computer/the family computer? Is it compromised? What about your computer manufacturer? Lenovo, perhaps? Or Apple?

The problem of information safety does go far beyond software. Neglecting this is not an option.

There is certainly life in the old dog called pgp, but I am yet to be convinced of a _mass market software concept_ that allows me to have the same level of confidence in its security as the open source packages I have installed on my offline computer.

 _" The paranoid will survive."_

<comment><USER3> I probably shouldn't go around and disagree with quotes from people I have never heard about, but there is no law that means encryption has to be ackward, user-unfriendly or badly designed.

<comment><USER12> Can you name a single counterexample?

HTTPS: You can give up freedom and some security for convenience of 3rd party registrars! You can roll your own which is less convenient and likely less secure… Not really sure of the 3rd option here.

<comment><USER3> WhatsApp.

<comment><USER12> WhatsApp fails freedom and arguably security.

Let's put things in context. Would you trust WhatsApp to be part of the US nuclear launch chain? Or talkes between the leaders of US, France, China, and North Korea.

<comment><USER3> No and yes.

The type of encryption used in whatsapp is not something I want to authenticate in anything that has to be dormant and used in an emergency without a network, but that doesn't mean it is bad crypto.

Yes, why not? You still need to ensure that the people involved with whatsapp can be trusted, but even the NSA can't do that in every case (ie Snowdon).

</comment></comment></comment></comment></comment></comment><comment><USER13> It's actually quite simple. PGP/GPG is the equivalent of the S in HTTPS. It's a trust model + ciphersuite + tools and libraries for using it. Would you ask your users to use TLS directly? No, you would not. Would you be surprised if the users hated using TLS directly? Nope.

What PGP/GPG actually needs is a nice UI. What's been holding it back is webmail. PGP and webmail are incompatible. Now that mobile is starting to take over people are using non-web based mail clients again and PGP becomes viable again. Imagine integrating Keybase with Mail.app. Now when you send an email to people it's automatically encrypted and verified. As you compose your message you can see that <firstname.lastname@example.com> is known on Twitter as @exampleIgor and on Facebook as /exampleIgor and on HN as exir.

This is where we should be heading: building on top of the PGP protocol (and fixing the protocol where 20 year old warts are popping up, such as issues with forward secrecy).

<comment><USER14> The problem is that, on the whole, users simply don't care. They have more important things to worry about than email encryption (you know, stuff like spouse, kids, mortgage, partying, etc).

The only way I can see end-to-end crypto really being adopted is if it's turned on by default everywhere. The selling point can't be the security, because people don't care about security -- the selling point has to be something else. Anything that requires a manual adoption step is going to automatically limit itself to a very niche audience.

<comment><USER13> Which is why I proposed that UI: it works automatically. While using the system, it simply indicates to you as you compose the message "Yes, the email address you are sending mail to belongs to the person you intend to message" or "No, that email address belongs to someone else" or "This is a brand new email address." This way when you are emailing your lawyer to set up a will, or you email your accountant with your tax info, you can be sure that (a) you are emailing just the right person and (b) that only they can read the communication.

While setting up the system, you are simply required to link your identity. Twitter, Facebook, LinkedIn, GMail, HN, the government all should allow you to easily link your identity. This should be a normal part of your account creation (this can tie in nicely with another change: getting rid of passwords and using a browser UI for identifying yourself a la Persona, but PGP based). Basically when signing up for Twitter, it should optionally let you upload one of your public keys (via a nice browser UI no less). Twitter (Facebook, GMail, etc.) would then expose a nice API for querying by public key fingerprint: "Who has 0xDEADBEEF?" => "@DeadBeef". Revocation, key updates, all that should be automatic: you revoke your public key and issue a replacement, all your services update automatically.

This is the point where lots of people will raise concerns about privacy: "This means Twitter now must have access to my GMail account! WTF?!". No. This process must be voluntary and optional. If you want to have 200 different identities online, go for it. I don't care to know your real name, I just care to know that I am talking only to the person I intend to talk to. Think emailing a well known developer on GitHub, having never met them. You don't need their real name, you are only interested in them in so far as they publish OSS on GitHub. This allows you to prove that github.com/example is @example on Twitter, email@example.com, firstname.lastname@example.com, and HN user "example".

And the benefit to "regular Joe" is that when he emails his accountant, he can now send his credit card number and SSN without fear of eavesdropping.

</comment></comment></comment><comment><USER15> Addressing the Moxie Marlinspike post - PGP being flexible and comprehensive is not the problem. The implementations/libraries (including GPG) being a fossil from last century are.

From my own experience I can say that the libraries are not well designed - they couple things together (like key management and decryption) and make some things hard to do even if they are conceptually simple (like checking a signature without first building a key-ring data structure with all keys from a file).

Progress comes from many experiments, going in all kinds of directions in UI design, infrastructure building etc. You need a comprehensive fundations to base on these experiments on.

<comment><USER16> >You need a comprehensive fundations to base on these experiments on.

The problem is that people build this huge, powerful libraries, but nobody seems to write the "limiting" libraries that take a subset of the powerful base library and create an easy to use experience from the user.

I suppose this largely stems from "implicit knowledge": People who can write cryptography apps likely know enough crypto that they can't themselves identify usability problems.

<comment><USER15> Libraries are more powerful if they are decoupled. For example I wanted to build a website that would let you check message signature against public keyservers. This sounds trivial - but unfortunately the libraries that I've reviewed don't provide a 'check_sig( message, public_key)' function - but only a 'key_ring.check_sig( message )' function. This complicated the matter a lot - because I would have to download the public key and save it into a temp file and then build a key_ring out of that file (with some other complications that I don't remember right now). Enough to discourage me.

Huge does not mean powerful when it is hard to mix and match (and limit) the functionality.

<comment><USER17> we're using and contributing to openpgp.js and so far this has worked pretty good for us, because it provides a sane API, properly documented, coherent code style, CI tested. here's a snippet for decryption and signature checking in Whiteout Mail: [https://github.com/whiteout-io/mail- html5/blob/master/src/js...](https://github.com/whiteout-io/mail- html5/blob/master/src/js/crypto/pgp.js#L314L351)

</comment></comment></comment></comment><comment><USER18> "We make the client available on all relevant platforms, desktop, Chromebook, Android, iOS, Windows phone."

And on the [https://whiteout.io/](https://whiteout.io/) website: Chromebook, Android, iOS, Web, Windows and Firefox OS.

So no Linux version at all??? Isn't Linux a relevant platform?

<comment><USER19> OS X is also missing, isn't OS X a relevant platform?

Of course both are relevant, they just went for the most popular platforms first, plus FF OS, which probably was a kind of low handing fruit due to its HTML5 apps.

<comment><USER17> the chrome app will run both on linux and mac os. and windows. and the chromebook. same does the web app. one of the marvels of the modern web :)

</comment></comment></comment><comment><USER20> The main crux of this whole post is that PGP is still good it's just that it's hard to use and there's no mobile support. Unfortunately for whiteout, there are many different mobile PGP implementations that do everything these guys are planning to (like automatic key lookup) that already exist today. Notably among them K9 + OpenKeychain, which already support keybase.io and HKPS.

People who say that security is meant to be hard are missing the point, OTR has been around for a long time and is not only much more secure than PGP but more useable. If you want truly secure asynchronous messaging you should checkout Pond by Adam Langley, which is arguably the only traffic analysis and metadata resistant messaging protocol on the planet. Unfortunately PGP has been designed in a way that careful key management is pretty much a requirement if you want. PGP key verification is extremely painful on every implementation I've ever seen, and unless you do it you are vulnerable to extremely potent attacks which are the result of plaintext key sharing algorithms like HKP and LDAP.

I'm with moxie on this one, I get sad that people are still trying to build modern 'secure' architecture based on a ~20 year old cryptosystem. It sets everyone who cares about useable security back because you are continually encouraging people to cling on to the old paradigm. Unfortunately there is no clear, universal successor to the PGP kingdom... yet.

</comment><comment><USER21> Is it possible to have a PGP-powered layer that requires close to no setup from the end user?

This seems to be the issue, and until the encryption is, in effect, totally invisible to the average person on the street, I'm not sure we'll see widespread deployment.

<comment><USER17> your no setup scenario would be when security is handled on the host, which is not really an option for email w/ PGP. if you want gmail to do your de/encryption, you might as well use what you currently have with TLS

</comment></comment><comment><USER22> Would you trust your life on a protocal such as HTTPS?

How often is https broken? It seems like every six weeks OpenSSL comes out with a new advisory full of vulnerabilities. I'm assuming that the world powers out there have zero days in OpenSSL, they attack SSL and VPNs all day long. People will be harmed if they trust HTTPS to secure their correspondence.

It is great that you're trying to give PGP a new life, but you're weakening it by using a vulnerable protocol with it. The weakest link will always break first.

</comment><comment><USER23> Is anybody working on making identity-based encryption mainstream?

<comment><USER24> IBE requires a trusted third parties to issue private keys to users. That's essentially the lavabit model with slicker math.

</comment></comment><comment><USER25> Assuming you have access to your mail server to make the necessary configurations, wouldn't something like ciphermail's ([https://www.ciphermail.com/](https://www.ciphermail.com/)) pgp /smime gateway make it easy for joe user to have encrypted email?

I've used it in my own setup and think it works quite nicely so far but then again I only have 8 people using it.

</comment><comment><USER26> On thing that always wonders me. Assuming NSA or whatever spying agency is sniffing mail in the Internet, I'm pretty sure that once you use PGP, you are basically a strong suspect for them. Obviously the situation would be different if mail was encrypted by default, but now such people just stand out. Not sure if this is a situation I want to be in...

<comment><USER27> they have openly said that people who use encryption are of interest to them, or at least their communications are. however this is easily solved if there is wide enough adoption. If I encrypt the shopping list that I send to my other half, or I encrypt the text messages i send siblings or any text communication to anyone, and there are 100's of 1000's who do the same it becomes impossible to store/monitor and decrypt all that information. it would essentially overwhelm them with data, which even if they managed to decrypt (through the theft of private keys or a serious brute force attack) would be of no value other than to prove the task a waste of time.

<comment><USER26> Agree, if mail encryption becomes common, then it is completely different story.

</comment></comment></comment><comment><USER28> I have Exchange/Outlook (w/Gpg4Win) at work and OS X Mail w/GPGTools) at home. I can't send GPG signed/encrypted emails between work and home. But I can with the S/MIME implementation built into Outlook (w/certificate from Comodo, hmmmm). Add-ons for email programs just don't seem to be the answer, at least so far.

<comment><USER29> Hi Tom, gpg4win is really awkward in terms of compatibiltiy (beside other lacking features). I exchange a lot of encrypted mails with different people - among them Apple users. Take a look at gpg4o ([http://www.gpg4o.com](http://www.gpg4o.com)) which is my preferred solution. <USER29>
</comment></comment><comment><USER30> Gosh. I like the concept, but the web site really needs more information and better design -- starting with whether this is a free or paid service.

<comment><USER17> website update is coming soon :)

</comment></comment><comment><USER31> Why do hip startups always use the .io domain? It seems to have something to do with GitHub, but I don't know what.

<comment><USER32> cheap, newish (so simpler names are available), trendy, technically relevant phrase (i/o).</comment></comment>


How do companies like Blue Apron/HelloFresh find early adopters? - <AUTHOR> The idea of not ordering food delivery, and not shopping for groceries is new and a discontinuous type of innovation, which requires a change in consumer behaviour.<p>How do these aforementioned companies find their early adopters? Everyone eats, so how do they find people willing to try this? <comment><USER2> Press, advertising, outreach to their networks. The same way you find early adopters for anything!</comment>


Literate Programming (and why Docco isn't enough) - <AUTHOR> http://timeless.judofyr.net/literate-programming.html

<comment><USER2> I've messed around with my own literate programming system that was a more "true" system (where you have a kind of macro-like approach), and I have to agree: the docco approach is missing the key bonus: organizing your code into chapters and paragraphs and then statements.

Think of it this way, if you're building an API, you should consider usage. So you then generate examples. But not just one, maybe 3-5 of them to really do it right.

Then, as you proceed, you implement and build tests, probably down towards the end of "the chapter".

It's a different way of thinking about how your code is consumed. The "documentation systems" of the past (e.g., javaDoc) really are reference documentation on details of a method. But if you want to really grasp the system, you probably lead with examples. Maybe a picture. And you probably don't need to have to navigate to 7 different implementations of your interface Foo to figure out what the design is.

But having those 7 different implementation files is nice when you know the system later.

Where literate programming systems completely break down is the toolset. You typically can't work in "example down" mode and "single source file" mode at the same time - like, to debug, or make a small change. I've tried to build "diff and patch" between the source and the literate documentation, and when that's buggy, whew, that sucks.

</comment><comment><USER3> One idea about design I take from pg is that it's difficult to "even talk about good or bad design except with reference to some intended user." [1]

Now, I'm relatively new to literate programming, and perhaps the answer to my question(s) will be obvious to someone more experienced with it. What I wonder though every time I read about it is, who is the intended audience of all this extensive documentation? More specifically, how much knowledge of the programming language and problem space do we presume that the reader has?

There is something elegant about presuming the audience has expertise in both areas. That way you only have to worry about documenting one thing, the program, rather than three things: the program, the programming language and the problem space. Readers lacking sufficient understanding of the lang or problem space can refer to the more robust documentation about either one individually to get themselves up to speed. But I'm not sure Docco users are taking this approach.

\---

[1] <http://www.paulgraham.com/desres.html>

<comment><AUTHOR> Personally, I try to write at the same level that I always would for code, with maybe a larger preamble at the beginning.

Then again, I'm still a student of the ways of the literate myself...

</comment></comment><comment><USER4> Although I enjoy jashkenas' Javascript documentation a great deal and find it extremely helpful, I'm having a hard time getting excited by the resurrection of interest in literate programming. I enjoy and appreciate the notion that code should be designed by humans to read, but I'm reminded of (and agree with) Bob Martin's admonition in Clean Code that energy spent polishing one's code commentary is better spent improving the readability and flow of the code itself.

Also, comments that come across elegant and attractive when rendered in HTML look teeth-gratingly gratuitous when viewed in the context of the original code. For example,

      // Attach all inheritable methods to the SomeObject prototype.
      _.extend(MyLib.SomeObject.prototype, MyLib.SomeMixin.prototype, { ... })

Yes, that's an accurate description of what it does. But if you can read Javascript, you should know that; in code, it's visual clutter, and worse, it adds overhead to maintain it when you move or update your code.

<comment><USER5> Redundant comments are as annoying as repeating yourself in writing.

Redundant comments are as annoying as repeating yourself in writing (and I'm certainly as guilty of it as anyone). But the real point of the comments is to explain the "why", just as the code explains the "how". In your example above, to explain the reason that you've chosen to mix in the modules you're mixing in...

<comment><USER4> I was trying to be subtle about it, but the example I used was drawn from a fairly well-known (albeit, I should add, well-written) JS library. My point is that this style of documentation encourages that behavior, perhaps paradoxically because it's so attractive and big hunks of whitespace tend to stand out.

</comment></comment></comment><comment><USER5> Mr. Klabnik goes in depth into the argument that literate programming has to be tangled and weaved into a different order, in order to be considered "true" literate programming.

Respectfully, I have to disagree.

It may have been the case that a macro source-code rewriter was necessary before the advent of the subroutine, but in this day and age you can factor out functions as you see fit. Moving around source code with macros is an unnecessary obfuscation between the code you read, and the code as it runs. At any point where you're tempted to pull out a block of code and interpolate it later, just make it a function with a descriptive name, and hey presto.

Edit: Steve wrote it.

<comment><USER6> > _Mr. Klabnik goes in depth into the argument that literate programming has > to be tangled and weaved into a different order, in order to be considered > "true" literate programming._

> _Respectfully, I have to disagree._

If you're not re-ordering the code, there's no difference between a "literate program" and simple comments.

And I find code-reordering useful even in modern languages; most impose some structure like:

      1. module-level documentation
      2. exports
      3. imports / includes
      4. implementation (functions, procedures, etc)
         4.1 return type (static languages only)
         4.2 function/procedure documentation
         4.3 code

With literate programming, you can add supporting text (like export entries, module imports, or documentation) wherever would make the most sense for a human reader, not to the compiler.

> _At any point where you're tempted to pull out a block of code and > interpolate it later, just make it a function with a descriptive name, and > hey presto._

What if you've got a few dozen local variables? Do you want to pass them all as parameters, for every single logical block in your algorithm?

What sort of names do you give your extracted procedures? Literate programs can have block names like "Check for common post-traversal error conditions", which in a procedure name would likely be abbreviated to "traversed_errcheck()".

<comment><USER7> It seems that folding code to a natural language summary is what you really want.

<comment><USER6> Not at all; I want to be able to print stuff like this in the generated documentation:

\</comment></comment></comment></comment><comment></comment><comment></comment><comment></comment><comment></comment><comment></comment><comment></comment><comment>----

3.2 HTTPS driver

The HTTPS driver is currently implemented using either GNU TLS or OpenSSL:

== driver includes

      #include <gnutls.h>
      #include <openssl.h>

== driver list

      , {"HTTPS (GNU TLS)", driver_https_gnutls}
      , {"HTTPS (Open SSL)", driver_https_openssl}

== drivers.c

      static void
      driver_https_gnutls(DriverState *s) {
        ||common driver local storage||
        gnutls_init();
        gnutls_ctx_init(&ctx);
        check_extra_algos = "FOO:BAR";
        some_feature_flag = 1;
        // etc
      }

      static void
      driver_https_openssl(DriverState *s) {
        ||common driver local storage||
        openssl_init();
        check_extra_algos = "BAZ";
        some_feature_flag = 0;
        // etc
      }

\</comment><comment></comment><comment></comment><comment></comment><comment></comment><comment></comment><comment></comment><comment>----

<comment><USER7> I meant instead of pulling out a block of code and interpolating it later. For example instead of:

        <set up>
        <do stuff>
        <tear down>

And then describing them in the rest of the document, a better solution to me seems to be able to click on e.g. the <do stuff> part to expand it. This allows the reader to see the high level structure and drill down to the stuff he's interested in instead of spreading the code all over the place with references.

<comment><AUTHOR> That might be an 80% solution, but it still means that you're tying your presentation to the way that the computer needs to have things ordered. What if all of the parts need explained, but I really want to explain the <do stuff> before the <set up> and <tear down>?

</comment></comment></comment><comment><USER5> A bit more Twitter conversation here, for the curious:

<http://bettween.com/jashkenas/mentalguy>

<comment><AUTHOR> Man, I really wish that this tool showed stuff in forwards order rather than backwards order. And when I tried to flip it around, it showed a bunch of your previous conversations first instead...

</comment></comment><comment><USER8> org-mode in emacs with 'org-babel tangle' works well. But Docco looks great on the web.</comment>


Can you make money selling your data? - <AUTHOR> http://www.bbc.com/capital/story/20180921-can-you-make-money-selling-your-data <comment><USER2> Betteridge suggests No.</comment>


iPhone Hacking Attempt from the Police in Mexico? - <AUTHOR> Today a friend of mine who was kidnapped in recent days by the police of Guadalajara, Jalisco. Showed me a screenshot of her iphone with a message like this: error requesting configuration.
Diversion of: SMS. When you are not available
error requesting configuration.
Diversion of: Asynchronous data circuit. When you are not available
error requesting configuration.
Diversion of: packet access. When you are not available
error requesting configuration.
Diversion of: keyboard access. When you are not available.
I don't know what to think, i´m no expert. But we are very concerned about this, and how it may affect her. <comment><USER2> sounds like your friend was targeted by the gov / crime org</comment>


Ask HN: Help with my 'chicken or the egg' problem - <AUTHOR> I'm starting a for profit job board dedicated to the IT world in my area. There is a decent IT industry around me and it seems they are always hiring. I plan on offering fair rates, and eventually I would like to be able to offer a guarantee to the employers.
But for now I'm facing an oh so familiar issue. The old chicken/egg problem. Employers aren't going to want to pay for a service (even at the price I'll be offering) if there are hardly any users there looking for jobs. I've considered offering a large number of companies X amount of free listings to try and get potential job seekers to the website but I'm just not too sure how many companies will agree, and then actually spend the time to use the site.
I was also thinking of building a scraper to go and grab IT jobs from companies in the specific area that I am serving, but I'm not sure if this is really ethical. Could I get in trouble for scraping sites like craigslist, or monster.com?
I'm in the mindset now that I need to work on getting employers with little job seekers, rather than job seekers with little employers.
What do you guys think? <comment><USER2> Hey brandoncordell,

you could populate your site from listings from indeed and simplyhired. They have APIs for you to do that, so you don't have to scrape them. Plus you actually earn money when people apply for a job. Also, you can check out jobboarders.com it's a community of people hosting job boards, you'll find plenty of advice about starting a job board. Good luck :-)

<comment><AUTHOR> Great, thanks for the link and the suggestion!

</comment></comment><comment><USER3> Market research. Contact IT firms in your area and network/develop some rapport with senior people. Invite them to lunch (at your expense obviously) and ask them what their biggest recruitment headache is. After you've implemented the feedback you get, ask them if they'd be willing to take part in a closed beta/pilot project.

Everyone I know has a low opinion of IT recruiters, largely because they don't listen. Be the person that cares about their needs will be refreshingly different.

<comment><USER4> I think this is a great way off attacking it. Establishing a rapport and learning how you can set yourself apart from the other sites out there is crucial. Getting this info directly from the source and getting them involved in the process will be able to help you bring them on as clients!

</comment></comment><comment><USER5> One successful strategy I've seen is establishing a following first (stakoverflow, 37sugnals). It wasn't clear from your post if you are offering anything unique. that would be helpful to understand. One way to start out is to offer to operat a company's careers page. String a few together an you might hav a service.

</comment><comment><USER6> Why start Yet Another Job Board? What differentiates it from the many others?

<comment><AUTHOR> Well the thing that differentiates myself from the competitors is local edge. My job board will be for my local IT industry. All there is now for IT businesses in my area is Monster, Indeed, and Craigslist. All of which, I don't think are great solutions. Craigslist is very generic and full of spammers and scammers. Monster and Indeed are just large money making machines. I'm really just looking to make something a little more simple, and a lot more cheaper.

<comment><USER6> So your competitive advantage is that you can actually meet the people in your area.

Go to every single technology meetup of any kind and promote your site. Network with any and every person in your area that would be able to use your site. Build up awareness on both the company and employee side.</comment></comment></comment>


Big tech censorship reignites call for an Internet Bill of Rights - <AUTHOR> https://sociable.co/web/big-tech-censorship-reignites-call-internet-bill-of-rights/ <comment><USER2> Are there examples of this "censorship" available? I'm under the impression that "censorship" is something of a red herring.

Last month, I took a good look at Qanon twitter, and my opinion was that those folks are mainly having technical or conceptual difficulties when they claim "shadow bans" or whatever. A lot of folks haven't done the arithmetic about how many people they follow, rate to tweets and how many tweets they read in a day, otherwise they'd realize you can't possibly see everything all your follows put out.

It would be totally weird to have technically uninformed people dictate that, say, the CAP theorem be violated to protect their free speech rights.</comment>


How to take advantage of Redis just adding it to your stack - <AUTHOR> http://antirez.com/post/take-advantage-of-redis-adding-it-to-your-stack.html <comment><USER2> I've read about Redis before and heard how companies are using it, but never completely understood it's purpose. After reading this I can actually say I understand Redis now and how it's useful. Amazing that after hearing so much about it all it took was a relatively simple article.

<comment><USER3> On a basic level, you can think of it as just a faster version of Memcached.

Digging a little deeper, you can think of it as a faster, extremely powerful version of Memcached.

<comment><USER4> I'm not too familiar with memcached, but isn't one of Redis's strong points is the data structures it provides out of the box, and not simple lookups?

<comment><USER5> Yes, that's the advantage. Redis is a memory-based DB like memcache, but is more than just a simple key-value store.

</comment></comment></comment></comment><comment><USER6> We use a ton of Redis, but I think the main takeaway from this article applies to all "NoSQL databases".

The "movement" is about polyglot persistence and not leaving RDBMS completely. Pull pain points out into something that's a better fit. Rinse and repeat.

<comment><USER7> I'm a little concerned with the added complexity of decoupling the datastore into two different systems that are relied on for application logic. What are good strategies to maintain consistency between the two, in the event of a failure? In the first example, what if the SQL transaction succeeds but the redis one fails? Would you rollback the SQL transaction?

<comment><USER8> It's a big problem, at least for our application (chattybar.com - a chat plugin). We have 2 copies of every chat message sent, one in the DB (for persistency) and one in Redis for quick retrieval. Keeping them synced has a lot of edge cases, especially since multithreading means that adding things to the DB and then to Redis doesn't necessarily guarantee they'll end up in the same order.

<comment><USER9> If all you're doing is keeping a copy of objects in a memory store for quick retrieval you might want to implement a read-through caching pattern using Memcached (or Redis but I'd turn any persistence off). Most chat clients don't allow for editing of posts, so you could just use Redis as your persistent storage using lists and RPUSH.

</comment></comment></comment></comment><comment><USER10> What I get from this is that Redis is so powerful that its best to not use it as a simple read-cache where the database is still the cannonical source.

Its better to use it as the write-cache for complex datasets with the database being the backup.

<comment><USER11> Right. I believe we already have plenty of read-caches out there. Our web servers can implement them, we have Varnish and MemCache and ~100 others.

What Redis appears to do great is that it is much more than just a key-value store with CRUD operations. You can model sets, lists, queues, counters, and do complex operations on these in-memory values.

It looks to me like it is best thought of as providing an interface to data- structures that are well optimized and persist across requests.

</comment></comment><comment><USER12> Very good article: there is some good exemples on how you could use Redis.

We just added it to our stack for caching and storing sessions.

It's blazing fast !

We're now trying to use it for different other purposes; autocompletion, counting and ab testing.

</comment><comment><USER3> Resque and redis-store are like auto-adds for almost any Rails project I work on these days.

Resque is for background jobs (with many add-ons for locking, scheduling, retries, etc.), and redis-store is a drop-in store for Rack::Session, Rack::Cache and Rails.cache. Easy and super fast.

</comment><comment><USER13> In your first example, you use redis to cache the id's of the latests comments, with a fallback to SQL in order to populate the list. However, you still need to call the DB to load the comments. I don't see the gain here.

Yes, you've replaced a "select * from comments order by created_at limit 10" with a "select * from comments where id in (list_of_ids_from_redis)".

Wouldn't you cache the comment models in a top-10 list?

<comment><AUTHOR> Often the DB will show unacceptable time to reply to the ORDER BY stuff but will fetch comments by single ID without problems. But when that second part is a problem as well it is a good idea to use Redis as a "vulgar" cache for items as well, so that the recent stuff are probably into a Redis hash and you can fetch everything with a single pipelined MULTI/EXEC call (and fetch all the items returned as nils from the DB).

<comment><USER14> A composite index will work wonders for this situation. EXPLAIN is your friend.

</comment></comment></comment><comment><USER15> Does anyone know if hdf5 would be an acceptable optional replacement for the current Redis disk format?

I have a console app that's backed by Redis (in much the same manner as described in this post), but I save my sessions to h5 when I switch between datasets. That means I need to combine the Redis data with my app data and export -- I do this using two separate h5 files, with with the appropriate links.

It would be nice (for me anyway) if I could do a Redis-native save, and move the resulting file. That would also improve my startup times when I reverse the process.

But, while h5 is nice for My data, I can't say it would be any good for generic Redis data...

thoughts?

</comment><comment><USER16> These little fixes are how I got into Redis and a month or so later, it's a primary data store (with disk based fall back) and I find myself doing 99% of aggregation and temporary storage operations with it.

Really great tool for the belt.

</comment><comment><USER17> Would it help game servers to run it with Redis? Like say, a FPS or RTS server.

I'm not sure of the typical game server stack though.

<comment><USER18> Usually the game server would keep everything in the game instance in memory, for an FPS or RTS. As great as Redis is, it's way more cumbersome than having everything in your data structures ready to go.

Redis is suitable for persistence, so where it would really be appropriate would be storing things like player profiles and other misc data that is not tied to a specific server. In fact, that's exactly what I'm using it for.

<comment><USER19> I realize that for most FPS or RTS titles, Redis might not be a perfect fit because the dataset is small for each map and the number of players never exceeds some preset limit like 64, but I think Redis might really shine as a means of storing persistent world data - for example MMOs, without having to shard heavily. Most modern MMOs use heavy sharding and can really only support a couple hundred players in the same area or shard at the same time. With Redis, you might be able to handle many thousands of players in the same area without sharding. You don't need sub-millisecond response time - as long as you can do most operations in a few milliseconds, things will be fine, since most players are on ~100 millisecond WAN connections anyway.

You might finally be able to break the model that current large games like WoW use - heavily sharded with all persistent data stored in Oracle. Redis should allow you to shard less and use SQL less, resulting in a much better experience, especially when parts of the virtual world get crowded.

<comment><USER20> I would be extremely surprised if WoW has issues when lots of people are crammed in a small space due to their Oracle backend. More likely it is all the calculations that the serverside is performing. I know a few people who work on other MMOs and they keep _everything_ in memory, even accounts for logged off users.

I'm curious: what you think (or know?) they are doing that is causing the database to be a limiting factor?

<comment><USER19> I didn't mean to imply that the bottleneck was due to Oracle. I was saying the exact same thing you are saying - it's due to memory or CPU limitations in their application server, which forces them to heavily shard.

If Redis can help them scale from the small shard to larger shards that support more players in an area, it might be interesting.

</comment></comment></comment></comment></comment><comment><USER21> What are some inexpensive cloud options to run redis for large data sets up to say a gb or so?

<comment><USER3> "The Cloud" isn't always the right solution for everything. If you're looking to go inexpensive, you can put it on a 2 GB Linode VPS on a ramen budget.

<comment><USER5> The confusing thing is some people refer to VPSs as 'cloud', which sort of makes sense when you compare them to EC2. EC2 seems more 'cloudy' because... um, we don't know what kind of hardware it's running on? I don't know. I was trying to explain what 'cloud' means to my tech-savvy, but non-industry brother and was puzzled by the issue of whether Linode is a 'cloud' service or not.

<comment><USER3> I guess it sort of could be, but I'd consider a "cloud" service something where you either a) can spool up or shut down servers dynamically from an image without any additional configuration (ala EC2 or Heroku), or b) a commodity service like Amazon S3 or MongoHQ where you pay by usage and they handle resource allocation.

<comment><USER5> I see scriptability and automation through an API as important to 'cloud' classification too. Good point about the automatic resource allocation, I think that's actually the key. On Linode you're paying for a segment of a specific machine, more like a dedicated server than EC2.

</comment></comment></comment></comment></comment><comment><USER22> This is a great example of how to promote adoption of a new technology. More companies should pay attention to how their product can be used rather than what their product is.

</comment><comment><USER23> Anyway to use Redis in App Engine setting? Or does Google have similar service in App Engine?

<comment><USER24> No and no. They only offer Memcache.

</comment></comment><comment><USER25> "just adding it to your stack" is the best conclusion that i have seen so far. Great article!

</comment><comment><USER26> I'm looking forward to the book</comment>


Udacity Statistics 101 (a critique of Udacity's Stats class) - <AUTHOR> http://www.angrymath.com/2012/09/udacity-statistics-101.html

<comment><USER2> This articulates a lot of the misgivings I have about these attempts at reinventing education online. Fundamentally, the problem isn't a lack of technology in education--Udacity took a bad stats class and gave it to the world. Not a great way to prove their worth.

Good classes are hard to make and run. Technological facilitation can make a good class better and it can bring a good class to the world, but the limiting factor here is still quality.

As an aside, Thrun also demonstrated that in many cases those who "do," cannot teach. And he should continue to "do," and find better teachers to do the teaching.

</comment><comment><USER3> I did not go through the stats course from udacity, but went through the build your own robotic car class. I would like to say the opposite of what the author of this article says. The conventional education system is designed to suit people who lean deductively. They prefer to "stand on top of concepts they know"... Then there are people who like to learn by dabbling. People who learn by tinkering. Such people find too much formalism annoying.

Think of it like this... If your child is trying to stand up and walk, dont try to get its gait perfected. Let it enjoy the success of barely standing and moving a few steps forward. I would say "every child has the right to enjoy this moment"... Not just the Usain bolt's and the ballet dancers. I think irrespective of the subject, the initial learning event is just scaffolding. I don't think the best ballet dancer remembers any part of how she actually learnt to walk.

When I attended the robotics class, I left knowing that "Multiplying gaussians is a cool thing" I figured the skeleton of how particle filters work. Will I drive a car for which I wrote the code. Absolutely not. Will I ask my enemy to sit in a car for which I wrote the code? Absolutely not. But I still got something. Something beyond frustration of being less than "super genius". I left saying "hey that is neat". Tomorrow when I invoke a Kalman filter in say OpenCV, I will feel less insecure about the abilities of that "magic black box". To me the robotic car became a "product of extreme passion created by brilliant hackers" than "mysterious devices created by other worldly geniuses"

If I wanted definitions I would look up Mathworld or something better. If I am going to write a paper on genetics, I am not going to use a stats 101 lecture notes as my reference. But here exists a phase in my learning where I actually do not care about the difference in dividing by N or N-1 or N+1. What the guys obsessed with formalism do is that they make that phase such a huge pain, that I bother not to cross it.

I get a way lot more "per hour invested" or "per dollar invested" from online education. From my point of view. That is the major disruption. The barrier to exploration has been almost eliminated. I could dabble with writing science fiction and then take a peep at anatomy.

Somethings like "A guy walked in and it was left in the video" is a positive sign. It shows that they are a scrappy startup creating a minimum viable product. If people start using this product even without the polish, it indicates real value. This is a standard startup practice. Better production quality means fewer classes and slower growth and less data for analytics. So I actually congratulate them for not obsessing with stuff like Video quality. Do you have any idea how google looked when it was released?

Does it have its shortcomings? Yes it does. Am I glad such alternatives exist to "Colorless, Odorless, Tasteless, Expensive Credentialling". Between Stanford or MIT and an online course I would definitely go to the Physical school. But between a tier 2 Indian Univ, I would prefer an online education and use a university only for a credential.

</comment><comment><USER4> Neither MIT OCW or Khan Academy offer a structured introduction to Statistics. Are there any comparable online courses that are better?

Side note: The course is on MIT OCW, but it's simply notes and exams.

Edit: It looks like Coursera recently launched a course: <https://www.coursera.org/course/stats1></comment>


Ask HN: Skype account hacked and my contacts were asked for money via WebMoney  - <AUTHOR>

Woke up today to realize that my skype contacts received a frienly messages from me while I was sleeping, asking them to land me money via WebMoney, my password for Skype was not changed, I still was in control of Skype account. I changed the password right away. Some of my friends sent money to hacker.<p>Do you have any idea how could he do that without changing my password (my password was a combination of letters and numbers). And are there any steps I can trace him now? <comment><USER2> > Do you have any idea how could he do that without changing my password ...

Of course -- he logged on as you. How could Skype (or your friends) distinguish him from you, if both he and you use the same username and password? In fact, for his purposes, not changing your password was to his benefit, because it helps him cover his tracks.

> ... my password was a combination of letters and numbers

If your password was robust, then I recommend that you scan your system for a keylogger. Also don't use the same password in more than one place.

<comment><AUTHOR> I wonder if skype has a way to see IP addresses of all login attempts.

Do you know any good keylogger scanner for Mac?

<comment><USER2> > I wonder if skype has a way to see IP addresses of all login attempts.

That's not the operative question. The operative question is whether they would be willing to share their activity log with you.

> Do you know any good keylogger scanner for Mac?

Not being a Mac user, I can't say what's good, but start here:

[https://www.google.com/search?q=macintosh+malware+scanner](https://www.google.com/search?q=macintosh+malware+scanner)

</comment></comment></comment><comment><USER3> 1\. What was your password? If it's been changed, was completely random, and never used anywhere else, you should have no problem posting it here.

2\. There's an Adobe Flash zero-day out there: [http://www.macrumors.com/2014/02/04/adobe-flash- update/](http://www.macrumors.com/2014/02/04/adobe-flash-update/)

<comment><AUTHOR> 1\. Password was only used for skype, it was "blocks12", was set around 5 or 6 years ago only for skype, I have moved to password management apps lately and all my current passwords are 20+ character generated strings and if two step authentication is available I always choose it. However I completely forgot about skype since I opened it so many years ago.

2\. I have it already installed.

Should I reinstall OSx or is it just a skype hack?

[edit] when I went to download the adobe update - turns out I already have it installed...

<comment><USER3> Ya, hard to say. If it really isn't related to your username or identify somehow, it would be hard to think that someone would brute force you specifically. Not that it's a very strong password.

[https://code.google.com/p/skype- bruteforcer/](https://code.google.com/p/skype-bruteforcer/)

looks like a 60 second timeout between attempts.

<comment><AUTHOR> But if the 60 second timeout is IP specific - he could have work around it by using botnet?</comment></comment></comment></comment>


Ask HN: How common is illegal web scraping? - ng-user With web-crawlers being so prevalent today, and the only thing really stopping them is a Terms & Services page or robots.txt file, how often are those broken or plainly put ignored?<p>Asking out of curiosity and general naïveté on the topic. If it doesn't make a ridiculous amount of requests and won't trigger any flags on the server, what's stopping people from building large db's full of illegally scraped data? What can we do to prevent it? Surely it must be going on.. <comment><AUTHOR> Web scaring is an "arms race". I was asked this weekend by a client, who found a Python scraping library for their sit on Github; how to avoid scraping.

Either remove your content from the internet or put up with it like we do spam.

The client linked this: [http://stackoverflow.com/questions/3161548/how-do-i- prevent-...](http://stackoverflow.com/questions/3161548/how-do-i-prevent-site- scraping)

But the problem with all these approaches, is the arms race problem. These solutions take developer time, and they can affect end-users. The army of scrapers can easily undo your efforts in short order making alot of these approaches an effort in futility.

<comment> ng-user Thanks for the feedback and the helpful link, I appreciate the comment!

</comment></comment><comment><USER2> Illegal?

Someone's TOS is not a law but mere a wishful suggestion to others who usually won't bother to read it anyways.

"You are not allowed ..." is the most laughable statement in TOS.

TOS may however remind of existence of laws.

Laws do or do not allow, TOS are not.

Laws do exist to protect copyrights and trademarks but scraping per se is not illegal.

<comment><USER3> I'd say the law disagrees with you there.

* Facebook vs Power.com [0]

* AP vs Meltwater [1], where the ruling decided fair use did not apply to World Wide Web content unless explicitly written in the TOS.

There have been a ton of other smaller cases on this.

US law currently says that scraping, is trespass.

That being said, I find this utterly preposterous. How can one know that one may only access the data in a certain form, and not manipulate or publish it in certain ways, when one must first make a request to see that information?

[0] [https://www.techdirt.com/articles/20090605/2228205147.shtml](https://www.techdirt.com/articles/20090605/2228205147.shtml)

[1] [https://www.scribd.com/document/131847330/Meltwater-AP- Rulin...](https://www.scribd.com/document/131847330/Meltwater-AP-Ruling)

</comment></comment><comment><USER4> It is happening. You can not stop it, only slow it down to the "human" pace. Any scraper performing actions with a browser (be it headless or with head) in a human pace can not be detected. Every human action (on a webpage) can be simulated so any data accessable by a browser (be it behind a login or not) can be retrieved. Certain captchas are also hard to crack but that is about all you can do: rate limiting and captchas.

</comment><comment><USER5> You can put up ToS all you want. I'm not agreeing to your terms by visiting your website (I'm not even visiting the site, my scraper is. Can a scraper sign a contract? I doubt it). Maybe law is f'd up like that in the US. Is it? Fine, I'll buy servers in Russia. Or some banana republic. Come sue me in the wonderful state of Nevis and see how that goes.

You can do a lot to prevent it. The best way to prevent it is to not have valuable data. The more valuable your data, the more effort we will spend on cracking your countermeasures and we will always win because this is our core business - to you its just a cost center.

Linkedin is one of the most notorious sites for trying to prevent scraping and they certainly have the funds. Yet they can't do shit about it and you'd think that they have it easy because they hide everything behind a paywall. Yet they can't prevent it from happening. Not even close. If they can't do it, you probably can't either.

And that's the really boring part. Do you know how many blank APIs there are in the web? People do their node.js and their frontend SPA bs and then they just dangle an API that is open for the world to scan. I could make a business out of scanning for exposed user data and feed them into a lawyer doing class- action lawsuits all day. Would be easy. Like really easy. So-called "engineers" need to learn how cryptography works. Or just reject unauthorized requests. Its really not that difficult.

To call out one prominent example: The Tinder API has been exposed ca. 2012 and ever since then, they didn't give enough of a shit to secure it. You can still build a tinder 3rd party app using their api.</comment>


Kevin Rose interviews Trent Reznor - <AUTHOR> http://revision3.com/diggdialogg/trentreznor/

<comment><AUTHOR> The interview is a bit long (around 45 minutes) with some fluff (what are your favorite gadgets?) but Reznor clearly recognizes in what direction the music industry and online content are going, which I found very interesting. Here's a quick summary of some of the more interesting bits:

\- Reznor explains that since music is essentially free due to downloads, he doesn't feel it's necessary to require payment.

\- When they offered the option to pay, only 18% of people did so. But, offering limited edition physical products worked out well.

\- His feeling is that it's more beneficial for a greater number people to hear his music, even if it's for free, because the result is more ticket sales / merchandise purchases.

\- They're looking to build a platform for artists to distribute their music which will give power to the artists. They're still playing with potential business models that are fair for the artist and fan.

\- NIN is developing an iPhone app to connect fans by location, and will hopefully create a "live blog" of concerts for those who can't physically be there.

\- They're continuing to work on Year Zero with either a serial TV show or graphic novel, which will integrate the album and ARG [ [http://en.wikipedia.org/wiki/Year_Zero_(alternate_reality_ga...](http://en.wikipedia.org/wiki/Year_Zero_\(alternate_reality_game\)) ] that they produced.

\- He has had some interest in live-streaming concerts, and is pushing for venues to have an open-camera policy so that fans all over the world can enjoy the experience, even if they can't physically be there.

I'm not a NIN fan personally, but it's nice to see a prominent figure in the music industry with their head on straight.

<comment><USER2> Reznor is a great guy, but I hope there's a future where people who make awesome stuff can get paid based on that, not based on charity, limited editions, and tie-ins. We need a new economic model for goods with real production cost but no reproduction cost.

<comment><USER3> >We need a new economic model for goods with real production cost but no reproduction cost.

Not only do I think that is impossible but I'm not convinced that it is a good idea. Has any economist ever done any quantitative analysis when the marginal cost of (re)production is zero? I suspect that there are only obvious models that answer the question by saying that goods with production costs of zero should be priced at zero. In the short term there are all sorts of things you can do to maximize profits when the marginal cost is zero, but in the long run the price will fall to zero.

This wistful hope that recorded music will be worth anything is kind of like my wistful hope that a viable economic system will come along that doesn't have a nasty tendency to concentrate the wealth in the hands of those that already have the capital.

Is that future you hope for really better than the one where something with a marginal cost of zero is priced in the market as it should be? I strongly suspect that the total amount of music recorded over time will continue to increase even as the cost falls to zero. I'm not sure what is so great about being able to sell digital music anyways. The limited editions seem infinitely superior to me as a consumer and are a great way to add a little bit of scarcity to a good that isn't scarce.</comment></comment></comment>


My first side project as an entrepreneur Looking for feedbacks - <AUTHOR> Hi there, I've just launched Easybook.io: a bookkeeping software really easy to use, made especially for freelancers, digital nomads, small business owners... One of the main difference with our competitors is the Pay-As-You-Go Plan where you pay only when you use the platform and create a document (estimate, invoice and expense only). That will help you to save a lot of cash! I'm looking for feedbacks from the Reddit's community on the service. Would you be interested in this? Yes/No/Why not? https://easybook.io/ <comment><USER2> Add "Show HN" to the title and the title should be the name or description (rather than generic "looking for feedback". Then it will show up on [https://news.ycombinator.com/show](https://news.ycombinator.com/show) where much less upvotes are needed. It's a category just for makers to present what they've built.

> I'm looking for feedbacks from the Reddit's community on the service.

facepalm</comment>