Stop forcing arbitrary password rules - <AUTHOR> http://ryanwinchester.ca/post/stop-forcing-your-arbitrary-password-rules-on-me <comment><USER3> I completely agree with the author, but... what he completely misses is the biggest annoyance: maximum length rules.

I'd probably sacrifice my firstborn if Microsoft would finally wake up and accept passwords longer than 16 goddamn characters...

<comment><USER4> No, the most annoying thing is when they don't tell you the rules in advance, and you start typing a password that fits most standard rules.

Now either you get through but don't know if you could've used a better password. Or your password will be denied and you have to make adaptions or change completely.

<comment><USER5> No, the most annoying is when you paste your password manager's auto generated long password and it silently get cut after 16 characters but successfully let you create your account. But now the login form doesn't have the same max- length property and your password will fail.

Hopefully the reset password form is synced with the login form.

<comment><USER6> Sounds so insane I guess it must have really happened.

I had similar issue with paypal once. Their bank account number field expected to get N digits without spaces, so it was hard limited to N characters. Of course they didn't bother telling about that. When I pasted my number with some K space separators tossed between digits, the last K digits have been silently truncated. Boom, account locked.

<comment><USER7> Happened to me this week. "Luckily", they also implemented the "feature" where they can send you your password by email directly after you set it, so when I couldn't get in with the password I set, I reset the password, got a mail with a new password, logged in, changed the password to a new random long string, and this timeasked them to send the password to me. The password they sent was chopped off. That told me their limit, so I could set my password a third time to get something maximally secure.

<comment><USER8> So, they emailed your password to you? Plaintext? They store passwords in plaintext? Paypal?

<comment><USER7> _directly after you set it_

</comment></comment></comment></comment></comment></comment></comment><comment><USER9> One startup I was working for were enforcing different rules depending on where you were setting the password e.g. 4 characters when you follow the forgotten password procedure, 8 characters when registering, 12 characters when changing password normally.

Another thing, a visa broker in Turkey had a so ridiculous password rule and gibberish error message, I had to read the source code and parse the regex by hand. Only then I was able to type in a valid password.

<comment><USER10> Your password has expired :)

</comment></comment><comment><USER11> I agree with the point the author is making and generally am just as annoyed by arbitrary password rules; nevertheless:

> chili dog monkey nutso

Is definitely not 18 quintillion years at best it's approximately 250000^4 which is 2^72.

A good (though still requiring some memorizing effort approach) would be something along the lines of [https://github.com/bitcoin/bips/blob/master/bip-0039.mediawi...](https://github.com/bitcoin/bips/blob/master/bip-0039.mediawiki) which uses a standard dictionary of 1626 easily memorable english words with each password being a 12 word mnemonic phrase. 1626^12 ~= 12^128

<comment><USER12> _it 's approximately 250000^4_

If the attacker _knows_ you've used four dictionary words in a row. Now you need to multiply the number by the odds of _that_..

<comment><USER13> And the entropy of the password 0b0 is only a single bit - if the attacker _knows_ you've used a single-bit password. Hashed into the SHA-256 space that's only a 1:2^256 chance of someone colliding with it. Still, is that a password you'd set?

What does that have to do with the output of the password in terms of brute- force guessing? Aren't you kind of assuming that an opponent won't try a broad dictionary attack? Equal distribution of hash outputs fundamentally assumes equivalent distributions of hash _inputs_ , and that's not true for the distributions of common passwords.

Again - everyone knows the passphrase patterns everyone uses, everyone knows the password transformations everyone follows (eg l33t) and overall those patterns are significantly weak against attack because of this public knowledge. By those algorithm, with knowledge of salts/etc those passwords are very weak.

Should you use a song, or a slight modification, or a l33t transformation? Certainly not - if the input space isn't uniformly distributed across the input space, the probability of hitting pay dirt definitely isn't uniformly distributed across the output space.

Even the class of "valid transformations of sensical English sentences" is not an equal distribution because of the above. If you can compress a language you can attack it, because by the assumption of compression the data is not at its minimum entropy. Maybe not feasibly if you're lucky - but don't bet on it just because Webster's is big.

<comment><USER14> What are you going on about? Your giant post did nothing to refute the point that an attacker doesn't know you are using 1 word vs 4 words vs 12.

Also, you seem to imply that there is a distinguishable difference in sha1 outputs of good random inputs and English word inputs, which violates a security property of cryptographic hashes. So there is a lot of fame to be had if you can prove that.

</comment></comment></comment></comment><comment><USER15> 100 times yes. The mere presence of a single character from a different class does not increase entropy. It means that most common passwords move from 8 lowercase letters to 6 lowercase letters plus 1 number plus 1 symbol, which is even easier to search.

If you have a sophisticated entropy estimation tool, or library, great. Another simple thing is using a min length, plus restrict from a dictionary of the million most leaked passwords to outlaw those (can hold it in a bloom filter if size is a concern).

<comment><USER16> Must contain 2 of the following, (upper case letter), (lower case letter), (number), or (Symbol) is higher entropy than 8 lower case letters.

<comment><USER17> Mathematically, but not socially.

It means that humans reduce the entropy across the board to (more than, IMO) compensate for the mathematical advantage.

<comment><USER16> Do you have any data on this? Last I check pick to was a slightly better option.

<comment><USER17> I don't have any real data. I know my own behavior for "sites I don't care about" and every once in a while I'm exposed to someone else's password (when they tell me their password for a site they don't care about, or where I see a password in plaintext in our shared password manager).

There's an awful lot of "capitalize the first letter" and "replace e by 3, a by 4, o by 0" and "sometimes append a bang". That results in meeting the password complexity rules without any additional entropy in the first 2 rules and 1 bit of entropy for the last rule.

</comment></comment></comment></comment></comment><comment><USER18> Second worst password rule: preventing to paste in the password field.

As seen at The OS X FileVault dialog, PayPal, Blizzard and many more....

<comment><USER19> Chrome hackaround:

1\. Right-click the field, Inspect element.

2\. Go to Console tab, type: $0.value = 'mypassword';

$0 is the last selected element: [https://developer.chrome.com/devtools/docs/commandline- api](https://developer.chrome.com/devtools/docs/commandline-api)

<comment><USER20> I have this as a bookmarket named 'Reveal Passwords':

        javascript:(function(){var IN,F;IN=document.getElementsByTagName('input');for(var i=0;i<IN.length;i  ){F=IN[i];if(F.type.toLowerCase()=='password'){try{F.type='text'}catch(r){var n,Fa;n=document.createElement('input');Fa=F.attributes;for(var ii=0;ii<Fa.length;ii  ){var k,knn,knv;k=Fa[ii];knn=k.nodeName;knv=k.nodeValue;if(knn.toLowerCase()!='type'){if(knn!='height'&&knn!='width'&!!knv)n[knn]=knv}};F.parentNode.replaceChild(n,F)}}}})()

</comment></comment></comment><comment><USER21> What we really need is to stop using passwords.

Passwords are awful UX design solution. The users have to think of and remember some meaningless phrase to get the service they really needed. If you use easy password, your account can be hacked. If you use difficult password, you won't remember it in a week.

For some people remembering a password or login might be especially difficult.

You can use software to generate and remember passwords but all of them will be lost if you reformat your drive. Some of password managers are proprietary, non cross-platform and some would upload your passwords to the so called cloud so NSA can look at them too.

In any case your passwords can be easily compromised when your PC is infected or you use somebody's else device.

What we need is to get rid of this obsolete system. We need hardware authorisation key (that could look like a real key) that can be used for both registration and logging in and would generate and securely store private keys for all used services. Such device should use strong crypto, should not allow exporting private keys, update firmware or operate without user confirmation.

I think such kind of keys will appear sooner or later, they are superior to passwords and password managers and easy to use but I am not sure that they will be open source, cross platform, and backdoor-free unless we do something in advance.

<comment><USER22> No. They are not superior to passwords - they merely provide different tradeoffs. For example, the site that requires your hypothetical key-type key would require that either all of their customers have spent money on such a key (bad) or that they are willing to pay for keys for all of their customers.

This is something that has been thought about in detail - <USER2> [https://vtllf.org/blog/ssh-web-sign-in/quest-to-replace- pass...](https://vtllf.org/blog/ssh-web-sign-in/quest-to-replace- passwords.png) for Stajano's analysis.

<comment><USER23> More importantly, you just replaced the problems of virtual keys with the problems of physical keys: you can lose them. Recovery is much more difficult than recovery of a virtual key.

<comment><USER24> Really? You don't lose (forget) virtual keys? It's a far worse problem. A physical key can _always_ be found if you look hard enough. A virtual key can truly be lost forever.

<comment><USER25> > A virtual key can truly be lost forever.

Sure, but it's super easy to make backups of virtual keys. It's a digital file - just copy it.

Worried one backup isn't enough? Make a million. Space is cheap.

<comment><USER24> Ok, but that makes is harder, not easier, to manage virtual keys. And the more copies, the weaker the key (the easier to find). A million is a terrible, terrible idea.

No, I despise passwords. The whole scheme is backwards. I don't want to authenticate myself to the server; I want to authenticate the server to me. Why doesn't _it_ provide the password, and my computer verify it? Why is a fallible human being in this game at all?

<comment><USER25> > A million is a terrible, terrible idea.

Well, yeah. That was a hyperbole. I didn't actually mean that you should go out and make a million copies of your key.

It simply meant to show that it's easier to prevent the loss of a virtual key than it is to prevent the loss of the physical key.

> The whole scheme is backwards. I don't want to authenticate myself to the > server; I want to authenticate the server to me.

That's... interesting. My first thought when I read that is that the server is the one with your data and multiple users - so it needs to authenticate you to make sure that you only access your data and don't gain access to other users' data.

A server authenticating itself with you would tell you that you're actually talking to XYZ and not an imposter, but once that authentication took place you'd have access to everything on that server. Including other users' data.

How do you imagine this working?

<comment><USER24> Any authentication is one-to-one. The server authenticates to me, by first knowing who I am (my 'username' not a password), then using a scheme we agreed upon (our shared pair of private/public keys would be fine) to verify _electronically_ that my machine belongs to me i.e. has the right keys.

So we are both authenticated to one another. Except now, we're using sophisticated passwords and Digital Computer Logic to work them out. Instead of my fallible wetware.

</comment></comment></comment></comment></comment></comment></comment></comment><comment><USER26> I was in Chicago with my family recently and purchased a one-day pass for the CTA. When I returned home I found you could create an online account connected to the card, ostensibly to be able to fill it up automatically, but also to track your ridership. I thought it would be interesting to see what kind of data a rider could get from the card.

I spent quite a bit of time trying to create an account only to receive a generic 50x error page each time. I tried disabling adblockers, completely unblocking javascript, disabling extensions, and nothing worked. I tried it in different browsers, and on different computers. I still received the 50x error. Then, on a whim, I tried using a different password. When that worked, I tried changing the password using 1-character-removed tweaks of the original. There was a single character that, it turns out, would cause the application to choke on passwords (I don't remember which character). Presumably it was an escape character or a semicolon.

It's not impossible for account systems to fail in unexpected ways, so there are probably more than a few (poorly-designed) websites which have seemingly arbitrary password requirements which are designed to avoid the kind of error I found above. Still, it would be better to fix the error in the webapp than to try to patch it by limiting the user input - that's an obvious security problem waiting to happen.

<comment><USER27> Suggests to me that form-input is being used unescaped at some point in their software. I wonder what the password "foo'; drop table users;" would do.

</comment></comment><comment><USER28> Totally agree. At least 1 Uppercase means 99% of people just use an uppercase first character for their standard password. 1 number means they append 1,2 or a year and 1 special character means they append an exclamation mark. Entropy added = 0 (if password rules are known to the attacker). I don't get how anyone who cared the slightest about security would think otherwise and enforces these stupid rules.

<comment><USER29> I think a lot of it is driven by management. I've had to talk management out of it before. Then they still "don't get it" and think I'm OK with an "easier to hack" website, holding it against me as a character flaw - as if I'm a bad engineer who doesn't believe in diligence.

Anyway, that's probably where they come from - implemented by more obedient employees.

</comment></comment><comment><USER30> I see this a lot now, with the proliferation of libraries that allow for arbitrary passwords. Possibly some form of systemic trickling down of bad practices into software with horrible consequences

"Your password must contain the seventh circle of hell, and a Taco Emoji"

<comment><USER31> I liked the message saying something like "your password must contain three characters from Game of Thrones" :)

</comment></comment><comment><USER32> Out of all, my bank forces to have my password to have exactly 5 chars. This is madness. Fortunately to actually do anything I need a second authentication factor.

<comment><USER33> My bank does the same, and they limit everyone to 8 characters. It is Schwab.

</comment></comment><comment><USER34> I despise expiring passwords. My university makes me change my password every 6 months, and they keep a list of the last 2 year's worth of passwords so that you can't reuse them. That, and they disallow some characters such as "#" and " " (space). It's annoying.

<comment><USER24> Typically, folks just change the '1' at the end to '2'. And so on. I agree this is another totally pointless password rule that helps nobody.

</comment></comment><comment><USER35> I use pwsafe for iOS (yes trusting some guy built a binary ok) to keep my many passwords for each domain. It will generate (no I don't know the RNG) passwords based on policy settings (length, characters, etc)

And like the author, flat out my number one bugbear is that no matter what random(ish) password I choose before signing up, some idiot will decide that my 12 letter password is too long or my password must have an upper case, or a number of a special character - but never in a consistent manner.

I really want to just move to google authenticator and one factor auth. Just stick a QR code up, let me photo it on screen and I will give you a HMAC code right back.

No passwords. Just illusion

<comment><USER36> And then you lose your phone and...

<comment><USER35> Pretty much the same as everything else that happens when I lose my phone. Total disaster.

</comment></comment></comment><comment><USER37> Often the stupidest limitations in passwords come from the backend system being an ancient banking/insurance/healthcare system running on a mainframe using rules that were prevalent 30-40 years ago. Having worked for a smallish financial/banking company we had an AS/400 with code mostly written in a 4GL tool as the system of record. The database didn't even have unique keys. Everything modern (like web apps) was limited by what this system supported.

<comment><USER38> One that comes to recent memory is Schwab. A few points from this ([http://www.jeremytunnell.com/posts/swab-password-policies- an...](http://www.jeremytunnell.com/posts/swab-password-policies-and-two- factor-authentication-a-comedy-of-errors)) article from December 2014:

> "Schwab.com passwords are limited to eight characters, cannot contain > symbols, and are case insensitive. " > "I now know that on the backend, my > secure 16 digit password got stored in the system as only the first eight > characters." > "So what they do is allow UP TO eight characters to represent > the password, which is stripped from the contents of the password field. > Assuming that the user is activating a token, there will be characters left > over. Instead of using the LAST six characters to check the token code, they > pull the NEXT 6 characters."

Seems like these companies just do not want to put the effort or deal with the customer pain of updating to a newer model.

</comment></comment><comment><USER39> It always seems like banks are way behind the curve on this for whatever reason. My old credit union that I left a couple years back required specifically a 6 character ^[0-9]$ password for complete access to my account. I have no idea why this seemed like a good idea to them. Plus the idiotic security question crap that they usually store in plain text. Just let me store my email and send me an email if I lose access; security questions make you /less/ secure.

</comment><comment><USER40> I've been wondering about password complexity for a while now. If a password is cryptographically hashed, so long as it isn't a dictionary word or date, why on earth do people add in complexity rules that prevent arbitrary character sequences?

What I'm getting at here is that if you had a 8 character password (for the sake of the example), and you can type in lowercase letters, uppercase letters, numbers and punctuation marks the total combinations of values is 95^8.

But then you remove all possible values of just numbers, all values that are just letters, all values that are just special characters. That reduces the set of combinations by (52^8) for alphabetic characters, then reduce further by 10^8 for numeric only values, then 33^8 for special characters. That's quite a few characters removed!

But then you get Apple's recent password requirement of no repeated characters in their iCloud and App Store passwords. My math behind permutations is a bit rusty, but that removes a hell of a lot of passwords as well.

On top of this, if you require at least one special character, alphanumeric character, and at least one number, then that reduces the number of possible passwords even further!

I've never understood how this made passwords safer... What am I missing?

<comment><USER41> They assume people are using short stupid 90's style password rules. Which many unfortunately are...

<comment><USER40> Not following...

<comment><USER41> Stuff like aaa123

<comment><USER40> Well, if you put in xuui-659995!?)) then it won't let you use this either. How does that make any sense?

Edit: In fact, interestingly I used a pseudo-"random" generator made up in my brain. And there you have a big problem - I couldn't help myself but I grouped letters, numbers and special characters.

I have to ask - if most of us do this, as we pick our own passwords, then don't we by our very nature make it easier to crack our own passwords?

</comment></comment></comment></comment></comment><comment><USER42> Abstract: pls use [https://github.com/dropbox/zxcvbn](https://github.com/dropbox/zxcvbn)

<comment><USER43> For most websites, users should just be warned that their password is insecure, not forced to use a stronger one. Your website probably isn't important enough to me to get its own strong password.

</comment></comment><comment><USER10> I would also add to this statement that this shouldn't be user's problem, but service problem. By forcing setting strange passwords services transfer their problem to secure passwords to user's shoulders.

Instead of following shitty password rules in forms, it's better to make it very hard or expensive to brute-force these passwords. So any heuristics to identify ubnormal/dangerous activity and take an action by decreasing attacker chances like rate limiting/captchas and so on.

      * If you see one IP trying to login with incorrect creds with really high rate - then it's probably attack.
      * If you see really lots of IPs trying to crack specific user account at the same time - then it's probably attack.

Instead of that I can see the opposite practice: service set draconian password politics, but just allow requests with incorrect credentials without any limits: "30req/sec? You're welcome, buddy! Need an API maybe?"

I can suspect something like this happened before:

"It looks like a lot of work with rate limiting and all the stuff, let's just force our users to set 10+ character passwords with one+ capital letter, one+ number, one+ special character". Oh, and in these examples there is usually cherry on cake like:

      - Dev1: "Let's not allow 2 same characters or 3 characters of same type"
      - Dev2: "Let's also force our users to change their passwords every 3 months"
      - CEO: "Brilliant ideas! We're secure now!"

These surprises are up to every developer's/another genius infosec imagination :)

So, my conclusion is that best security systems should be almost invisible to normal users and let attackers screaming.

</comment><comment><USER44> Truncation: The browser could have warned users for years already. Pasting something that gets cut off is data loss. Relying on a JavaScript implementation to do this is a hack that only covers some sites (yes, you could have an add-on, but no, that would require JavaScript, and it's still a hack).

Binary Passwords: I could just copy/paste blocks of hexadecimal digits into the password field using a fixed number of digits corresponding with the key size, and they would be interpreted as a stream of unsigned binary octets. The only thing you'd need to know was the length.

A suitable format is using hexadecimal. If you're using a password manager, you already don't care about what the characters are, only that it is secure.

Passwords are also no longer limited to use printable characters.

How would you implement that for all users? Adding a prefix like 0x to enter hex mode?

</comment><comment><USER45> I was disappointed to learn that Android wouldn't accept a password with non- ASCII characters. I wanted to mix accents, punctuation, and symbols that are all easily typed with the soft keyboard, but it was rejected as "too short" because anything that wasn't between 0x20-0x7E was trimmed.

</comment><comment><USER46> Apologies if this is a common held practice which i'm unaware, but why not establish a policy amounting to: "if the system-admin can't somewhat trivially crack your password then i won't bug you about it". That is, allow any password, then if your chosen crack software can't crack that password, leave them alone. If you can crack it then: "Please select another password, as our trivial cracking software discovered it". This would strike me as a less user burdensome method which would be more generally secure than every 13 weeks: "9-12 characters including one of each of the following set"

</comment><comment><USER47> Or browser manufacturers could implement HOBA and we could stop worrying about passwords for the web. Just sayin...

[http://tools.ietf.org/html/rfc7486](http://tools.ietf.org/html/rfc7486)

</comment><comment><USER48> Allowing 100,000 login attempts per second is in itself insecure! Using longer passwords in that case is just treating the symptoms of a bad security strategy.

Good security strategy: Have many layers of security! For example: Limit the login attempts to max ten tries. And another layer, whether you like it or not is to make users not use their "standard" password.

More security layers: hash+random salt, SSL, password timing, logging, 2-way authentication, hiding, white-listing, ... a strong password =)

Please note the difference between cryptography and password authentication though. In cryptography, a longer key is most likely always better.

</comment><comment><USER49> I was looking for a cloud MongoDB provider, so I found MongoLab then I tried to register. It didn't let me because my password didn't have a number in it, do you want to know the password I used? Here's the password:

      U\"&%x#vdE

Their support site has more ridiculous password rules. Like one capitalized letter, one number, one lowercase letter, one non-alphanumeric, etc.

I asked the support and he said this:

      Not all of our users are as savvy with security concerns.

I told them Google allows "aaaaaaab"...

I wish them to get rich with their non-savvy users, best of luck to them.

</comment><comment><USER50> "Through 20 years of effort, we've successfully trained everyone to use passwords that are hard for humans to remember, but easy for computers to guess." Wow.

</comment><comment><USER51> Pick whatever high-entropy passwords you want, and tack on "Aa$1" onto the end of every password to deal with the silly systems.

</comment><comment><USER52> Combination of words are the way to go. Often, when I tell people that, they seem to get stuck probably because that's not what they've been used to doing. I would be inclined to give some kind of hint to the user: "a good password, easy to remember: pick at least 4 random words and separate them by a character of your choice."

</comment><comment><USER53> Password rules are needed, otherwise passwords would be even worse than they are now, with people going back to using '1' or '1234'. They just need to be better designed:

Must have at least 16 chars, OR Must have at least 8 chars with 1 number, 1 letter, 1 symbol, AND Must not have more than 3 repeating characters

<comment><USER24> I think the rest of this thread, _and the OP_ , have been systematically refuting that claim.

</comment></comment><comment><USER54> The worse ones are the constraints to which characters one isn't suppose to use, which on my book only reveals sloppy coding as the developer wasn't able to cope with arbitrary characters.

<comment><USER55> Sometimes it reveals things about the underlying system (e.g., that they are storing passwords on a mainframe, or in a database with a different character set).

</comment></comment><comment><USER56> The iCloud and Microsoft account password rules are bad examples. The first one forces one to use certain characters (uppercases, numbers) amd the second one supports only up to 16 characters.

</comment><comment><USER10> Surprised of 99% agreement here at HN, because usually IT/security people in majority of the cases don't understand me when saying ^^ the same.

</comment><comment><USER57> I also hate companies that won't let you use a space in your password. What kind of crazy stuff is that?

Stuff like "I like spaces" would not work.

<comment><USER58> Maybe it's because spaces are unicode characters and they are worried that their password system isn't unicode compatible.

</comment></comment><comment><USER59> Here's a collection of offenders:

[http://password-shaming.tumblr.com](http://password-shaming.tumblr.com)

</comment><comment><USER60> totally agree. just to make things worse, the password rule on the client side sometimes does not agree with the one on the server side. XD

</comment><comment><USER61> The problem is that those that enforce those rules does not read these articles :( its sad

</comment><comment><USER62> They want me to enter upper case letter. I enter Ö and it doesn't accept it

</comment><comment><USER63> Right on!

</comment><comment><USER64> Argh! The XKCD entropy comic. Back away slowly..

The "4 common words" password meme was around a lot after that comic. However, such passwords are definitely not secure because a good dictionary attack will break it in hours. You do not have as much entropy as you think with them.

The same problem tangentially applies to his suggested password schema to.

The problem is: as soon as you start using human word or pseudo words then your complexity decreases significantly.

With that said he has a point: password complexity schemas as employed by websites are not good enough. It should enforce length and complexity properly (not by saying "you must have these types of characters" but by really analysing the password).

I appreciate the UX problems involved there... but then we could push people towards using password managers and auto-generated passwords.

Summary: use a password manager and auto-generate complex passwords.

<comment><USER65> No. For a dictionary attack, someone would need to know that you are using words. They would need to know if they are separated by spaces, or dots or slashes. The probability of someone knowing your personal scheme is fantastically low, unless they already know one of your other passwords and can guess it. A password like:

"Żółć zżółkła w gąszczu fantazyji!"

Is absolutely impossible to crack in any reasonable amount of time. It's a perfectly valid sentence in Polish, but the attacker would need to know to use a Polish dictionary, and also a one of these words is an old-Polish word which will not appear in any modern dictionaries. Finally, they would need to know you have an exclamation mark at the end.

It's literally impossible for someone to guess all of that, and this password is infinitely easier to remember than !@#%#$@j4jnvdbst$#^@%$@#$#

<comment><USER41> It has literally been done already. Obscure African poems have been guessed.

<comment><USER65> By people who knew to look for obscure african poems? I mean sure, you could guess a Polish sentence if you used a Polish dictionary, but how likely is that?

<comment><USER41> Very, because password crackers are literally including everything they can come over.

<comment><USER65> If they include "literally everything", then such humongous dictionary attack would literally take longer than a simple brute force one.</comment></comment></comment></comment></comment></comment>


Show HN: Basilica – word2vec for anything - <AUTHOR> https://www.basilica.ai/ <comment><USER4> Hey all,

I did a lot of the ML work for this. Let me know if you have any questions.

The title might be a little ambitious since we only have two embeddings right now, but it really is our goal to have embeddings for _everything_. You can see some of our upcoming embeddings at [https://www.basilica.ai/available- embeddings/](https://www.basilica.ai/available-embeddings/).

We basically want to do for these other datatypes what word2vec did for NLP. We want to turn getting good results with images, audio, etc. from a hard research problem into something you can do on your laptop with scikit.

<comment><USER5> What's different between an "embedding" and a projection, which I believe is the more standard term for this kind of transformation?

<comment><USER4> "Embedding" is the term I've heard used for this most often. It's definitely the term that seems to dominate in the literature. (Just to pick a random paper off my reading list: [https://arxiv.org/abs/1709.03856](https://arxiv.org/abs/1709.03856) .)

In my mind "embedding" carries the connotation that you're moving into a smaller space that's easier to work with, and where things which are similar in some way are near each other.

<comment><USER5> The TensorFlow docs agree with you. This field has such jargon proliferation, it's hard to keep up.

[https://developers.google.com/machine-learning/crash- course/...](https://developers.google.com/machine-learning/crash- course/embeddings/video-lecture)

<comment><USER4> Yeah, I agree. The number of new nouns per year is kind of ridiculous.

</comment></comment></comment></comment></comment><comment><USER6> Interesting idea, but seems to very much fall within the category of something you would often want to build in-house. I always imagined the right level of abstraction was closer to spacy's, a framework that lets you easily embed all the things.

If you are interested in how to build and use embeddings for search and classification yourself, I wrote a completely open source tutorial here: [https://blog.insightdatascience.com/the-unreasonable- effecti...](https://blog.insightdatascience.com/the-unreasonable- effectiveness-of-deep-learning-representations-4ce83fc663cf)

</comment><comment><USER7> What is the use case for this? (And this is a general point for AI cloud APIs)

Specifically, I am trying to think of an example where the user cares about a vector representation of something, but doesn't care about how that vector representation was obtained.

I can think of why it would be useful: the ML examples given, or perhaps a compression application.

However, in each of these cases, it would seem that the user has the skill to spin up their own, and a lot of motivation to do so and understand it.

<comment><USER4> Apologies for the long answer, but this touches on a lot of interesting points:

1\. Transfer learning / data volume. If you have a small image dataset, embedding it using an embedding trained on a much larger image dataset is really really helpful. In our tutorial ([https://www.basilica.ai/tutorials/how-to-train-an-image- mode...](https://www.basilica.ai/tutorials/how-to-train-an-image-model/)), we get good results with only 2-3k animal pictures, which is only possible because of the transfer learning aspect of embeddings.

You could do transfer learning yourself, if you have the time and expertise. And for a domain like images, it's really easy to find big public datasets. But long-term we're hoping to have embeddings for a lot of areas where there _aren 't_ good public datasets, and pool data from all our customers to produce a better embedding than any of them could alone.

2\. Ease of Use. You can take a Basilica image embedding, feed it into a linear regression running on your laptop CPU, and get really good results. To get equally good results on your own, you'd need to run tensorflow on GPUs. This is harder than it sounds for a lot of people.

3\. Exploration. Because of the other two points, if you have a thought like "huh, I wonder if including these images would improve our pricing model", you can whip up some code and train a model in a few minutes to check. Maybe if it's a big model you go grab lunch while it trains.

If you're doing everything from scratch in tensorflow, it can take days to try the same thing. This activation energy reduces the amount of experimentation people do. It's bad for the same reasons having a multi-day compile/test loop would be bad.

<comment><USER7> Hi mlucy,

I agree with what you're saying here. I just wonder how it would work in practice.

So imagine I have this monster text or image, and I want to know if it looks like another text or image.

I send each to Basilica, it gives me back two vectors and I compare the vectors.

I use the cosine of the vectors as a similarity score, and lets say it comes out to be 0.6.

However, I think this is too low, and I want to tweak my algorithm.

At this point, doesn't the question of how the vector was generated come to the front. Did you get rid of common words, how did you treat stems, and so on? Or did what biases did you introduce into training?

Furthermore, these questions come up right away, and they seem fundamental to whatever the main practice is.

In other words, can I even experiment or start without knowing how the word2vec works?

<comment><USER4> You're definitely right that you sometimes need to know the exact details of how an embedding is produced, especially if you're doing cutting-edge work. That's one of the things we really need to improve documentation-wise. I'd like to have a page for each embedding that talks about how it's generated, what to watch out for while working with it, etc. etc.

I'm going to narrow in on the question of how to go about tweaking a model that uses an embedding, since I think it's a really interesting topic.

To use your first example, let's say you're doing the image similarity task. You probably wouldn't be computing the cosine distance on the embeddings directly. You'd probably normalize and then do PCA to reduce the number of dimensions to 200 or so.

If you weren't getting good results, you'd have a few options. You could fiddle with the normalization and PCA steps, which can have a big effect. You could also include other handcrafted features alongside the embedding. But let's say you have a fundamental problem, like your similarity score is paying too much attention to the background of your images rather than the foreground.

There are two major approaches to solving that sort of problem with embeddings: preprocessing or postprocessing. You could preprocess the images before embedding them to de-emphasize the backgrounds (e.g. by cropping more tightly to what you care about). You could also postprocess the embeddings. For example, you could label which of your images have similar backgrounds, and instead of naive PCA you could extract components that maximally explain variance while having minimal predictive power for background.

<comment><USER8> I definitely agree you should add more documentation to how the word vector model(s) is generated. Also you may want to have a set of models that the user can choose from. For example, Wikipedia is good for a general language use case. But something more technical, such as finance, SEC filings are a better data source.

</comment></comment></comment></comment></comment><comment><USER9> How do you plan to counter the harmful societal biases that embeddings embody?

See Bolukbasi ([https://arxiv.org/pdf/1607.06520.pdf](https://arxiv.org/pdf/1607.06520.pdf)) and Caliskan ([http://science.sciencemag.org/content/356/6334/183](http://science.sciencemag.org/content/356/6334/183))

While these examples are solely language based, it is easy to imagine the transfer to other domains.

<comment><AUTHOR> Hi. Jorge from Basilica here.

We don't have any concrete plans to tackle this right now but it is something we're definitely mindful of. Thanks for the links! We'll be sure to go through them.

</comment></comment><comment><USER10> Aren't these embeddings task-specific? For example a word2vec embedding is found by letting the embedder participate in a task to predict a word given words around it, on a particular corpus of text.

The embedding of sentences are trained on translation tasks. A embedding that works both for images and sentences is found by training for a picture captioning task.

The point I'm asking about is that there may be many ways to embed a "data type", depending on what you might want to use the embedding for. Someone brought up board game states. You could imagine embedding images of board games directly. That embedding would only contain information about the game state if it was trained for the appropriate task.

<comment><USER4> You can definitely improve performance by choosing an embedding closely related to your task. In the future we're hoping to have more embeddings for specialized tasks.

Kind of surprisingly, though, if you get your embedding by training a deep neural network to do a fairly general task -- like denoising autoencoding, or classification with many classes -- it ends up being useful for a wide variety of other tasks. (You get the embedding out of the neural network by taking the activations of an intermediate layer.)

In some sense you'd expect this, since you'd hope that the intermediate layers of the neural network are learning general features -- if they were learning totally nongeneral features, it would be overfitting -- but I found it surprising when I first learned about it.

</comment></comment><comment><USER11> You quote a target of 200ms per embedding, not sure if it's one type of embedding in particular. I am using Infersent (a sentence embedding from FAIR [https://github.com/facebookresearch/InferSent](https://github.com/facebookresearch/InferSent)) for filtering and they quote a number of 1000/sentences per second on generic GPU. That's 200 times faster than your number, but it is a local API so I am comparing apples to oranges. Yet it's hard to imagine you are spending 1ms embedding and 199 on API overhead. I am sure I have missed a 0 here or there, but I don't see where, other than theirs is a batch number (batch size 128) and maybe yours is a single embedding number. Can you please clarify? Thanks

<comment><USER11> So I am going to answer it myself. On batched data, it's a lot faster than 200ms per embedding and I'd say on a par with Infersent. On the other hand, I couldn't get statistical performance in the same ballpark as Infersent and I had to backtrack. This was training a logistic regression on the embeddings to filter some text streams according to my preferences. If I had, I would have preferred Basilica as Infersent is py2 only, hard to install and distribute and a battery killer on my laptop. Its vectors are also 4x bigger. I experienced some server errors and the team at Basilica was very responsive and fixed it, very pleased with the interaction. It would be important IMHO to publish some benchmark results for these embeddings, as it's usually done in the universal embedding literature, or serve published embeddings with known performance when licensing terms are favorable.

<comment><USER11> Another update, on their v2 sentence embedding basilica is ahead of Infersent for my task. Well done basilica!

</comment></comment></comment><comment><USER12> How much does this depend on the data type? I.e. do you need people to specify: this is an image, this is a resume, this is an English resume, etc. Could you ever get to a point where you can just feed it general data, not knowing more than that it's 1s and 0s?

<comment><USER4> That's a really interesting idea.

I can't really think of a barrier to this. Detecting the file format is straightforward, and generic image/text/etc. embeddings work surprisingly well. (In fact, you can actually get some generalization gains by training subword text embeddings on corpora in multiple languages.)

If we wanted to able to use specific embeddings (e.g. photos vs. line art, English vs. German), we could probably do it by running the data through a generic embedding, and then seeing which cluster of training data it's closest to and running it through that specific embedding.

It would be really important in this case to make sure that all the specific embeddings are embedding into the same space, in case people have a mixed dataset, but that's very doable.

</comment></comment><comment><USER13> Slightly different topic but what are some approaches to categorize webpages. Like I have 1000s of web links I want to organize with tags. Is there software technique to group them by related topics?

<comment><USER14> The task is known as document clustering [https://en.wikipedia.org/wiki/Document_clustering](https://en.wikipedia.org/wiki/Document_clustering) or topic modeling [https://en.wikipedia.org/wiki/Topic_model](https://en.wikipedia.org/wiki/Topic_model)

Generally, you'll want to extract features (e.g. word counts) and then apply a clustering algorithm to group related documents together. The precise details are the subject of thousands of papers, each one doing things slightly differently.

</comment></comment><comment><USER15> Is this actually 'for anything'? I see references to sentences and images. If I, for example, wanted to compare audio samples, how would it work?

<comment><USER4> "Word2vec for anything" is where we want to get to. Right now we only support images and text, but you can see the other data types on our roadmap at [https://www.basilica.ai/available- embeddings/](https://www.basilica.ai/available-embeddings/) .

</comment></comment><comment><USER16> Hi Lucy. Looks great. Do you have any production use cases you can tell us about? Are you a YC company?

<comment><USER4> Thanks!

No production use cases yet. This is the first usable release, and it's the bare minimum we felt we could build before showing it to people.

> Are you a YC company?

We have a YC interview on Friday, so hopefully in a few days I'll be able to say yes.

</comment></comment><comment><USER17> So the actual code is closed-source?

<comment><AUTHOR> Hi, Jorge from basilica here.

Yes. We intend to run this as a cloud service API for now.

</comment></comment><comment><USER18> Do you think board game states might be a good target later?

<comment><USER4> Sort of depends on how late "later" is.

In the very-long term, I want us to literally have embeddings for everything people want to embed, which will probably include the states of popular boardgames.

I'm not sure how we'll get there. Maybe we'll have community embeddings, or an embedding marketplace, or we'll abstract away the process of creating an embedding so well that we can create simple embeddings just by pointing our code at a dataset. But I'd like to get there eventually.

In the less-very-long term, we're still focusing on embeddings that are either very general and useful across a lot of domains (e.g. images, text), or embeddings that have clear and immediate business value (e.g. resumes), since running GPUs is expensive.

</comment></comment><comment><USER19> How are you embedding images?

<comment><USER4> We're feeding them through a deep neural network and using the activations of an intermediate layer as an embedding.

You can read more about this technique in [https://arxiv.org/abs/1403.6382](https://arxiv.org/abs/1403.6382) if you're interested.

</comment></comment><comment><USER20> >Job Candidate Clustering

>Basilica lets you easily cluster job candidates by the text of their resumes. A number of additional features for this category are on our roadmap, including a source code embedding that will let you cluster candidates by what kind of code they write.

Wonderful! We were in dire need for yet another black-box criteria based on which employers can reject candidates.

"We're sorry to inform you that we choose not to go on with your application. You see, for this position we're looking for someone with a different _embedding_."

<comment><USER21> word2vec:

        king
      - <USER3>
      + woman
      </comment></comment><comment>--
      = queen

Basilica:

        resumes of candidates
      - resumes of employees you fired
      + resumes of employees you promoted
      </comment><comment></comment><comment></comment><comment></comment><comment></comment><comment></comment><comment>---
      = resumes of candidates you should hire

<comment><USER22> word2vec[0]:

          computer programmer
        - <USER3>
        + woman
        </comment></comment><comment></comment><comment></comment><comment>---
        = homemaker

Basilica?

[0] - [https://arxiv.org/pdf/1607.06520.pdf](https://arxiv.org/pdf/1607.06520.pdf)

<comment><USER23> One thing I've been tempted to research but never had time for myself: can one use that aspect of wording embeddings to automatically detect and quantify prejudice?

For example, if you trained only on the corpus of circia 1950 newspapers, would «"man" - "homosexual" ~= "pervert"» or something similar? I remember from my teenage years (as late as the 90s!) that some UK politicians spoke as if they thought like that.

I also wonder what biases it could reveal in me which I am currently unaware of… and how hard it may be to accept the error exists or to improve myself once I do. There's no way I'm flawless, after all.

<comment><USER24> > For example, if you trained only on the corpus of circia 1950 newspapers, > would «"man" - "homosexual" ~= "pervert"» or something similar?

If it did, what conclusion would you be able to draw?

As far as I know, there's no theoretical justification for thinking that word vectors are guaranteed to capture meaningful semantic content. Empirically, sometimes they do; other times, the relationships are noise or garbage.

I am wholeheartedly in favor of trying to examine one's own biases, but you shouldn't trust an ad-hoc algorithm to be the arbiter of what those biases are.

<comment><USER25> I think there's a further problem that there's never been a shortage of evidence, about things like this. The point is, prejudice and discrimination are not evidence-based in the first place. People who support existing unjust structures are generally strongly motivated to turn a blind eye. Even people who don't support them are - it's simply far easier and more socially advantageous to stop worrying and love the bomb.

</comment></comment></comment></comment><comment><USER26> Am I really missing something here or this thing is a complete nonsense with no actual use cases what's so ever in practice?

There are a number of off-the-shelf models that would give you image/sentence embedding easily. Anyone with sufficient understanding of embedding/word2vec would have no trouble train an embedding that is catered to the specific application, with much better quality.

For NLP applications, the corpus quality dictates the quality of embedding if you use simple W2V. Word2Vec trained on Google News corpus is not gonna be useful for chatbot, for instance. Different models also give different quality of embedding. As an example, if you use Google BERT (bi-directional LSTM) then you would get world-class performance in many NLP applications.

The embedding is so model/application specific that I don't see how could a generic embedding would be useful in serious applications. Training a model these days is so easy to do. Calling TensorFlow API is probably easier then calling Basilica API 99% of the case.

I'd be curious if the embedding is "aligned", in the sense that an embedding of the word "cat" is close to the embedding of a picture of cat. I think that would be interesting and useful. I don't see how Basilica solve that problem by taking the top layers off ResNet though.

I appreciate the developer API etc, but as an ML practitioner this feels like a troll.

<comment><USER27> > Calling TensorFlow API is probably easier then calling Basilica API 99% of > the case.

Maybe, but training/curating data appropriate for your application isn't. It's not in that state right now but I think this service could save you some time if they had a domain-relevant embedding ready to roll for your application and it performed decently well -- that would save you a lot of time gathering training data and help you focus on the "business logic" ML needs that accept the embeddings as input.

That said, they'd need to be more performant than, say, GloVe 2B, which I can get for free off of torchtext, meaning they have to do the domain-specific heavy-lifting.</comment></comment>


COS has been released for the commodore 64 - <AUTHOR> http://64jim64.blogspot.com/2015/09/cos-has-been-released-for-commodore-64.html <comment><USER2> This is the same story as [https://news.ycombinator.com/item?id=10452550](https://news.ycombinator.com/item?id=10452550). We almost changed its URL to point to the blog post earlier. Perhaps we'll do so now.</comment>


Pics of Web Company HQs in California - Digg, Facebook, OpenDNS.. - <AUTHOR> http://paulstamatiou.com/2007/07/21/california-hq-pics/

<comment><USER2> Wow, Aeron chairs at a conference table. I usually only see them in posh marketing firms where executives sit in conferences all day.

<comment><USER3> One of my mentors told me once that once he sees Aeron chairs at a startup, he not only doesn't invest, he cuts the meeting short and leaves running away from the company.

</comment></comment><comment><USER4> From the corner office straight to the chiropractor (classy verizon T, by the way):

[http://paulstamatiou.com/wp- content/uploads/2007/07/opendns_...](http://paulstamatiou.com/wp- content/uploads/2007/07/opendns_hq_2.jpg)

I guess that might be a holding pen for reporters, and not an office, though. Then check out this stupid car:

[http://paulstamatiou.com/wp- content/uploads/2007/07/sf_carre...](http://paulstamatiou.com/wp- content/uploads/2007/07/sf_carrera_gt.jpg)

How stupid is that? The King Tiger tank (69 Tons) produced 690 horsepower. That car (1.5 tons) produces 612 horsepower. What a waste. And your typical diesel locomotive might product 3000+ horsepower (about 5 times as much) while hauling millions of pounds. And it doesn't even compare to a real car for looks:

[http://firsttraderegistry.com/forsale/9_06/65mustang2/1965%2...](http://firsttraderegistry.com/forsale/9_06/65mustang2/1965%20mustang%20critean%20022.jpg)

We're so vain.

</comment><comment><USER5> "A chain is still a chain, even if made of gold" :)

</comment><comment><USER6> This is OK but I'd be more interested in seeing where these companies started out -- in a dorm room, around a kitchen table... any place else?

</comment><comment><USER7> Most of these places look like they have a great work environment. Getting a tour of these companies must have been an awesome experience.

</comment><comment><USER8> We want all our companies to have offices like this.

<comment><USER4> Why you so dumb, huh?</comment></comment>


Thrusters powered by ionic wind may be an efficient alternative - <AUTHOR> http://web.mit.edu/newsoffice/2013/ionic-thrusters-0403.html <comment><USER2> > Give off no heat

I'd love if someone had a link for this.. I thought termodynamics ruled out running tons of current through something about any heat?</comment>


Audience Interaction System running on iPod  - <AUTHOR> http://demetree.yourqforme.com

<comment><AUTHOR> Go ahead and try it. Ask a question.</comment>


MythBuster's Adam Savage complains about $11K data roamining bill on Twitter - <AUTHOR> http://www.techcrunch.com/2009/06/26/mythbuster-adam-savage-leads-twitter-revolt-against-att/

<comment><USER2> It's worth checking out his full twitter page, <https://twitter.com/donttrythis>

In addition to being pissed at AT&T, this is what he says about the RIAA/MPAA:

"I believe the RIAA to be a bunch of venal idiots. I don't think much of the MPAA either for that matter."

He seems to be a genuinely cool guy, with a well calibrated BS detector.

<comment><USER3> "Today the tweeps became twoops. Just got off the phone with AT&T and they've taken care of everything to my great satisfaction.#twitterrules"

Seems like it's all settled, for him at least. Wonder about everyone else, though.

<comment><USER4> What's impressive is that this whole episode took only 5 hours from start to finish. Sometimes, it seems like it takes that long just to get talking to a real person.

<comment><USER5> It has to help that he's semi-famous. I remember the exact same thing happening to a Verizon customer a few years back, and it certainly took longer than 5 hours for him to get his money back.

[http://verizonmath.blogspot.com/2006/12/verizon-doesnt- know-...](http://verizonmath.blogspot.com/2006/12/verizon-doesnt-know-dollars- from-cents.html) (the audio link has gone dead)

These companies need to learn math.

</comment></comment></comment></comment><comment><USER6> There was a big stink a few years about about the same 0.015 "cents" with Verizon. The guy recorded his phone call, and wrote a series of blog posts about it. [http://verizonmath.blogspot.com/2006/12/verizon-doesnt- know-...](http://verizonmath.blogspot.com/2006/12/verizon-doesnt-know-dollars- from-cents.html)

</comment><comment><USER7> We have had this problem within Europe for a long time, but the European Union have acctually started taking care of the problem. Now there is a maximum fee on sms and voice calls when roaming within the union. About $0.50 to call another country and $0.20 to recive a call. From march 2010 the operator must provide a limiting function on data roaming, which cuts you off when you reach a certain amount.

</comment><comment><USER8> I just realized something odd about this oft-repeated billing scenario: the very few people who could realistically pay $11k overage bills also always have the means to fight it.

Does _anyone_ ever pay these bills?

<comment><USER9> Everyone has the means to fight this.

Let's see, I was in 7th grade or so (6 years ago?) and Comcast came to install their service as we followed their wish to ditch the dish. The man that came did some damage to the house before he finished. Dad refused to pay, he refused to finish the job. We had no TV, a large hole in a ceiling, and more...

Dad was pissed but apparently I was even more angry after I was told someone would come by only in a week or so. So what did I do?

I went on Google and found the email of the head of the south east region of Comcast. I sent him a long email expressing my unhappiness and I told him I was going to post my story on my blog. I didn't think it would lead to anything, I did this without my family's knowledge.

Next morning I awoke as my dad shook me asking me what I did. We had 3-5 Comcast vans parked outside our house in addition to an insurance repair firm fixing the hole. I had a blog for the purpose of learning what a blog was, messing around with HTML/CSS, no viewers. I didn't say that of course. The threat was enough.

With twitter/facebook, it's even easier. Managers know that company reputation is in the grips of one unhappy customer.

</comment></comment><comment><USER10> God this is a perfect post for techcrunch. complaining and Twitter.

</comment><comment><USER11> In other news, now we know what 50K followers on Twitter are worth (or more accurately, we have a lower bound for it's worth).</comment>


The AMD Ryzen Threadripper 3960X and 3970X Review: 24 and 32 cores on 7nm - <AUTHOR> https://www.anandtech.com/show/15044/the-amd-ryzen-threadripper-3960x-and-3970x-review-24-and-32-cores-on-7nm <comment><USER2> [https://news.ycombinator.com/item?id=21628149](https://news.ycombinator.com/item?id=21628149)</comment>


Bitcoin mining and energy consumption - <AUTHOR> https://blog.bitcoin.org.hk/bitcoin-mining-and-energy-consumption-4526d4b56186 <comment><USER3> One of the big selling points of Bitcoin is that it's meant to be decentralised but it doesn't look like this plays out. Proof of work seems to inevitably lead to all the power going to parties that can afford specialised hardware who are in countries with low electricity costs.

Running a Bitcoin miner on commodity hardware now is pointless which seems to go against the spirit of Bitcoin when it started. Was this predicted at the inception of Bitcoin? It's interesting how economics impacts the security of the protocol like this.

Proof of stake is meant to consume less resources but then the power is then handed to people with the most money? Coming up with a way to have a decentralised currency where everyone involved gets a fair say without consuming too many resources is a super interesting problem.

<comment><USER4> _> Was this predicted at the inception of Bitcoin?_

It was not predicted in the original Bitcoin whitepaper. Yes, it discussed the theoretical feasibility of a 50+% attack to double-spend but it didn't explicitly predict a "consolidation" to specialized hardware miners that leaves out the miners at home using regular computers.

 _> It's interesting how economics impacts the security of the protocol like this._

Every decentralized protocol suffers from _unintended centralization_ caused by economics. The same thing happened to other protocols like NNTP (Usenet), SMTP, HTTP+HTML, and Git.

The underlying issue is that technical protocols still have to be realized on _real hardware_ like cpus + harddrives + network bandwidth and consume _real costs_ like human labor. You can decentralize a protocol _specification_ but you _can 't decentralize the amount of money_ different entities are willing to spend on that protocol. Each protocol whether NNTP/SMTP/Git/bitcoin does not come with a $1 million grant for homeowners to spend at Newegg to keep protocols decentralized.

That's why protocols consolidate towards big players in a power law distribution.

<comment><USER3> > It was not predicted in the original Bitcoin whitepaper.

> Every decentralized protocol suffers from unintended centralization caused > by economics.

> That's why protocols consolidate towards big players in a power law > distribution.

Hmm, given the amount of foresight in the original whitepaper I'm surprised the current situation wasn't predicted especially if it's common with other protocols that are intended to be decentralized.

<comment><USER5> It could have been foreseen and omitted. We still don't know who Satoshi is or what their motivations are.

</comment></comment></comment></comment><comment><USER6> The electricity consumption is a red flag, but perhaps it could be cheaper than the cost to build and maintain Guarda/Loomis trucks that carry physical currency, energy to melt metal to mint coins, and all the costs of building brick&mortar banks (even then, bankers provide auxiliary services like loans and advisement).

We are all spending time talking about it as a collective effort to determine whether or not it should be adopted in order to optimize the transfer of value. Ultimately it could flush out the middlemen who just seek rent by being a financial gatekeeper. Bitcoin isn't entirely different from that.

Many folks see it as a slot-machine. Buy, sit on it, cash out back to federal reserve notes. Profit. The question here is - what will those proceeds be spent on? If people are still conditioned from an early age to seek out excessive hedonistic experiences, what are we gaining by implementing bitcoin?

Part of me sees bitcoin explanations as a Rorschach test - you can observe some character traits of the person through their explanation or prediction of bitcoin.

I am wary of the entire thing because we are starting to spend more time talking about money. Or making money on money. It just has nothing to do with what we actually DO with our time/resources. But that is content for a different thread.

I am not a financial expert. Just working-class trying to invest surplus cash. I want to use simple terms and concepts and condense the complex discussion.

<comment><USER7> These numbers are far beyond the energy required for trucks hauling cash. And they dont fully account for the truck-type energy to build and manage the bcoin mining industry.

<comment><USER8> We're starting to stop using cash anyway though right? Regardless of blockchain.

</comment></comment></comment><comment><USER9> Bitcoin's electricity consumption is unfortunate, it's difficult to assess whether it is a waste or not. The current position blockchain tech can be compared to the state of networked computers in the 70s and early 80s. Not a huge amount of value delivered at the time, yet very few could have foreseen the impact of the internet and personal computing on the world.

If blockchains and distributed computing does present a new leap forward in how we communicate, this electricity will not be wasted.

<comment><USER10> Is it a waste to burn CPU cycles, to have a chunk of entropy to wave around so that you can possibly buy shit?

Yes.

<comment><USER11> It's not so different to burning diesel to dig up rocks so you have a chunk of metal to wave around.

Which isn't, you'll notice, necessarily a counter argument to yours.

<comment><USER12> I would actually argue it is very fundamentally different. Once you've dug up your chucks of metal, trading them with others requires very little extra energy to be expended (Especially if we're going to go all the way to talking about actual cash, which is easy to split-up into smaller denominations).

Vs. Bitcoin, which will still require miners to expend the same amount of CPU cycles to keep the network working so you can transfer your Bitcoins to others, even after all of the Bitcoins are mined and the blocks provide no new Bitcoins into the market.

<comment><USER11> It'd be interesting to see a table of 5 year energy cost from inception of a bar of gold vs. equivalent value of Bitcoin, at various #s of transactions.

</comment></comment></comment></comment></comment><comment><USER13> From a quick research online, it seems that most mining pools are located in China [1]

Roughly 2/3rd of electricity in China is produced from burning coal or gas, which, upon combustion, emit their own weight in CO2, basically. [2]

From this two facts, one can conclude that Bitcoin has a pretty huge carbon footprint overall. This is not the impression that I got from reading that article.

[1] [https://bit-media.org/bitcoin/where-in-the-world-are- bitcoin...](https://bit-media.org/bitcoin/where-in-the-world-are-bitcoin- mines-located/)

[2] [https://en.wikipedia.org/wiki/Electricity_sector_in_China](https://en.wikipedia.org/wiki/Electricity_sector_in_China)

<comment><USER14> If I remember correctly, mining farms (and data centers in general) are usually colocated to an energy source, i.e. a plant.

Some of those plants are not fully stretched, because there's almost no civilization nearby, so delivering to the next city comes with a huge loss in the energy network.

So miners can get cheap energy from those plants - energy that would otherwise be unused or lost anyway.

Not sure how much this saves in the end, but this shows that you can't simply convert energy to CO2 emission without considering which types of plants are used and where these are located.

<comment><USER15> That's a very convenient myth.

It misses the fact that China didn't build power plants out where it's too far away from civilisation to be useful.

Transmission losses are also nowhere near the suggested values. 1.5% per 200 miles is today's standard. Meaning you could cross _all of China_ with a loss of:

        east-west axis: 2200 miles
        0.985^(2200 miles / 200miles) = 0.846 = 15.4% loss

(and that's from the farthest desert to the east coast. Go south instead, or note that there actually are a billion people living in inland China, and actual losses are <5%)

<comment><USER14> Thanks for making this clear.

</comment></comment></comment></comment><comment><USER16> "Also, one can argue that Bitcoin actually saves energy. The world's financial system requires many resources beyond the electricity to run servers."

What a bunch of crap. Bitcoin also needs servers (i.e exchanges, uses computers, routers, switches etc. )

<comment><USER17> The mental gymnastics crypto cultists perform to justify their chosen greed outlets become more impressive every day.

</comment></comment><comment><USER18> Makes a reasonable effort to calculate bitcoin energy use (and it's good to share reasoning, not just headline totals!), but it falls short with the whataboutism of, for instance, shipping gold by aircraft.

Assuming all flights are independently profitable, the cheapest ticket I've seen from London to San Francisco is £261 return, which sets an upper limit on fossil fuel emissions of about 6 barrels of oil equivalent. Me plus luggage is about 100kg, which at current prices is worth £3.292.122,21

</comment><comment><USER19> Is this not the price paid for the trust, security, and utility that bitcoin brings? I don't hear alternatives being suggested.

<comment><USER15> How about not bitcoin, where a transaction doesn't consume 9 day's worth of an average family's energy budget?

Utility: a transaction costs $10, right? It's useless.

Security: Have you followed this? The actual recommendation is to print your private keys and stick them in a vault.

Trust: It's funny that, fundamentally, people started doubting the Federal Reserve when the gold standard was abolished, because suddenly "value of money is just fiction". Yet here they are, imbuing some far more ephemeral nothingness with the same sort of faith-based value.

<comment><USER20> People doubted the Federal Reserve _long_ before 1971...

<comment><USER15> ....and Bitcoin can't even dream of the trust that comes with having been doubted _long before 1971_.

</comment></comment></comment></comment><comment><USER21> Can someone explain something to me:

So people mine bitcoin, which is really just validating the transactions on the network, and they get paid in bitcoin for doing so. However, they will receive fewer and fewer bitcoin over time because of the fixed supply. At some point doesn't mining become unprofitable, causing the network to crash. And if the price drops doesn't this exacerbate this problem? Would love some clarity here.

<comment><USER22> > However, they will receive fewer and fewer bitcoin over time because of the > fixed supply. At some point doesn't mining become unprofitable, causing the > network to crash.

This would actually cause the price of Bitcoin to increase, so that it keeps up with the cost of mining it. The miners won't sell their Bitcoin for less than the cost of electricity.

At some point, large mining operations will run their own power stations, and they'll just run on solar / hydro / geothermal power. They might even sell any surplus energy. Other miners might start paying for their electricity with Bitcoin, so the price of electricity might become directly related to the cost of mining Bitcoin.

</comment></comment><comment><USER23> > It is also easy to make this number look very small:

>The energy that Bitcoin consumes in a year would only last the U.S. for 19 hours.

That doesn't sounds small to me, given the relative utility of 320 million people against that of Bitcoin.

</comment><comment><USER24> Why is energy consumption itself become a bad thing? I get greenhouse gas emissions being bad, but why make energy consumption itself a negative?

Not to mention that "consuming" energy is somewhat a fiction: it's more so changing the entropy of a system. True consumption of energy, at the extreme, would imply sending energy into deep space (any direction other than the sun really) where it can never be recovered.

Is the problem that such entropy changes usually (always?) result in at least some heat loss? And the the planet is warming, so heat itself is seen as a negative?

To make my point a different way, if a mining rig were 100% solar and 100% efficient (both fantasy, of course), then it could consume unlimited solar energy with no ill effects. Thus basing our judgment on energy consumption instead of waste heat and energy source issues seems misguided.

Is it just too complicating to discuss this in terms of CO2 emissions and waste heat, instead of Watts alone?

<comment><USER25> Every form of electricity production consumes an array of limited natural resources. Wind and solar power don't have no direct CO2 emissions, but minerals need to be mined, plenty of clean water is consumed during manufacturing, and wildlife habitats are impacted by land-use change (wind and solar plants have a comparatively larger land footprint than fossil fuel plants).

Fun Fact: The article says Bitcoin mining consumes ~1.1 GW of power. There is currently a massive political struggle over the construction of "Site C", a 1.1 GW Hydroelectric dam in British Columbia, Canada. Construction on that dam began last year, but public opposition caused a new provincial government to review the dam construction. The dam will flood 93 square kilometers of land but produce 5.1 GWh of low-carbon electricity annually, about 53% of the articles assumed Bitcoin mining consumption of 9.6 GWh (which is = 1.1 GW x 8760 hours/year). Whether construction of the dam will be cancelled or resumed, the decision will be controversial! (The decision will be announced before the end of the year).

In summary, bitcoin mining from renewable energy sources is desirable, but bitcoin mining (and all other forms of computing) will always have some negative impact. That is something we need to be aware of.

</comment></comment><comment><USER26> Slightly off topic but would it be an advantage to use a low energy miner such as a Raspberry Pi as a bitcoin miner, or would that just be a waste of time and yield fractions of a cent / a cent per day? Admittedly little knowledge on bitcoin mining

<comment><USER27> Bitcoin is usually mined through mid-to-high end graphics cards. People running a pool of 6-7 of these cards per machine are seeing diminishing returns, so using a Pi wouldn't cover the electricity needed to keep it running.

<comment><USER28> Bitcoin is mined using ASICs, and not GPUs.

But, it's true that mining using a raspberry pi would not be very useful.

</comment></comment></comment><comment><USER29> _Its base reward (currently at 12.5 Bitcoin per block) will half every four years, until it reaches zero._

That's not how halving works.

<comment><USER30> BTC is discrete, to 8dp, rather than being represented by rational numbers, so eventually it will reach zero from rounding.

</comment></comment><comment><USER31> When a handful of people control 40% of your market: it's not decentralized.

When you consume more energy than your house does over 1 week to make a single transaction: it's not efficient.

When it takes hours or days to settle: it's not fast.

So what is bitcoin, anyway? From what I can tell a way for people to get rich.

</comment><comment><USER32> > It is also easy to make this number look very small:

Most of these comparisons sounded pretty identical to the ones from the "If I had the intention to lobby for a ban of Bitcoin mining, I would use references like the one below" list to me.

</comment><comment><USER33> digiconomist has a more "realistic" estimate by including real world factors like cooling costs, the operation of older, but still profitable hardware etc. into their balance sheet. Their number is 32 TWh over the year (vs 9,6 TWh in the article above)

[https://digiconomist.net/bitcoin-energy- consumption#assumpti...](https://digiconomist.net/bitcoin-energy- consumption#assumptions)

Also, their yearly estimate was only 26 TWh a month ago. If we project this into the next year, then we'll end with 130 TWh at the end of 2018 - <USER2> roughly 40% of what the UK consumes.

</comment><comment><USER34> Take your best guess when Bitcoin will fall of the cliff: [https://twitter.com/BitcoinPlunge](https://twitter.com/BitcoinPlunge)

</comment><comment><USER35> Can also utilize the waste heat...by selling a miner as a "heater that makes money".

</comment><comment><USER36> Looks like the HN domain name extraction rules need an update for .hk

<comment><USER37> The current public suffix list ([https://publicsuffix.org/](https://publicsuffix.org/)) already has org.hk as a suffix, so HN might be using an outdated version of the list.

</comment></comment><comment><USER38> "Stellar maintenance entities, are those pesky Star-beavers ruining your convective zone? Do their hydrogen dams constantly clog up radiation from your photosphere? Never fear! With new extra-strength Dam-begone, your star will shine as brightly as 4 galactic years ago! Stay radiant! Stay prominent! Don't wait. Get Dam-begone this rotation! Operators are standing by.". - (I think you might have meant "hydroelectric generator dams")

</comment><comment><USER39> Another aspect that is frequently overlooked is that keeping bitcoin requires no energy at all (literally you can write the key on a piece of paper), and a transfer (not considering the mining aspect) requires very little as well (it can be an SMS message).

Contrast that with the all the office buildings (e.g. Manhattan) and computer systems required by banks, ongoing. The banking industry has google-size datacenters as well, they just don't like to talk about it. As well as the energy to print the paper/plastic cash and all the statements, etc. It is very difficult to estimate the true energy consumed by fiat, while bitcoin energy is in the hashrate, and that's pretty much it.

<comment><USER16> You don't need banks to store cash. Bitcoin without electricity is useless, cash on the other hand is not.

<comment><USER39> But you _do_ need banks to store cash. And the bank better have a secure vault and security guards and cameras and all that.

An elliptic curve key, which is what possession of bitcoin is, is just a large number. To send a bitcoin you need to create a "transaction", and in fact the only aspect of generating a bitcoin transaction that requires energy is the SHA calculation, not something you can do in pencil, the EC math is actually simple enough that a computer is not required.

Also your argument that cash does not require energy does not apply to the cashless society we have become today, when was the last time you actually used cash? Credit cards are useless without electricity.

<comment><USER16> I have cash in my pocket. I'm not a bank. Therefore you don't need bank to store cash. Simple? And I use cash on a daily basis.. We're far from being the cashless society you claim we are.

<comment><USER39> But you can only have so much cash in your pocket. The bitcoin private key is 32 bytes always, regardless of the amount.

For example here is the famous 37 BTC ($500,000+) donation to Andreas Antonopoulos yesterday, this couldn't be done with cash, at least not as seamlessly, inexpensively and quickly:

[https://blockchain.info/tx/e68ebe16e6aaa08e3f9d271ba78aaeb4c...](https://blockchain.info/tx/e68ebe16e6aaa08e3f9d271ba78aaeb4ce7db01b0dc0f3b4d5eb29a14dbddf69)

</comment></comment></comment></comment></comment><comment><USER40> I don't get why journalists are not writing about the criminal side of Bitcoins.

Lots of "users" ( not investors ) interested in Bitcoins are actually .onion sites or other shady / crime organisations.

They actually don't care for human life and I doubt they will care for electricity consumption.

<comment><USER41> Honestly Bitcoin is probably an improvement for law enforcement. Cash is more anonymous. Bitcoin is completely public and traceable while in the network. And the moment you pull it out of the network you are connected to a real world entity.</comment></comment>


The Immigration-Obsessed, Polarized, Garbage-Fire Election of 1800 - <AUTHOR> https://longreads.com/2017/03/28/the-immigration-obsessed-polarized-garbage-fire-election-of-1800/ <comment><USER2> Sorry to be a lazy HN member but that article is HUGE! Does anyone have a TLDR / are there any insights of note that are worth discussing / learning? No problem if no one can be bothered to answer!

<comment><USER3> Well, it is longreads.com.

</comment></comment><comment><USER4> I find this subtitle, trying to draw the parallel with the current election, quite distasteful: "A madman versus a crook? Unexpected twists? Fake news? Welcome to the election of 1800."

These were the very people that fought for our freedoms and independence. Neither party in the current election has even an iota of such an accomplishment. Even the purported civil nature of private communications between the two is at a level of cordiality not seen these days.

<comment><USER5> It sounds like you're making the mistake of looking at the 1800 election through our modern lens. At the time, there was no promise that the U.S. would succeed to the degree it has today. The "founding fathers" were not deified quite to the degree we hold them in our culture (assuming you're in the US).

As such, the subtitle isn't a literally calling Jefferson and Adams a madman and a crook. If that was the case, then yes I agree it would be distasteful and a difficult argument. Of course there are major differences. It's pointing out that someone at the turn of that century might have experienced something similar to what we saw this past year and that maybe we could learn something from knowing that.

<comment><USER4> Their contemporaries were certainly able to call them such and worse. However, I do not see that we can compare these figures to Clinton and Trump. Neither has the accomplishments or the potential for accomplishments that could compare.

<comment><USER5> Yeah, it sounds like we're talking past eachother a bit then. I agree that you can't compare the four figures. That is not what the author is attempting to do, however.

The argument isn't that IF there are similarities between political rhetoric now and in 1800 THEN you may make a comparison between historical figures. It's that IF there are similarities between political rhetoric now and in 1800 THEN similar political problems may have existed in the year 1800 that we are facing today. The author is highlighting his premise, not making a character comparison.

<comment><USER4> Not sure how that precludes me from finding the comparison to be offensive. In the end, that is a personal view and an emotional reaction at that.

<comment><USER5> I agree. It is an offensive comparison, and being offended is a personal view.

Misunderstanding what the author wrote is not a personal view or opinion, it's just ignoring the facts. Ironically, that "facts as a personal view" argument was heavily used by both candidates in the 2017 election which you have decried.

That is what I was addressing with my previous comments. Sorry if you thought I was challenging your personal/political/emotional views in some way. I think you are correct, you're just using poor argumentation.

</comment></comment></comment></comment></comment></comment><comment><USER6> ...Of course a revolutionary war had just been won, and civil war would later be fought, so maybe this isn't terribly comforting.

<comment><USER7> The time of the incident was a generation on from the American revolution and the war that was brewing over the issues of citizenship and British naval practice was the War of 1812. The US Civil War was predicated on internal 'international relations' between states and the citizenship of native born persons and the conundrum of the 3/5ths clause in the US constitution and the political power it provided agricultural interests.

<comment><USER6> I'm not claiming a causal relationship, I'm pointing out just how fresh and unstable the country was, by bracketing the period in question.

</comment></comment></comment><comment><USER8> What is interesting is that a bit more than a decade later, the US entered the "Era of Good Feelings" where bitter partisanship went down and people worked mostly together for the common good, at least on surface. Let's just hope it happens again.

<comment><USER9> and then we killed a whole pile of cherokee. good times.</comment></comment>


Ask HN: What are you reading? - <AUTHOR>

This is the 7th installment of the distributed thread that wont die until i do!<p>Previous: https://news.ycombinator.com/item?id=9636361 https://news.ycombinator.com/item?id=9543693 https://news.ycombinator.com/item?id=9443897 https://news.ycombinator.com/item?id=9394397 https://news.ycombinator.com/item?id=9342886 https://news.ycombinator.com/item?id=8918181 <comment><USER2> Docker Up & Running

[http://amzn.to/1CSEi2a](http://amzn.to/1CSEi2a)

</comment><comment><USER3> The Fall of Hyperion by Dan Simmons (really good SF)

</comment><comment><USER4> Coding Freedom by E. Gabriella Coleman

</comment><comment><USER5> The Quants by Scott Patterson

</comment><comment><AUTHOR> Team of Rivals by Doris Kearns Goodwin.</comment>


Three MacBook mistakes: Will Apple correct course? - <AUTHOR> https://www.macworld.com/article/3234865/macs/three-macbook-mistakes.html <comment><USER2> Nope. It is too late. It is finally going to be the era of the Linux Desktop :).</comment>


Proposal - A Mountain Range for Texas - <AUTHOR> http://a1.blogspot.com/2012/07/proposal-mountain-range-for-texas.html

<comment><USER2> This is one of the most ridiculous things I've ever read. The amount of energy and engineering involved would be ridiculous. If you want mountains then move to a place where there are mountains. I moved to colorado from florida for just that reason. You need to learn to adapt to the world instead of expecting the world to adapt to your expectations.

<comment><USER3> When you say "the world," do you mean the "laws of physics," or "the way things happen to be right now"?

</comment></comment><comment><USER4> Related: Dutch mountains (<http://www.rnw.nl/english/article/brainstorm-dutch- mountains>)</comment>


Bitcoin Industry VCs Shift Focus to Non-Financial Apps and Ethereum Startups - <AUTHOR> http://www.nasdaq.com/article/bitcoin-industry-venture-capitalists-shift-focus-to-non-financial-applications-and-ethereum-startups-cm593925 <comment><USER2> With all the success they've had in bitcoin (or lack thereof) you expect me to believe that jumping to ethereum is going to bring profit? Isn't the whole goal of ethereum to displace companies? I have yet to figure out just what problem ethereum even solves

<comment><USER3> Many fans of Bitcoin anticipated decentralized organizations, decentralized identity, etc... in a way comparable to the way Bitcoin is a decentralized currency. Unfortunately Bitcoin and other blockchain platforms do not provide easy development tools to build those other decentralized organizations. Yes, similar things can be built on Bitcoin as on Ethereum; it's just not as easy.

> I have yet to figure out just what problem ethereum even solves

Ethereum makes the blockchain easily programmable.

<comment><USER4> In exchange it makes computation incredibly expensive not only because the Etherum VM is grossly inefficient but also because every node has to compute the same computations to verify the results (with bitcoin proof is easy but finding a block is hard).

This means the processing power of the etherum blockchain with one full node is the same as with a million nodes. We are jumping back to the nineties from a performance perspective despite potentially spending millions of dollars.

I wouldn't be surprised if etherum could be DOS'd simply by a government spending a lot of cash on a smart contract that does absolutely nothing for a whole week.

Etherum startups don't make a lot of sense unless you want to make some betting platform that would usually be taken down by the government.

</comment></comment></comment><comment><USER5> It is inspiring to see the additional ways people have implemented blockchain- based VC money plundering. While initial Bitcoin plundering efforts have stagnated, future Ethereum initiatives will continue that refreshing Ponzi- esque flavour we all crave.

</comment><comment><USER6> From the early comments placed here, it is clear that blockchain tech needs to be put to more uses than solving money.

The core of what bitcoin has solved is of course hugely innovative, important, and useful to many people. But it is also grossly mundane IMO.

Enabling new kinds of relationships creates far more interesting and enabling outcomes than value storage or exchange. This is why bitcoin side chains and the wide world of Ethereum will lead to the creation of far more economic value than bitcoin itself.

</comment><comment><USER7> I have doubts about whether any proof of stake cryptosystem can work in the long-term.

The devs have stated that this coin is going to switch to PoS at a later date and I think that transition is going to make a Bitcoin hardfork look tame in comparison.</comment>


Pugmarks.me: Google Now for Reading. Contextual recommendations on your New Tab. - <AUTHOR> http://pugmarks.me

<comment><AUTHOR> I'd love the HN community to review this product. We are ex-Google News, Microsoft Research, thats trying to build the ultimate discovery product for your browser.

Pugmarks is social search too. It indexes documents going through your social circles and makes it available in the right context.

Pugmarks works great on LinkedIn profiles - giving you news and articles about a person, or interesting to the person.

Pugmarks extracts themes to your browsing, and makes your New Tab a mirror of the ideas going through your mind.

</comment><comment><USER2> Awesome. I am moving my stuff out of Google Reader. Thanks for the link.

<comment><AUTHOR> Many times you have excellent articles come and go in your subscribed streams. Pugmarks indexes them and surfaces them again - in the right context, and increases the shelf life of good quality content. Hope its useful to you.</comment></comment>


Internet to be lowercase in New York Times and Associated Press - <AUTHOR> http://www.nytimes.com/2016/05/25/business/media/internet-to-be-lowercase-in-new-york-times-and-associated-press.html <comment><USER2> As early as 1990, I recall the guidance on capitalization being, if you are referring to internet as a class of internetwork, (possibly an internal network) then you use the lower case. If you are referring to Internet as _the_ Internet (that is, the world wide network, globally routable, connecting all sites) you use the capital letter. The difference is between the noun, internet, and one particular internetwork, the Internet.

In the last 25 years or so, the concept of an internet has basically disappeared (for the most part, maybe we just call it a LAN/WAN now), while the Internet (soon, according to NYT and AP, internet) is all that people mean when they refer to internet now. Which I guess is how language evolves.

</comment><comment><USER3> This makes sense. The "internet" belongs alongside air, water, electricity etc. It's now a ubiquitous part of this world and in my books everyone has the right to access it without censorship or surveillance.

</comment><comment><USER4> One Internet, one Earth. Proper nouns, all capitalized.

<comment><USER5> And one Universe.

\---

When television first came out, it was capitalized. If you think the internet is a medium, it makes me sense lowercase.

</comment></comment><comment><USER6> Guess I'm old. I've always capitalized it and generally never seen informed people using lowercase. And that's ignoring the public Internet versus any internet.

<comment><USER7> Have you ever used the term 'internet' to refer to networks other than the main worldwide network? I never have, so lower-casing it seems pretty reasonable to me, based on my current use of the term.

</comment></comment><comment><USER8> Maybe now the iOS spellchecker might do the same thing.

</comment><comment><USER9> > "It will probably take a while to get shift-I out of our muscle memory."

If only there was technology to flag potential spelling errors in the editor along with all of the other automated style rules used by the industry.

</comment><comment><USER10> But the Hindernet will still be capitalized.</comment>