Storing Passwords in a Highly Parallelized World - <AUTHOR> https://hynek.me/articles/storing-passwords/ <comment><USER2> It does not really matter what password hash you choose in 2016. Argon2 is great, and if you have a library for your platform that does it, use it. But if all you have is PBKDF2, please don't freak out and come up with some complicated alternative to storing passwords altogether.

The real, big risk is not using a password hash at all, and instead using "salted hashes".

<comment><USER3> Unless you generate a high entropy password for the user, in which case you can just use one round of a hash function and be done. For web authentication, most users will just have their browser remember it anyway and if not they can write it on a pice of paper.

<comment><USER4> Are you referring to the comment -

> The real, big risk is not using a password hash at all, and instead using > "salted hashes".

If so, your statement is incorrect. Without salting you are vulnerable to trivial rainbow table attacks [1]

[1] [https://en.wikipedia.org/wiki/Rainbow_table](https://en.wikipedia.org/wiki/Rainbow_table)

<comment><USER5> I think the parent was assuming a lovely world where all passwords would be large randomly-generated strings. In that case, salting is no better than just increasing the size of the randomly-generated string, right?

<comment><USER3> Yes, thanks. And the thing is they don't even have to be that long. Still inconvenient to enter a bunch of those every day, but the same is true with any password and most of us don't do that - we maybe set a master password and our web browser remembers them.

Salting is one of the best and easiest measures if there is a possibility of lowish entropy passwords, but if not it doesn't help (edit: on further thought I think there is a fairly substantial entropy range where the only viable attack is a very large scale batch attack and salting would prevent that, although the cost/benefit on actually doing that makes it an unlikely attack anyway). Iteration and memory hard functions also effectively add just a few bits of entropy. But when users choose passwords, a huge portion will be low enough entropy that nothing you can do will help enough.

A second factor does help in that case, and can provide additional protection to high entropy passwords (as long as the second factor is distinct from the system that stores the high entropy password, which in theory it should be to be called a second factor).

technion: yeah, my suggestion won't work if there is any way for the user to set the password.

</comment></comment></comment></comment></comment><comment><USER6> I don't have anything to add to this discussion, I just wanted to thank the author; this blog is essentially what I've always wished my blog to be: clean, with few but very informative updates, especially for us Pythonistas. The articles about deployment have been particularly useful. Thanks!

<comment><AUTHOR> Aw thank you!

</comment></comment><comment><USER7> Is there any reason to prefer Argon2 over scrypt? The article dismisses it as not popular, but doesn't say why we should invest in making Argon2 popular over scrypt.

<comment><USER8> One of the nice things that Argon2 has is server relief. Instead of doing the entirety of the computation on the server you can have the client do the most demanding parts and send those intermediate values to the server which can then finish the much easier computatation.

<comment><USER9> Really, any password hash `PH(m,p)` can have server-relief by adding a regular cryptographic hash function `H` and making H∘PH your hash function. By this, you store `p||H(PH(m,p))` in the database, and have the client compute `PH(m,p)` and send it to the server rather than sending the password `p`. You still have to send the salting information to the client though or use .

I don't see the server relief in Argon2's latest specification document though.

<comment><USER10> I.E., cram-md5 and related password schemes. It's a shame it fell out of favour, I'd sure prefer something like this over sending plain text passwords via TLS…

<comment><USER9> Ideally we eventually get rid of passwords and use secret/private keypairs in the browser, or have some browser-based SRP.

</comment></comment></comment></comment></comment><comment><USER11> Can someone help me understand why the author of the python library chose to raise an exception on verification failures as opposed to returning false?

<comment><USER12> On that topic, if it's raising an exception in the middle of testing the hash, could this permit a timing attack?

<comment><AUTHOR> It isn't raised in the middle of testing the hash. The testing is completely done in the Argon2 C library and the bindings raise an error if it returns an error. The Python library doesn't do anything smart at all except calling C functions on strings.

</comment></comment></comment><comment><USER13> Given the recent ACM article about faster-than-cpu persistent memory on the horizon, how does that affect memory-hard hashing algorithms?

A machine with a measly 256GB of memory can do around half a million Argon2 hashes simultaneously (using the Python library defaults for memory complexity of 512k), and that doesn't include the memory built into high end GPUs, or what could be added to ASICs or FPGA boards.

<comment><USER14> I do not think ACM article is very relevant here, because what is speaks about is, essentially, the fact that what we traditionally thought of "disks" is now getting as fast as RAM, because persistence is achieved via new technologies.

 _A machine with a measly 256GB of memory can do around half a million Argon2 hashes simultaneously_

Not necessarily. There is always the limitation of memory bandwidth. Although, I do not see saturating that bandwidth as a design goal of Argon2.

<comment><USER15> precisely.. if a hash is _too_ intensive, it's a clear vector for DDOS attacks.

</comment></comment></comment><comment><USER16> I'm skimming info from all over the place. Does anyone know the performance of argon2 over the other stuff? I'm figuring out whether to use bcrypt or argon 2 for a side project.

[https://www.npmjs.com/package/bcrypt](https://www.npmjs.com/package/bcrypt)

[https://github.com/ranisalt/node-argon2](https://github.com/ranisalt/node- argon2)

edit: found it. I'm still reading

[https://password- hashing.net/submissions/specs/Argon-v3.pdf](https://password- hashing.net/submissions/specs/Argon-v3.pdf)

</comment><comment><USER17> Can you unroll argon2 for cpu-vs-memory tradeoff like scrypt? Both are supposed to be memory-intensive, but in scrypt you can get 2x slowdown by using 2x less memory and just computing the missing bits. ([http://blog.ircmaxell.com/2014/03/why-i-dont-recommend- scryp...](http://blog.ircmaxell.com/2014/03/why-i-dont-recommend-scrypt.html))

<comment><USER9> From the white paper [https://password- hashing.net/argon2-specs.pdf](https://password-hashing.net/argon2-specs.pdf):

> Argon2i is more vulnerable to tradeoff attacks due to its data-independent > addressing scheme. We applied the ranking algorithm to 3-pass Argon2i to > calculate time and computational penalties. We found out that the memory > reduction by the factor of 3 already gives the computational penalty of > around 214. The 214 Blake2b cores would take more area than 1 GB of RAM > (Section 2.1), thus prohibiting the adversary to further reduce the time- > area product. We conclude that the time-area product cost for Argon2d can be > reduced by 3 at best.

</comment></comment><comment><USER18> Password hashes focused too long on computation time, neglecting memory usage. It's great to see some progress on that topic.

On the other hand, we still see password databases stored in plain MD5, sometimes even without salt. So in addition to provide better password hashes, making them mode widespread is important, too.

<comment><USER19> One thing I wonder about, as password hashes get both processor and memory- hard -- are we presenting a trivial DoS attack on our servers, by basically letting any IP submit a request for a server to dedicate this (non-trivial) level of RAM and processor to a given task (hashing a random string to verify that this login is not correct)?

I suppose the answer is yes, but it's worth it. It's possible to fix this sort of hole (partially, at least) by capping the number of hashes processed concurrently. But most simple implementations will just assume "we're not likely to have more than X users every signing in concurrently, so we can set the work factors based on that plus some headrooom".

<comment><USER20> If you're not throttling the number of incorrect password guesses per IP, or per IP per account, or both, you're opening your users up to a dictionary attacks anyway. If I can get a box close your yours (~5ms is doable if we use the same colo), then it's completely feasible.

You're right however that this is going to be painful, because most web frameworks don't have any of this throttling architecture (like persistent in- memory hashtables) in place, and as soon as they switch to Argon2/Scrypt, it's going to be an easy DoS vector... particularly for services running on weak VPSs.

It's another reason why solid, secure password authentication protocols, that do client-side hashing, will eventually happen. Even in fancy zero-knowledge asymmetric protocols however, online rate-limiting is essential.

<comment><USER19> Rate limiting per-IP assumes an attack from a single IP, or a very small range of them (so, only defends against a trivial DoS, not DDos... which are sadly easy to set up these days).

Per-IP per-account as well doesn't work if the attacker has a large list of usernames. Even brute-force "dictionary" attacks can dodge simple limiters by submitting one password with 2 million diff usernames, then a second password with 2 million usernames, etc..

I'm not saying these are _bad_ (though if someone can trivially stop your real users from signing in by hitting the limit on their accounts, that's just a DoS in another shape). But we're agreed already... these are non-simple problems, really.

</comment></comment></comment></comment><comment><USER21> It's sounding like it's too difficult to keep up with password security to try to do it all yourself. You'd be better outsourcing it such as with OAuth if you can.

<comment><USER22> While it certainly would be nice to outsource this problem,

        ph.hash("s3kr3tp4ssw0rd")
        ph.verify(hash, "s3kr3tp4ssw0rd")

Is pretty damn easy.

<comment><USER23> You're missing the point (which is hilarious, because the very first sentence of the linked article spells it out explicitly). The APIs for password hashing are simple, and always have been (c.f. man 3 crypt).

It's the problem area that is complicated and rapidly changing. And use of APIs and mechanisms that just a few years back were Best Practices is now discouraged. That's not something that can be treated by a cute Python API.

<comment><USER10> The best practice a few years ago was "use the best PBKDF available", which hasn't changed – just what algorithm to use. This isn't different from any other part of crypto – we have a constant algorithm churn as we identify and replace problematic algorithms.

And OAuth is no exception: If you implemented OAuth five years ago, it was OAuth1, which was obsolete and ripe for replacement… _four_ years ago.

<comment><USER23> Uh... the specific advice I was replying to was _precisely_ to use a specific python wrapper around Argon2, which is certainly not isomorphic to the "best PBKDF available" (even if it might be right now).

And the OAuth point is misdirected: we're talking about hash choices here, which OAuth is not. An OAuth1 client, while perhaps obsolete for other reasons, _would certainly_ be protected against the need for a change in password hash, because all that stuff happens upstream and it never sees the password. That is by design, therefore remote authentication against highly- trusted password verifiers[1] is a good idea, which is what the great- grantparent post was saying.

[1] Which, sure, has its own list of worries unrelated to hash behavior.

<comment><USER10> So what is your point then? No matter what you do – whether you verify password yourself or externally – the state of the art constantly changes and you have to update your code _either way_. There's no escape from that. The reasons change, but not the code churn.

<comment><USER23> Uh... no. That's exactly wrong. If bcrypt or whatever gets broken tomorrow, and my OAuth provider needs to reset all the passwords in the database (Linode literally did this last week, though not because of a hash change, it's a common kind of thing), _I don 't need to change a line of code._

Ergo, if you're worried about changes to password hash vogues, OAuth is a good idea. Thus the point way upthread, which both you and the other poster seem to have missed.

<comment><USER10> Because it's a nonsensical point. You trade one very simple thing to worry about – password hashing algorithms, for which you can count all variations in the past ten years on one hand – with a massively complex system like OAuth (1.0? 1.0a? 2.0?), which in turn relies on TLS for transport security, for which best practices change every two or three months.

If bcrypt gets broken tomorrow, your passwords are safe until you have a data breach. If OAuth gets broken tomorrow, you're _immediately_ at risk.

</comment></comment></comment></comment></comment></comment></comment></comment><comment><USER24> One concern I have with Argon2 is that it uses BLAKE internally. This is understandable, since BLAKE was developed by one of the conveners of the Password Hashing Competition, but it seems to me that using SHA3 internally would be a better choice, in order to leverage the industry-wide investments in development & testing which we're likely to see.

Assuming that it's well-designed, Argon2's structure shouldn't rely on BLAKE.

<comment><USER25> SHA-3 (Keccak) is not a better choice, because it has a considerable performance difference between hardware and software implementations, while BLAKE is very fast in software (or current general-purpose CPUs), which is good for most defenders (as they use general purpose hardware) and bad for attackers (as the cost of specialized hardware is increased). None of the Password Hashing Competition submissions used Keccak, and even the authors of Keccak recommended against using it for password hashing.

<comment><USER26> Gambit used Keccak.

<comment><USER25> Ah, true, I forgot about it. With the following remark in specs ([https://password- hashing.net/submissions/specs/Gambit-v1.pdf](https://password- hashing.net/submissions/specs/Gambit-v1.pdf)):

 _Keccak is known to be very fast in hardware, which opens up the path to highly optimized cheap circuits._

(although with strange follow-up: "but the same can be said about modern CPUs and GPUs, which closes the gap")

As I wrote when discussing candidates, "I like the simplicity of Gambit, but it would look a bit silly if we selected an algorithm as a winner of competition with a notice that it better be used in the future, when Intel adds a SHA-3 instruction."

</comment></comment></comment></comment><comment><USER27> When can we expect something for PHP?

<comment><AUTHOR> Cf [https://twitter.com/ircmaxell/status/685207462992035841](https://twitter.com/ircmaxell/status/685207462992035841) & ff

</comment></comment><comment><USER28> (Serious question) why don't people just use firefox Sync or Chrome password store? Why worry about all these? In both cases passwords are salted+encrypted and uploaded.

<comment><USER29> Firefox Sync and Chrome password store are for users storing passwords. They are not hashed (because that would make it impossible to log into a site, because you need the actual password to log in, and a hash is irreversible). I don't know what you mean by salted, because as far as I know, salts are only used with hashes and these are not hashed.

Bcrypt, scrypt, and argon2 are for server storage of login information. The server does not store the actual password, it just stores a hash.</comment></comment>


Fed, worried about recovery, will buy US debt - <AUTHOR> http://finance.yahoo.com/news/Fed-worried-about-recovery-apf-2353655020.html?x=0

<comment><USER2> Does this mean the foreigners have lost faith in the US government and stop buying the US debt? The Fed buying US debt is like the left hand buying from the right hand.

<comment><USER3> No, I believe it has almost nothing to do with foreign investment. (Incidentally, foreigners have _awesome_ faith in the US government, as evidenced by the fact that interest rates are as low as they have ever been, and -- frighteningly -- perhaps as low as they can be.)

No, the government buying its own debt is a very famous accounting trick designed to create more money in the economy. This act is the basis of the fractional reserve system, which in turn is the basis of all modern money.

The Federal Reserve bank buys bonds from bondholders (either newly issued bonds from the government or bonds from my safe deposit box). And in return it puts money in a bank account. Where does the money come from? It is just numbers in a database. The Fed just edits the database. Only the central government, which controls the currency, has the power to create money out of thin air in this way, but that power is very important.

(In other times, when the economy is really hot and inflation is soaring and we need to cool down, the Fed does the opposite: It raises interest rates, which causes people to want to buy bonds to get more interest, which causes people to spend their cash for those bonds. And, as each bond is bought, the Fed just erases the cash from bank accounts. The money vanishes! Remember, money is just a score in a game, the game of "keeping the economy working so that people can eat". It's not made of metal or anything. That was the old way, the way that didn't work.)

You can't get the same effect by selling bonds to England, because then England will pay for the bonds in dollars, and England can't create dollars. They have to buy dollars with pounds. So the total supply of dollars in the world would remain constant, which is not the point of this exercise.

(Now if I really understood econ we could go on to talk about why this little exercise might not work this time, about the "liquidity trap" and the "zero lower bound". But my time is up.)

<comment><USER4> _(Incidentally, foreigners have awesome faith in the US government,..._

This could also reflect a lack of faith in other governments rather than faith in the US government. If assorted European debt became 10x more risky but US debt only became 5x more risky, that would cause a flight into US debt.

<comment><USER5> Exactly correct. The U.S. can have economic problems, but as long we are still at the top, we remain at the top.

 _One of the more interesting reactions to the FOMC's gloomy appraisal of the economic state of affairs is the surge of strength into the dollar and the corresponding tumble in the euro. The U.S. dollar index — which is heavily weighted to the euro — is up 1.8%. The euro is down against the buck to the tune of 2.1%, a giant move in the forex markets._

[http://blogs.wsj.com/marketbeat/2010/08/11/oh-man-not- europe...](http://blogs.wsj.com/marketbeat/2010/08/11/oh-man-not-europe- again/)

<comment><AUTHOR> "but as long we are still at the top, we remain at the top."

Said the captain of the titanic.

<comment><USER5> Don't get me wrong. I'm the last person to suggest we're unsinkable. I'm only pointing out that our troubles become the troubles of everyone else, just as our success has influenced global economic success.

</comment></comment></comment></comment></comment></comment><comment><USER6> Does this give the Fed more influence over treasury note yields? If it were to start buying in higher volume, I would think so.

</comment><comment><USER7> i suppose it makes more sense to print money and lend it to itself than lending it to banks to lend to itself

<comment><AUTHOR> Why waste all that paper/electronic bits? Just enact a large tax increase...it's the same thing. Oh, but then the American sheeples will notice.

</comment></comment><comment><AUTHOR> massive printing of dollars, here we go!

<comment><USER8> Cause that's really dangerous when you're suffering deflation....you know, when you take actions to correct this situation.

<comment><AUTHOR> dangerous only for the banks.

<comment><USER7> oh, no, we're paying less for things. help!

the massive housing bubble and it inflationary effects HAD to collapse. those who waited out the insanity deserve lower prices. those who were foolish do not deserve to have the rest of us prop up their property values, although our government is foolishly backing every bad bet out there.

<comment><USER9> I'd like to say thank you, stretchwithme, for all of us who saved enough to buy a house, responsibly, in what just happened to be the middle of last decade and now have cause to sell (moving jobs because the economy stinks). We so deserve to seriously be screwed over because of our timing. Thank you US government. Thank you stretchwithme, your sentiments help the situation tremendously.

Surprisingly, there were actually many people who bought a house in 2003-2005 because they had a down payment at that time and they had jobs and they bought what they could afford.

<comment><USER10> You seem to think because you had a down payment, had a job, bought a house you could afford, you "deserve" to be screwed over less by a falling housing market, because you were "responsible".

Just the opposite! The migrant field worker who got a 0% down interest only loan on a 500K house is far less "deserving" of the screw than you, because he has no money to be screwed out of.

<comment><USER7> sorry, I didn't mean to be insensitive. you don't deserve this mess, but we own the risks we take on. There's simply no other way to run a society.

When the dot com bubble burst and I lost a lot of paper profits, that was unfortunate and unforeseen and I was responsible for the choices that got me there. It would have been nice if someone sent me a check for my losses, but how could the government do such a thing without making someone more responsible pay for my mistakes?

And we don't deserve to have our government doing this to us, again and again, but we own that until we make it clear that they should stop.

Its not China's responsibility or Saudi Arabia's. It ours.

</comment></comment></comment></comment></comment></comment></comment><comment><AUTHOR> The US government is printing/robbing you of your savings. Don't believe it for a second it will trickle down to you; the last batch of printing/bailout went to banks and shareholders.

On a side note, man, there's alot of sheeples on this board. Do you all like get up early in the morning to go to work and get robbed?

</comment><comment><USER11> It's the financial equivalent of eating your own poop.</comment>


Modern dynamic linking infrastructure for PLT - <AUTHOR> http://lambda-the-ultimate.org/node/3474

<comment><USER2> Some really nice points. I would love to use ELF and GOT rewrites for languages that I hack on.</comment>


Git Tips - <AUTHOR> http://www.alexkras.com/19-git-tips-for-everyday-use/ <comment><USER2> Here's my favorite git oneliner:

      git push origin --delete $(git branch --merged origin/master -r | grep -v master | grep origin | cut -d/ -f2-)

It deletes all remote branches that are merged into the remote master. Because people always forget to click the "Delete branch" button after merging their pull requests.

EDIT: And another thing. Turn all your Git aliases into shell aliases (e.g. "git status" is aliased to "git st" is aliased to just "st" on my system) with this one weird trick! [https://github.com/majewsky/devenv/blob/2c4252d37597617a493f...](https://github.com/majewsky/devenv/blob/2c4252d37597617a493f1addadacfbf8d88b4942/toplevel/zshrc#L55-L59)

<comment><USER3> Why would you want to delete merged branches? i like to keep them. It's not costing you any storage.

<comment><USER4> Personally because it makes both `git branch -a` and tab-completing a branch virtually useless because there's just too many branches.

<comment><USER2> Exactly.

</comment></comment></comment></comment><comment><USER5> Love #16 and the associated article. I cringe when I see something like "js fix" as a commit message.

However, not on board with the hate for rebase. If it's your feature branch and you aren't sharing it with others, I'd much rather get a cleaned up PR than one filled with junk commits because the author was afraid to rebase.

<comment><USER6> Nitpick with #16: I don't like when there is no "why" associated with a style rule. I'm speaking specifically to #3, "Capitalize the subject line". Why is this preferred to not capitalizing a subject line? Is there a reason, or is it an arbitrary decision (which would be fine - sometimes a style decision is, "We need a standard, and we chose this one." But I think the author of the style guide should explain that that's what they're doing).

But in general I liked the article. It's under-appreciated how helpful a good commit log is.

<comment><USER7> Capitalize the commit message because ...

Sometimes commit message deserves to be moderately long, like a paragraph or more. If one did not reference a code review or issue tracker link, for example, the need would likely be common. Even with an issue tracker there's value in capturing a few sentences of detail and sometimes more in the commit.

By capitalizing the commit message, it is natural to extend it from a single sentence phrase to a paragraph with an introductory sentence. I think of commit messages as analogous to JavaDoc where the first sentence or first line is taken as the subject, or like a document with a heading. It would be silly for short commit messages not to use capitalization while long do. A capitalized message works as a short description of the intro to a paragraph. Generally. once you're writing more than a single standalone sentence, capitalize. In a permanent medium where it can be either one, also capitalize.

A capitalized sentence or sentence fragment works in many contexts, like within a document or email. It would be annoying to convert going back and forth. Lastly, commit messages can sometimes act as names or proper nouns, and capitalization reinforces that role. Capitalization looks better as project or item headers in an issue tracker, and so to the extent that commits follow items, it's nice to consistently capitalize.

Capitalize commit messages because when Git tools generate commit messages, they generate capitalized ones. (Be consistent.)

I grok not capitalizing IM or IRC or other chat conversations, though I do if I start writing messages longer than a sentence. For everything but IM, capitalization constitutes the norm and good professional style.

This rationale might not be entirely satisfying, because why does English have capitalization to begin with? That's a linguistic question I don't know the answer to. However, I do know that the norms that lead to sentence capitalization also apply to commit messages and code documentation Not capitalizing them feels like refusing to capitalize any other writing: a potential distraction from the content.

<comment><USER8> Most of my commits start with a component name prefix and a colon. I don't capitalize those. I still write literate English commit message bodies.

<comment><USER7> That sounds reasonable to me.

      foo: Add utf8 support

If the component name is conventionally written in lower case, because it's a technical term, then I think a comment written in this way counts as good style. I might choose to capitalize Foo myself, but I think foo is defensible in that context.

</comment></comment></comment></comment></comment><comment><USER9> I'm an avid supporter of rebasing over merging. I'll refrain from getting into my workflow, but here's what i want to say: For the love of all that is holy, never squash a merge commit nor squash onto a merge commit.

Squashing a merge commit is a perfect way to reintroduce old code back into master.

Squashing onto a merge commit is great way to lose changes. It's been a while since I have tried this, but creating a didactic repo if fairly easy. Create a repo with two feature branches, a file on each of master and the feature branches, merge featureA to featureB, make some changes or delete a file, `commit --amend` on the merge, and merge featureB to master. Then use `git cat-file` to look at those commits and commit trees. I've seen mysterious things such as simple as unreported changes to files mysteriously being deleted from the repo.

</comment><comment><USER10> I use git often on the command line, especially for non-trivial tasks, but do not quite understand why someone would pick it for routine committing/tree view over a GUI. I find the ability to graphically see the state of the working tree and the index, and being able to stage/unstage individual lines by visual selection in a non-linear way absolutely natural. Likewise, the ability to get a context menu for a commit in a tree view and fire a command (rebase/branch most often) seems like something git was "built" for.

<comment><USER11> I spend a lot of time in the commmandline, so it makes sense for me to not use a GUI. This alias (in ~/.gitcofig) is gold for most of the "graphical" stuff I'd use something like SourceTree for.

        [alias]
    	lg = log --all --color --graph --pretty=format:'%Cred%h%Creset -%C(yellow)%d%Creset %s %Cgreen(%cr)%C(bold blue)<%an>%Creset' --abbrev-commit

<comment><USER12> I have these alias, plus :

        lg2 = log --graph --abbrev-commit --decorate --format=format:'%C(bold blue)%h%C(reset) - %C(bold cyan)%aD%C(reset) %C(bold green)(%ar)%C(reset)%C(bold yellow)%d%C(reset)%n'' %C(white)%s%C(reset) %C(dim white)- %an%C(reset)' --all

<comment><USER13> these are really good! I have this one..

      lg = log --graph --decorate --pretty=oneline --abbrev-commit

</comment></comment></comment></comment><comment><USER14> My tip #20: tig [1]

[1]: [http://jonas.nitro.dk/tig/](http://jonas.nitro.dk/tig/)

<comment><USER15> gitk is great if you are outside a terminal

</comment></comment><comment> Pirate-of-SV Since interactive rebase was mentioned I'd recommend taking a look at its --autosquash flag and the --fixup flag of commit. With autosquash the interactive rebase will move fixup commits to where they belong:

      git commit --fixup=<commitref>
      git rebase -i --autosquash HEAD~5

<comment><USER16> Also enable --autostash (git config --global --add rebase.autostash true) for that workflow so that dirty files do not get in the way.

</comment></comment><comment><USER17> My recommendation: configure the shell prompt (add __git_ps1 to $PS1), this saves a ton of time and makes it much easier to understand what is going on. Especially recommended for beginners (but obviously not limited to).

<comment><USER12> Use fish shell and setup it to use the git prompt. Far better that bash + __git_ps1

</comment></comment><comment><USER18> Mandatory xkcd reference: [https://xkcd.com/1597/](https://xkcd.com/1597/)

<comment><USER19> So true

</comment></comment><comment><USER20> For vimmers: [https://github.com/tpope/vim- fugitive](https://github.com/tpope/vim-fugitive) is awesome, especially for only adding only parts of a file to a commit.

<comment><USER21> "fu git i've" is yoda speek for the common scenario "f-ed up git I have"

</comment></comment><comment><USER22> Site is non-responsive for me. Cached link:

[http://webcache.googleusercontent.com/search?q=cache:04x449v...](http://webcache.googleusercontent.com/search?q=cache:04x449vvRhkJ:www.alexkras.com/19-git- tips-for-everyday-use/&num=1&client=ubuntu&hl=en&gl=nl&strip=1&vwsrc=0)

</comment><comment><USER23>
        git checkout -

Switches to the previously checked out branch.

<comment><USER24> More generally "-" is an alias for the previous branch, so it can be used in other commands as well (ex: git merge -)

</comment></comment><comment><USER25> I hope I'm not the only one who thinks that "Update README file" is a poor commit message. It's overly nondescript. A better commit message would be "Update README install instructions" or "Fix README links to dependencies" or anything else that better describes what about the README file was updated.

</comment><comment><USER26> My #1 git tip to anyone who will listen: use magit. Having the options for each command within easy reach makes getting git to do the right thing so much easier, and the UI feels very emacs.

[https://github.com/magit/magit](https://github.com/magit/magit)

<comment><USER27> A golang+termbox clone of this would be great...

(Of course, parsing text is a joy with Emacs/Lisp and a PITA in Go).

</comment></comment><comment><USER28> For git commit messages. I've began liking to initiate or end a statement with stuff like these:

¯\\_(ツ)_/¯

( ͡° ͜ʖ ͡°)

(ง'̀-'́)ง

(ノಠ益ಠ)ノ彡┻━┻

(゚∀゚)

ಠ_ಠ

How do you guys feel about these in git commit messages?

<comment><USER29> If they're relevant to the code change ("Not sure what I did, but the tests are passing now ¯\\_(ツ)_/¯", "Removing production usernames and passwords from front-end config files, MIKE ಠ_ಠ"), it's fine.

If I see a commit that's just "¯\\_(ツ)_/¯", I will force-push over it, don't you even think about it.

</comment></comment><comment><USER30> At my old job I had to do a major cleanup of a git repo with dozens of old branches. I got really good with the `git log` command. Here my alias form my ~/.gitconfig that shows author, branches, and remotes in a colorful manner:

        [alias]
            ll = log --graph --oneline  --decorate --date=short --all --pretty=format:'%ad %h %Cgreen%an %Cred%d %Creset%s'

With this you have a pretty good picture of what's going on accross all branches.

<comment><USER31> Can you provide an example of the output?

<comment><USER30> Sorry for the delay, the main idea is to show all the branches on the different remotes:

        * 2016-04-11 86f68c3 Ivan Savov  (HEAD, origin/bugfix/math_valign, bugfix/math_valign) Better solution to
        | * 2016-01-20 df77345 Ivan Savov  (origin/miniref, miniref) Customizations necessary for no BS math render
        |/
        * 2016-04-07 364619f Michael Hartl  (upstream/stable, upstream/master, origin/master, origin/HEAD, master) Change to bigger line height for EPUB/MOBI
        * 2016-04-07 92952da Michael Hartl  Bump version
        *   2016-04-07 6a56c24 Michael Hartl  Merge pull request #152 from jackkinsella/patch-2
        |\
        | * 2016-03-23 216d410 Jack Kinsella  softcover check compatible with brew
        * |   2016-04-07 d792d4b Michael Hartl  Merge branch 'minireference-feature/math_valign'
        |\ \
        | * \   2016-04-07 5fa560e Michael Hartl  Merge branch 'feature/math_valign' of https://github.com/minireference/softcover into minireference-feature/math_valign
        | |\ \
        | | |/
        | |/|
        | | * 2016-01-31 77a4e5e Ivan Savov  (origin/feature/math_valign, feature/math_valign) Set veritcal-align for better looking inline math
        * | | 2016-04-07 a369139 Michael Hartl  Update README
        |/ /
        * | 2016-03-21 0e94095 Michael Hartl  Bump version number
        * | 2016-03-21 a5fbc82 Michael Hartl  Reuse menu headings in MOBI ToC
        * |   2016-03-21 fadd558 Michael Hartl  Merge branch 'master' of https://github.com/softcover/softcover
        |\ \
        | |/
        | * 2016-03-16 60dc7c9 Nick Merwin  (tag: v1.2.5) added exercise generator for course chapters
        * | 2016-03-12 fe2f844 Michael Hartl  Lower the EPUB/MOBI line height
        |/
        * 2016-03-10 bba610b Michael Hartl  Make anal changes
        * 2016-03-10 c9bad17 Michael Hartl  (tag: v1.2.4) Bump version number
        * 2016-03-10 8edf40a Michael Hartl  Add symbols for euros and pounds

</comment></comment></comment><comment><USER32> Great set of tips. Unfortunately I find that so many post that suggest rebase recommend against force pushing. This takes a lot of the power away from rebase/amend. So much does this happen that is wrote a blog post titled "You should force push more"[0].

0: [https://hugotunius.se/2014/09/08/you-should-force-push- more....](https://hugotunius.se/2014/09/08/you-should-force-push-more.html)

<comment><USER33> My _current_ understanding is that... "it depends". I like that you identified ground rules/preconditions for aggressive force-pushing.

Imagine a scenario for a large team (think kernel). Chat logs of a team developing some feature look something like:

A: Ok, I have finished with issue1, check out commit abcd1 B: Hm, does not work for me. Looks like race condition in foo.c, fixed in abce2. Going to sleep. C: You broke init_foo(), reverted in qwer3 A: Ok, modified init_bar() works on my machine, check dvor4 C: LGTM

Yet release branch contains all those commits squashed into abcd1. Now person B (or anyone else) has no idea whether which versions of init_foo() and init_bar() work on their machine and what to fix.

We sometimes tend to forget (and tools like github/gitlab help with that) that not only git, but the whole workflow might be distributed and changing history in one place might desync it with other places. I think that unless all that history is absolutely unnecessary, e.g. done by a single developer, is extremely well tested, contains loads of "oops" commits, history should not be messed with. That's why all the warnings: git cannot manage all the references to history

</comment></comment><comment><USER34> My favourite tip for commit messages: write them as if you would explain someone how to do what was done in the commit. So, instead of "fixed a bug" use "replace datetime object with date". It can be more abstract for larger commits (e.g. "refactor payments"), but it should be roughly analogue to what you would write to a ticket title to the same effect.

</comment><comment><USER35> My must have: pre-commit unit tests. See [http://daurnimator.com/post/134519891749/testing-pre- commit-...](http://daurnimator.com/post/134519891749/testing-pre-commit-with- git)

<comment><USER36> I usually rely on travis-ci to do this after I've committed and pushed to a branch (and only merge back when I get a green result, this is shown neatly in the Github UI).

A pre-commit hook probably works fine if your tests run in a few seconds. But in one of my projects, tests run for 10+ minutes and I want to get a test run on debug and release builds with a few different compiler versions. That's about 50 minutes of CPU time (but embarassingly parallelizable, although I typically don't do that). I don't want to have to wait for this to happen before every commit (not all of which even require testing, e.g. README commits).

<comment><USER35> Yeah I only run the unit tests against my current locally installed packages.

Via travis I then run a full matrix.

</comment></comment></comment><comment><USER37> Talking of commit message styling, Antirez (of Redis fame) also has a say. [http://antirez.com/news/90](http://antirez.com/news/90)

</comment><comment><USER38> love this one :

        git diff --name-only | uniq | xargs $EDITOR

(opens all modified files)

and this one to open files with conflicts

        git diff --name-only --diff-filter=U | uniq  | xargs $EDITOR

<comment><USER12> List the files with changes

        git diff --stat

And git merge-tool & git diff-tool So I can use kdiff3 when I have merge conflicts.

</comment></comment><comment><USER39> Am I the only one who's strugling to see the website? My Phone is having a bad time trying to show it (Windows Phone, Internet Explorer)

<comment><USER40> Works fine on mine (Windows Phone 10, Edge).

</comment></comment><comment><USER41> I seriously must be doing something wrong, but when I'm doing an interactive rebase, if it stops due to a merge conflict I sometimes find I need to tweak other files, and to add the files to git I find I do the following:

      git add $(git diff --name-only)

Am I wrong, or is this the right way of doing this?

<comment><USER42>
      git add -u

<comment><USER41> Awesome :-) I guess I need to re-read the git man pages yet again. There's also git add -A which I can use if I add a new file.

</comment></comment></comment><comment><USER43> I work with git daily for couple years and I find it quite tiring to follow git branches/commits in command line.

I rather use Git Extensions on windows and on mac Git Kraken. For beginners I think it would be better to use GUI tools.

<comment><USER44> For experienced users, I do agree wholeheartedly. However, to learn how and why git works, it is much more intuitive to work from the command line for a while. For example, you can browse the .git directory, see how refs work, browse through the logs, etc. etc.

Another reason why I advice learning the command line interface first is that it is the most versatile, so it's helpful to be able to fall back to it when necessary. Every GUI tool I've seen so far lacks a feature or two.

<comment><USER43> For learning what 'rebase', 'merge', 'sqush', 'git fetch', 'git pull' does it is nice to have a GUI. Command line on the other hand is usefull to restore lost commits with 'git log' and 'git reset' and I find it quite advanced topic. I think all basic commands are all in GUIS so those that I listed in first line.

</comment></comment></comment><comment><USER45> I'm still not even quite sure how to a) discard edits that I've made but not added/committed yet and b) delete something from a repo that I added accidentally.

<comment><USER46>
        git checkout somefile

will discard edits which you've not added to the index. This command is destructive of course.

        git rm -f somefile

will delete a file from the repo (you then have to commit the change). If you want to permanently delete a file, including from history, eg because it contains a password, that's a bit more difficult, but possible in some circumstances.

</comment></comment><comment><USER47> I'm collecting my personal notes/tips at: [https://gingkoapp.com/git- notes](https://gingkoapp.com/git-notes)

</comment><comment><USER48> I never recall the command, but it lives in my bash history somewhere. Running your test suite over a series of commits can be very handy.

<comment><USER42> Something like

      (set -e; git rev-list --reverse upstream.. | while read r; do git checkout -q $r && make check; done)

(Aped from the very excellent [http://blog.extracheese.org/2010/12/rebase-is- safe.html](http://blog.extracheese.org/2010/12/rebase-is-safe.html))

</comment></comment><comment><USER49> git log --format=%ae | sort | uniq -c | sort -nr

</comment><comment><USER50> "Table of contents"

</comment><comment><USER51> git reflog is my favorite

<comment><USER52> I used reflog for the first time a few days ago. I rebased a branch, and somewhere i messed up the conflict resolution, so I wanted to undo the rebase. It was surprisingly easy!

<comment><USER2> reflog is one of these things where you think "that would have been awesome to have" in hindsight.

In 2009 or so, I was using git-svn on the KDE SVN. I was preparing a major refactoring of Kolf in a local branch (about 60 commits with 10000 lines changed), and pushed it to the SVN once I was satisfied with it.

Unfortunately, the 20th-or-so commit failed because a SVN pre-commit hook rejected it (there were some DOS line-endings in a source file that I imported). At that time, git-svn was (AFAIR) already prepared to handle a failing SVN pre-commit hook on the _first_ commit (e.g. when authorization is missing or stuff like that), but not somewhere in the middle.

The end result was that my working copy, my index, and my master branch was completely trashed. And I didn't have a backup... Luckily, the tip commit of my feature branch had not yet been packed, so a clever grep on .git/objects found it, and I could recover my pre-push state.

Since then, reflog has always been enabled on my systems. Luckily, I haven't needed it since.

</comment></comment></comment><comment><USER53> Wow, I had no idea git bisect was so easy to use

</comment><comment><USER54> Brilliant tips</comment>


Dynamic langs have a refactoring libraries that know as much as a static IDE - <AUTHOR> http://rope.sourceforge.net/

<comment><AUTHOR> I use this in Emacs on a daily basis, gives me all the completion, "go-to method definition", and refactoring power I get in Eclipse+Java.</comment>


What's your favorite TED talk? - <AUTHOR> http://www.reddit.com/r/AskReddit/comments/kfl17/what_are_the_best_ted_talks/

<comment><USER2> I typically like the most scientific ones rather than the touchly-feely ones. There was one about stock market crashes which I found really interesting . here it is [http://www.ted.com/talks/didier_sornette_how_we_can_predict_...](http://www.ted.com/talks/didier_sornette_how_we_can_predict_the_next_financial_crisis?language=en)

</comment><comment><USER3> Haven't kept up very well, so there might be more recent ones that best these, but from what I remember...

The funniest while still being somewhat interesting was Ze Frank. The most interesting while still being really funny was Sir Ken Robinson. Lessig's talk about Sousa was also pretty amazing.

</comment><comment><USER4> <shameless_plug> Are we allowed to suggest our own? :) [https://www.youtube.com/watch?v=MY3XWYr726I](https://www.youtube.com/watch?v=MY3XWYr726I) </shameless_plug>

I'll get my coat ...

<comment><AUTHOR> If this is your favorite, of course ;)</comment></comment>


Lock-free multithreading with atomic operations - <AUTHOR> https://www.internalpointers.com/post/lock-free-multithreading-atomic-operations <comment><USER2> > The spin until success strategy seen above is employed in many lock-free > algorithms and is called spinlock: a simple loop where the thread repeatedly > tries to perform something until successful. It's a form of gentle lock > where the thread is up and running — no sleep forced by the operating > system, although no progress is made until the loop is over. Regular locks > employed in mutexes or semaphores are way more expensive, as the > suspend/wakeup cycle requires a lot of work under the hood.

A compare-and-swap loop is not a spinlock. (It is the primitive used to implement a spinlock, but that's different.)

Pure spinlocks are almost always a bad idea, at least in userland, because threads spinning trying to acquire the lock look "busy" to the scheduler, so the scheduler may run them instead of the thread that owns the lock. It does make sense to spin a limited number of times before going through the slow path to acquire a mutex, but most OS mutex implementations have that functionality built in, so there's usually no need to do that manually.

<comment><USER3> Why would it make more sense to "spin a limited number of times before going through the slow path to acquire a mutex"? Does a mutex have more overhead in the short term than a spinlock but after a few cycles become more efficient?

Off topic, but I remember reading that the Linux kernel prefers spinlocks to mutexes. Is there a good technical reason for that?

<comment><USER4> > Does a mutex have more overhead in the short term than a spinlock but after > a few cycles become more efficient?

Think about what a true mutex does. The true mutex switches into kernel mode (aka: your program is no longer running, Linux is running). That means a spectre-guard / meltdown guard is executed (your TLB buffer may be flushed, as well as various other memory-guards to prevent Spectre from leaking data).

Once the guards are executed, kernel-mode has to find more work to do. Traversing the kernel-data structures can take 1 to 10 microseconds, depending on how cold the cache is. Finally, since another thread may be running (before your thread comes back), you probably lost all your data from L1 cache (and at minimum: your branch-predictor state because of Spectre).

A spinlock without any contention takes less than 10-nanoseconds to run, to maybe 50-nanoseconds with a bit of contention (!!). You're basically reading/writing data to L1 cache, maybe L3 cache under contention.

However, a scheduler invocation will be on the order of 5000 nanoseconds (~5 microeconds) or so, due to all of the work that the scheduler has to do.

\</comment></comment></comment><comment>--

Window's default spincount is something on the order of 4000 cycles. Spinning for 4000-cycles (or less) is an advantage towards a spinlock-like methodology. (4000 cycles x 4GHz == 1-microsecond). Just to give you an idea of the speed- magnitudes that are being discussed here.

<comment><USER5> That's 4000 cycles @ 4 GHz == 1us right?

<comment><USER4> Yes, sorry. I'll go edit that correctly really quick...

</comment></comment></comment><comment><USER4> Yet another article discussing atomics without discussing the memory model or memory fences.

Atomics are the easy part to understand. Memory fences and memory-ordering is the hard part that needs far more discussion. If you use atomics with the wrong memory fence, everything will break.

Memory fences are necessary to make the compiler, CPU, and caches put the data into main-memory in the correct order.

\</comment><comment>----

Do NOT write Atomics unless you understand fences. It is absolutely essential, even on x86, to put memory fences in the correct spot. Even though x86 automatically has acquire/release consistency... the compiler may move your data ordering to the wrong location. Even with "volatile" semantics, your data may be committed to memory in the wrong order.

Fredor Pikus has an amusing example where you can have a properly coded lock but the compiler (may) mess you up: [https://youtu.be/lVBvHbJsg5Y?t=811](https://youtu.be/lVBvHbJsg5Y?t=811)

\</comment><comment>----

EDIT: if you use locks (spinlocks, mutex, condition variables, etc. etc.), the author of the library would have put memory fences in the correct place for you. Only if you use synchronization "manually" (and... you probably are doing that if you are writing atomics), do you have to think about memory fences.

<comment><USER6> > Topics like sequential consistency and memory barriers are critical pieces > of the puzzle and can't be overlooked if you want to get the best out of > your lock-free algorithms. I will cover them all in the next episode.

Not every article has to be an exhaustive description of everything you need to know.

Anywho, sequential consistency is the default for C++11 and C11 atomics, so you pretty much can use them without a complete understanding of memory consistency models.

<comment><USER4> Ah, you're right. They call them "barriers" instead of "fences", so I didn't pick up on that paragraph.

Still, C11 and C++ atomics aren't widely implemented yet in the GPU world. OpenCL 1.2 doesn't have them (and OpenCL 2.0 is not commonly implemented), CUDA doesn't have them yet, AMD ROCm doesn't have them yet.

And anyone with older compilers will probably be working with fence intrinsics, instead of "innate" barriers per atomic instruction.

</comment></comment></comment><comment><USER7> Note that lock-free is not always faster because contention is complicated.

<comment><USER8> Also note that lock-free can scale worse, again due to contention.

If you want scalability, you want to minimize inter-core and inter-socket traffic.

<comment><USER7> This is the more important point, realistically. I think I probably would've been better off saying 'lock-free is not a panacea.'

All in all, scaling up contentious operations is hard.

</comment></comment></comment><comment><USER9> I think "lock free" is a misnomer because in many cases these atomic operations still perform locks at the hardware level.

<comment><USER10> "lock free" regards to being free from any process/thread/device holding a lock for an undetermined amount of time.

The formal definition IIRC is "a system is lock free, at least one concurrent process makes progress towards finishing in a unit of time" (There's a hard to achieve version called "wait free" which means "every process makes amortized progress towards finishing")

The important property of a lock free system is that pausing any thread/process does not stop the others from progressing; whereas with classical locks (mutex, semaphore, spin, whatever), if you pause a lock- holding process you risk starving the entire system.

<comment><USER11> I love your last paragraph. It's extremely intuitive and yet I don't think I've ever seen it explained that way.

<comment><USER12> Not to take anything away from the parent's thoughtful comment. That important property is, in fact, the definition of lock-free. Lock-free and wait-free (which has a stronger guarantee) algorithms are sub-classifications of non- blocking algorithms.

If I could recommend one book on this topic it'd be "The Art of Multiprocessor Programming". The authors have contributed to this subject through original research and make the topic quite approachable.

<comment><USER11> > Not to take anything away from the parent's thoughtful comment. That > important property is, in fact, the definition of lock-free.

According to Wikipedia, it's the definition of non-blocking, not the definition of lock-free... which I guess is a correction for both the parent comment and yours. [1]

As far as I recall ever seeing, people always talk about it in terms of guaranteed systemwide progress, which I don't find as enlightening as what happens when you pause a thread.

[1] [https://en.wikipedia.org/wiki/Non- blocking_algorithm](https://en.wikipedia.org/wiki/Non-blocking_algorithm)

</comment></comment></comment></comment></comment><comment><USER13> I found it surprising that atomic operations in Java are converted in to locking when compiling for some versions of android. The guarantees of Java (before 9) on ordering are slightly too strong and result in locks being used under the hood.

<comment><USER14> Interesting. This probably is because of some hardware limitations on some older android hardware platforms.

The java.util.concurrent package was added in Java 5 and before that existed as a third party library (by Doug Lee). It contains a lot of concurrency primitives and makes use of a lot of things, including lock free instructions and optimistic locking: [https://en.wikipedia.org/wiki/Java_concurrency](https://en.wikipedia.org/wiki/Java_concurrency)

</comment></comment><comment><USER15> Based on some experience on non-X86 systems, lock-free is where you start finding bugs in compilers and runtime libraries, and also hardware.

How much do you trust random ARM-based SOCs to get this right at all of the necessary levels of cache consistency and memory access queues? Lots? Great, I am happy for you. Now, extend your confidence into some other chips, like PowerPC (various versions), MIPs and maybe a couple of others.

Eventually you are going to hit some very odd bugs, the really difficult bugs that will make you tear your hair out, and eventually that lock-free stuff, too.

"But we'll never run our stuff on a MIPS, or a Fthaghn-V1000." That's good . . . but the chipset you trust might get an update with slightly different memory behavior and you'll be sunk.

The projects I've been on that have shipped lock-free structures have done so _only_ when using them was (a) high value, (b) there was a way to choose an alternate method (usually at compile time), and (c) the use was very limited (e.g., just a handful of critical places).

</comment><comment><USER16> If multi-threaded code is too easy for you, try multi-threaded code which is executed in a different order than written!

Depending on access mode, the compiler, or the hardware, can still change the order of operations. It adds whole another level of WTF:

[https://llvm.org/docs/Atomics.html#atomic- orderings](https://llvm.org/docs/Atomics.html#atomic-orderings)

> It is also possible to move stores from before an Acquire load or read- > modify-write operation to after it, and move non-Acquire loads from before > an Acquire operation to after it.

<comment><USER17> > If multi-threaded code is too easy for you, try multi-threaded code which is > executed in a different order than written!

 _Most_ multi-threaded code works this way. Even safe languages like Java permit this. I believe the exceptions tend to be interpreted languages like Python, using green threads.

At the assembly level, most modern CPUs are permitted to perform out-of-order execution. I believe the exceptions are pretty rare these days. The custom PowerPC chip in the Xbox 360 guaranteed in-order execution, for instance [0]. GPUs are a different beast.

[0] [https://en.wikipedia.org/wiki/Xenon_(processor)#Specificatio...](https://en.wikipedia.org/wiki/Xenon_\(processor\)#Specifications)

<comment><USER16> The difference is that reordering is not observable with just one thread. Both language optimizations and CPU OOO don't change semantics within a single thread. If you use higher-level primitives like mutexes, they come with fences that also preserve that illusion.

It does become observable when you try to do synchronization yourself with atomics, without setting appropriate ordering requirements. Lock-free algorithms are usually tricky, and implementing them with minimum ordering requirements makes them even more puzzling.

</comment></comment></comment><comment><USER18> This presentation was fun, especially the part where he shows his spinlock beating atomic CAS.

[https://youtu.be/ZQFzMfHIxng](https://youtu.be/ZQFzMfHIxng)

</comment><comment><USER19> I see that, in a related article, they missed a nice in-joke:

> A gentle introduction to multithreading — Approaching the world of > concurrency, two steps at a time.

(I wonder if "multiple steps at a time" sounds better)

:-)

</comment><comment><USER20> The site is dead unfortinately.</comment>