Why There Is No Hitchhiker's Guide to Mathematics for Programmers - <AUTHOR> http://jeremykun.com/2013/02/08/why-there-is-no-hitchhikers-guide-to-mathematics-for-programmers/ <comment><USER2> All right, so this article can be probably condensed to these two points:

1\. "Mathematics is hard and inconsistent, but so is programming".

Well, yes, but this is exactly the problem! I pity poor souls who started their journey into programming with, say, C++ — the champion of consistency and elegancy. But there are other ways and other languages. And, with some effort, it _is_ possible to provide gentle introduction into programming world. The same should apply for mathematics.

2\. "You can't learn mathematics on your own, but it's the same with programming".

And here I deeply disagree. Programming as activity is _the_ perfect learning environment. Find some coding sample, compile and run it (or paste to REPL), see if it gets you the same result. Try to change some variables in the code, see how behaviour changes. Can you run it without this function? What this method does? (A quick look to the documentation reveals everything, including additional arguments to try). You can tinker and hack and play and learn with every language and environment out there, it's like a perfect experimental setup where experiments are cheap, and everything is perfectly reproducible, at least on your own machine. ;)

With math, there are no docs, no playground, no explanations, no nothing. Either you are the part of mathematical community and can hope for explanations and knowledge transfer, or you are not. In programmers parlance, it's like being the part of a huge legacy project with long compile times and cryptic documentation, where the only way you can learn is ask your senior colleagues questions like "what this class is doing?" and get answers like "oh, we are not doing _that_ anymore, this is an artefact from like 15 years ago, still widely referred in the docs, though. Here is the right way instead". Sounds familiar?

This is why learning math is way harder than it should be.

<comment><USER3> > With math, there are no docs, no playground, no explanations, no nothing. > Either you are the part of mathematical community and can hope for > explanations and knowledge transfer, or you are not

what's the difference to CS here, where did you get that coding sample you mentioned? The playground is your mind, pebbles or pen and paper, maths are a projection of the real world.

Mathematics is not inconsistent. Isn't this the premisses of rigor? Math has mathematical consistency and that's not the same as literal consistency, but maybe that's a problem in semiotics and language. The language and the literature is inconsistent.

<comment><USER2> > where did you get that coding sample you mentioned

I learned programming by reading books and trying samples from there, then, step by step, beginning to understand how parts play together and why some things are faster than another. I could _feel_ loops and recursion, tinker with them, before trying to understand them theoretically.

I can't do that in learning math (yet; Wolfram Alpha is changing it). It's either I understand the exact meaning of symbols, or I don't. There are no intermediary steps.

And being rigorous is not the same with being consistent. (There isn't much consistency in programming, either, but this is a minor problem).

</comment></comment></comment><comment><USER4> Applied science, i.e. taking something from text book abstract and general to something specific is always extremely difficult. It's one thing to read about a nuclear reaction but a whole different thing to actually set one off.

To implement something you need to have a level of understanding of it that is way deeper than what is required to get a passing grade in your math class. The problem isn't mathematical notation or the programming language (I am quite competent in and have no quarrel with either).

I recently set out to implement Triple Exponential Smoothing (aka Holt-Winters method), but I didn't want to use someone else's code, I wanted to understand the algorithm so that I knew enough to compare various implementations and/or write my own. It took _months_. There is a lot of material on the subject, but it all disagrees ever so slightly on one thing or another, or skips over something essential. I don't think anyone or anything is to blame for this - that's just life.

You can read about it here, btw: [http://grisha.org/blog/2016/01/29/triple- exponential-smoothi...](http://grisha.org/blog/2016/01/29/triple-exponential- smoothing-forecasting/)

<comment><USER5> > There is a lot of material on the subject, but it all disagrees ever so > slightly on one thing or another, or skips over something essential. I don't > think anyone or anything is to blame for this - that's just life.

Isn't this precisely what you said _wasn 't_ the problem? The resources out there all differ slightly on one thing or another. I suppose you mean small semantic differences? I think when I wrote this article originally I intended superficial semantic differences in the math to count as well as notational differences. Doing math for too long has made me think of these as the same thing, since the true underlying insight doesn't depend on either.

<comment><USER4> Yes, every text used its own conventions for variable names, different words to say the same thing, etc, but no, that wasn't a problem (for me at least).

I think the point I was trying to make (not so well) is that perhaps it's not that programmers do not like math, but rather that when you learn to view everything through a programmer's eyes, i.e. "how am I going to actually build this damn thing", you tend to get very wary of abstract or generic approaches which is what scientific books are made of.

For example, if I were given a book with a chapter on Holt-Winters and told that there'd be a quiz later, I could have "mastered" it in a night probably. But when I had to write the code to actually do that, I'd find out that this book does not mention how you come up with initial values, while wikipedia does, wikipedia cites this NIST publication, but who wrote that and is it any good? Then I came across a 2014 paper that says that the forecast can be improved with a slight change in one of the formulas, only when I tried it, my results were worse; no book mentioned how to find best alpha, beta and gamma, because for that you need some kind of a gradient descent or nelder-mead method, and on and on. _That's_ what took months (and it was challenging and fun too), and I think we agree that there isn't a solution to this, it's a dilemma, not a problem. (As someone once said - avoid trying to solve dilemmas and avoid managing problems).

<comment><USER6> I think that's much the same situation as with programming.

If I were given the postgres docs, I could "master" the syntax pretty quickly. But when I had to write code that actually uses postgres in production, I realized that the docs don't carefully explain 3NF, making 3NF fast and user comprehensible, good practices on writing complex queries, etc.

Is that really a math-specific problem?

</comment></comment></comment></comment><comment><USER7> I think a big barrier to understanding math for a lot of people, and definitely for me, is names. Things get named for people instead of what they are: "the Kronecker delta function". You came up with some genius model and proved it, and after all that work you decided that your name and some greek letter are what people need to think of when they hear about it and not anything about what it is? Fuck you Kronecker!

The problem extends beyond mathematics, to essentially everything except programming. And in fairness out of all the fields with shitty names, mathematics has the best excuse: unlike the natural sciences and engineering, math is completely abstract. But programming is also pretty abstract, and yet we don't call stacks Hawk-elbower's double-u structure! No, we notice a semantic similarity between our abstract concept and something more familiar, and we come up with a monosyllabic, five-letter word that conveys a lot of information about the construct to someone completely uninitiated. This both makes it more approachable and easier to learn, serving as a mnemonic device. And once you've internalized it it makes no difference whatsoever.

I hear people whine that naming abstract concepts after concrete objects will only give people a harder time thinking abstractly. Bullshit, I say: have you ever heard of a novice programmer thinking that stacks are only good for representing stacks of paper or dictionaries for looking up definitions of human words.

I understand that naming things nicely in math is _harder_ than in programming, and that in some more cutting-edge branches there might simply be no good analogies to draw, but I don't think it's impossible for more mundane topics. Math professors teach by drawing analogies and thinking of applications all the time. In any case we can do a lot better than "Kronecker's delta function".

<comment><USER8> What would you call the Kronecker delta?

<comment><USER9> "equality operator"

<comment><USER10> Except it is not the equality operator, but a special case of an indicator function (c.f. comment by pash).

</comment></comment></comment></comment><comment><USER11> Just yesterday I found myself reverse engineering checksum algorithms in various domestic bank account systems about the world. [https://github.com/globalcitizen/php- iban/issues/39](https://github.com/globalcitizen/php-iban/issues/39) I went for an explanation of some of the newer ones I hadn't heard of before (Damm) and it was painful. (The code was only a few lines, though!)

While some algorithms had a pre-existing implementation, wiser than rewriting I simply nabbed them and evaluated. Most of the implementations were unreadable/undocumented and of dubious origin, many of the implementations claiming to provide the same algorithm delivered different results. Some of these work for some countries, some for others. No idea where the differences lie and not enough interest to bother finding out.

Then of course there's the origin of these 'standards', ISO, who insist on charging 88CHF for a 5 page document, thus we can't actually read the damn standard. I would up porting a family of algorithms from Java using code generation.

Why we don't yet have a library delivering smackdown-documented functions in arbitrary languages for named algorithms escapes me...

</comment><comment><USER12> "Doing mathematics" and "using math in my code" are so far apart. In my programs that Really Utilize Math, I'm taking advantage of some existing mathematical concept, but I'm not advancing the state of the art in mathematics.

However, the few times I've really wanted to delve into math and really understand the concepts underpinning that formula I'm using, I need a machine to iterate on. My computer isn't ideal because the software that's available is too expensive for 'hobby' use; there's no library that lets me use code in a language I know to manipulate symbols and equations. The TI series of calculators are not ideal because A) they're intentionally crippled (to prevent student cheating) and B) they have a bit of learning curve to them when you don't use them regularly.

Can I get a touch-device app for mathematical symbol/equation manipulation? That'd be ideal.

<comment><USER13> [http://detexify.kirelabs.org/classify.html](http://detexify.kirelabs.org/classify.html) that will yield a TeX symbol which can trivially be transformed into something acceptable into most of the engines (e.g. Sage, Octave, etc) for evaluation. It's not "evaluate definite integral (0,5] for .." but it'll get you half the way there.

PS. Cheating on TI's is trivial. If you're a math professor, adopt a no graphing calculator rule. (You can still cheat on your standard secretary's 20$ calculator, especially in the age of low-power Cortex M0+'s, but it's somewhat less trivial. I'd imagine memorizing 5 or 6 rules on taking the derivative would be easier for someone capable of doing that. (Though if those tools had been around when I was in uni, I might have just done it for the sake of doing it.))

</comment></comment><comment><USER14> Well sure, mathematicians are so nutty about notation, it's amazing any mathematics gets done at all.

Not at all convinced it's a feature. More like an unfortunate side-effect of tradition.

<comment><USER8> I feel a need to stand up for mathematics notation. The main complaint seems to be that it is too minimalist, using the minimal number of symbols (or less) to be unambiguous.

But mathematics notation needs to be extremely succinct because the notation is not just for reading. When you're working something out you often need to write pages and pages of formulae and diagrams by hand. So using a notation more verbose than absolutely necessary would be unnecessarily painful.

Programming is different because you can use a text editor with copy, paste, autocomplete etc. But I imagine that if I wasn't allowed to use any of those then my variable names would shrink down to one letter when I was programming too.

Also, I think that making formulas smaller makes it easier for the eye to see the whole formula together. This makes it easier to quickly understand the meaning of the formula, at least once you've got used to the notation for that particular area of mathematics.

<comment><USER14> Yes, that's the tradition part. I certainly don't object to using whatever notation you like when you're at a whiteboard or scribbling on paper. But math papers are carefully formatted and published using LaTEX, not by taking pictures of whiteboards. This isn't because it's easiest. It's supposedly to benefit the reader, and yet fails in practice.

It seems like coming up with a format that at least allows definitions and usages to be hyperlinked might be pretty nice, instead of a format whose primary benefit is looking pretty when you print it on paper.

<comment><USER15> Go to definition (for symbols or notation) would certainly be helpful in reading a mathematical paper, but how is that related to choosing good notation? Do you have an example of mathematical notation that "fails in practice"? In my experience, mathematical notation becomes established _because_ it works well enough in practice.

<comment><USER14> I don't have any specific example, but I've read that many (most?) math papers are incomprehensible to outsiders, even mathematicians in other fields. I also seem to remember an article about a proof that's gone unchecked for years.

So, from the outside, it seems like there are big problems with communication in the mathematics community, and it's simply accepted that it has to be that way because math is hard.

<comment><USER16> Good notation certainly aids understanding, but math _is_ a vast endeavor where the details matter. In switching from one programming language to another, often a rough understanding of corresponding concepts is enough: how to print has different details in different languages but it's not that different, and the details are easy to look up. On the other hand, just switching from topology to symplectic geometry involves asking entirely different questions -- having different values and pursuing different goals. Maybe it's more like switching from web dev to scientific computing as well as switching languages.

Yes, a lot of mathematical papers are incomprehensible to others in the field. But it's not just because of notation. You wouldn't ask a content marketer who is great at A/B testing to deal with tweaking something in the linux kernel because it's all "computers".

</comment></comment></comment></comment></comment></comment><comment><USER17> There is One Book to rule them all (sorry, could not find a link to the original 1973 edition):

[http://www.amazon.co.uk/Mathematical-Handbook-Scientists- Eng...](http://www.amazon.co.uk/Mathematical-Handbook-Scientists-Engineers- Definitions/dp/0486411478)

</comment><comment><USER18>
        With a program, you can always write test cases and run them to ensure they all pass. If your tests are solid and plentiful, the computer will catch your mistakes and you can go fix them.
        There is no corresponding "proof checker" for mathematics. There is no compiler to tell you that it's nonsensical to construct the set of all sets, or that it's a type error to quotient a set by something that's not an equivalence relation.

This is something that I have been noticing for quite a while. As I got out of high-school and self discovered programming - something that I gradually realized is that programmers have an infrastructure of tooling setup to help themselves. They made their things to help themselves. Compiler compilers (and cross-compilers!) simply blew my mind away.

I joined college after a gap of an year, and as I faced Physics and Mathematics again, that feeling strengthened. I now sometimes think of a very strict compiler for mathematics that catches errors from the math lingua. Could possibly be extended to Physics too. Don't even know if that is feasibly possible, but I find the idea intriguing.

<comment><USER19> > I now sometimes think of a very strict compiler for mathematics that catches > errors from the math lingua.

Is that what Coq ([https://en.wikipedia.org/wiki/Coq](https://en.wikipedia.org/wiki/Coq)) is?

<comment><USER20> Coq is either too immature or too strict. Writing a proof in coq is far too difficult for what it does.

I agree that we need a proof 'compiler' though.

<comment><USER19> Thanks for the insight on this, by the way; I'm more of an applied math person, so I have no firsthand experience of things like computer-verified proofs, Coq, etc.

What do you think are the specific ways in which a proof compiler would be different?

</comment></comment></comment></comment><comment><USER21> I hate how programming is so inconsistent. I mean, in vim I press 'w' to move one word, whereas in Notepad I press Ctrl-Right. They're not even close! I hate it when I use someone else's editor and I have to re-learn all the commands. I wish programmers could standardise this stuff. Maybe there could be a committee whose job it is to standardise editor commands, then we could program in any editor without having to worry about it.

[I'm joking of course, but only half-joking. There's a serious point to be made, which is that mathematics exists completely independently of notation. When mathematicians don't care about notation, it's because they care about the actual mathematics instead. The reason mathematics notation is non- standardised is because standardising it wouldn't really help mathematicians that much. Unfortunately this means that non-mathematicians who use mathematics suffer.]

</comment><comment><USER22> There's "The Hitchhiker's Guide to Calculus" by Spivak, and it's actually a really easy read while giving you an excellent mental model of Calculus.

</comment><comment><USER23> The difference is this: When I write code that has an error, I usually get either a compile or run time error. Not so with math!

<comment><USER24> One can, however, catch some of these with unit tests, in maths just as well as in code. The unit tests for maths are tiny, trivial cases of the main theorem. For instance, if I wrote down Fermat's Last Theorem as "x^n + y^n = z^n has no solutions for n > 1", I would run my unit tests by checking the boundary case n=2, and discovering the easy counterexample of (3,4,5).

</comment></comment><comment><USER25> [http://www.amazon.com/Numerical-Recipes-Scientific- Computing...](http://www.amazon.com/Numerical-Recipes-Scientific-Computing- Second/dp/0521431085/ref=sr_1_4?ie=UTF8&qid=1455840459&sr=8-4&keywords=numerical+recipes)

This or a variation is on almost every physicist's shelf I know...

<comment><USER26> _This or a variation is on almost every physicist 's shelf I know..._

Unfortunately, that is a significant part of the problem.

 _Numerical Recipes_ was an OK book for its time, and certainly a popular one given the limited material available in the field. However, neither the recipes themselves nor the book's general presentation are ideal today, and in many cases, someone interested in implementing robust, efficient algorithms for various mathematical constructs would do better now to read other sources.

For example, substantial linear algebra computations are probably going to use some variation of BLAS/LAPACK today. There is also a lot of background material available about the algorithms used within these libraries and the underlying mathematical foundations, for those who need to implement something a little different or who are simply curious.

In most fields, and excluding those who are actually writing this kind of mathematical library, a programmer will do better to use the tools that are already freely available today rather than trying to implement their own code based on ideas from _Numerical Recipes_.

<comment><USER27> I'm an Engineer I have NR sitting on my desk shelf I refer to it often. A lot of my peers use it as well.

NR is hugely helpful. One example a few years ago I needed to implement technique from a math paper in some C code.

The paper included such classic math terms like "Tridiagonal Matrix" and "Cholesky Decomposition" I'm sure they mean a lot to someone with a math background but baffled me at the time.

It wasn't until I pulled out my handy copy of NR I was able to even slightly wrap my head around just what the hell a "Cholesky Decomposition" even was. Let alone how I would code it. And even then I had to dredge up my old Linear Algebra textbook from Uni to re familiarise myself with basic Matrix operations.

The code I wrote ended up being full of comments like this:

"/* Newton's method to compute positive root f(p)^2 = (u^T)(Q^T)(D^2)Qu and F(dF/dp) = (u^T)(Q^T)(D^2)Q(du/dp) */ "

Which is basically me trying to wrap my head around just what the hell the mathematical paper is talking about whilst writing the algorithm programatically.

Even now opening the .C file some years later it took me a few minutes to make sense of that comment I had to realise that U^T was shorthand for transpose of Matrix U.

Numerical techniques are hugely complicated especially when you don't deal with them every day I have yet to find a book better than NR at breaking them down and presenting them in a way someone like me (with Science/Eng background) can understand them.

I'll gladly take any recommendations you have for updated references. As far as I know NR and the GSL (GNU scientific library) documentation are considered the gold standard by all my peers.

<comment><USER26> It's been a few years since I did serious computational linear algebra, but at that time _Matrix Computations_ by Golub and van Loan seemed to be the standard textbook in the circles I moved in. Rather than trying to provide ready-made production-quality implementations as _Numerical Recipes_ does, it instead concentrates on presenting the relevant mathematics, often in more depth and with more discussion than _NR_ because it's a specialised text, with outlines of the relevant algorithms. For production use it refers to relevant BLAS/LAPACK functions so you know what to look for. If you decide to get a copy, make sure it's at least the third edition, as I think earlier editions referred to predecessor numerical libraries.

BLAS and LAPACK themselves are constantly evolving to add new or better production-quality algorithms as the field develops, and I highly recommend using the functionality they provide instead of trying to reimplement any of the basic algorithms in-house. There are heavily optimised versions of these libraries available for almost every platform you can imagine, their own documentation is pretty good, and often recent research in the field also gets written up as background papers and incorporated fairly quickly (the magic words to search for are "LAPACK working note").

If memory serves, the GSL actually depends on having a BLAS implementation available for some of its functionality, so if you're not limited by the licensing you might already be using the same sort of code under the hood anyway. :-)

</comment></comment></comment></comment><comment><USER28> I read a lot of papers about programming recently, and the weird kind of math people use there is a huge barrier. So, no even math related to programming is readable by programmers!

</comment><comment><USER29> The opinions of what the subscript could mean are very coloured by Python :P Didn't mention the fact that this could be a type, etc

</comment><comment><USER30> I love programming, but never got the hang of Mathematics due to various reasons. Failed a big project as a solo programmer long time ago, when I was 17, because I could not create the necessary mathematical model and no help was available, with everything else completed, infrastructure, web site/interface database etc. Soon afterwards set off on a different career path, but always kept an intimate relationship with programming, doing some both at work and for fun at home.

Recently, after watching R.W. Hamming lectures and reading his book "Art of Doing Science and Engineering" [1] [2] decided that I would try learning mathematics again.

I found it really hard to do. There are so many implicit assumptions, so much ground that is not covered. I was about to give up again, when I found a book [3] by retired computer science professor, who seemed to have written it specifically for me! it really felt that way.

Now I recommend it to almost everyone, especially tech-savvy people. I am no longer afraid of mathematics, with the help of this book, and great advice by Hamming, I am finally "getting it". If you find learning new programming languages easy, but struggle with mathematics, check it out!

The book is not aimed at teaching you mathematics, rather, it teaches you those missing pieces, that almost every other book either assumes you already know, or describes vaguely and badly, gives you context and valuable advice. To me it almost feels like it is teaching you a functional programming language. A strange and very flexible programming language, with its warts and all. You learn the language, and then you can go on to explore programs written in it, algorithms, data structures whatnot.

My ultimate goal is to, at least, learn enough so that I can really understand books by Hamming, especially "Methods of Mathematics Applied to Calculus, Probability, and Statistics" [4]. And, of course, to do interesting projects along the way, now that I am starting to understand some of the research papers.

[1] [http://worrydream.com/refs/Hamming- TheArtOfDoingScienceAndEn...](http://worrydream.com/refs/Hamming- TheArtOfDoingScienceAndEngineering.pdf)

[2] [https://www.youtube.com/watch?v=AD4b-52jtos&list=PL2FF649D0C...](https://www.youtube.com/watch?v=AD4b-52jtos&list=PL2FF649D0C4407B30)

[3] [http://www.amazon.com/The-Language-Mathematics-Utilizing- Pra...](http://www.amazon.com/The-Language-Mathematics-Utilizing- Practice/dp/0470878894)

[4] [http://www.amazon.com/Methods-Mathematics-Calculus- Probabili...](http://www.amazon.com/Methods-Mathematics-Calculus-Probability- Statistics/dp/0486439453)

</comment><comment><USER31> I think this goes beyond programmers and those who want to learn "Mathematics": it can be frustrating to talk with someone who confuses equivalence and implication in everyday conversations; this inability to use proper, basic, logic seems to be too widespread.

I can relate to the funny part on notation. I've been fortunate enough to have an older sister who's Algebraist and I remember when I was in primary school and I'd have a question, she'd explain on paper using different symbols than the ones I was using just to make sure I wasn't a slave of notation.

She insisted I had to think in an "abstract" way (that's precisely the word she used) and not be tied by what the letters are called, and only look at the relations between them and context. This one piece of advice served me well.

It's also funny the author mentions Fermat's Last Theorem because the first time I had heard of it, it was still called "Conjecture de Fermat" and I had read a piece on "Science & Vie" on Andrew Wiles' proof and the fact there still was work to be done to see whether it was correct or not. I was 7 at the time and it seemed strange and amazing how something that can be stated in terms deceivingly simple even I could think I understood, can be so difficult to prove.

I've studied Electronics Engineering in college and we had really cool Maths courses (again, in the spirit of different notation, I noticed a peculiarity: in the U.S., the term "Calculus" is used, whilst we used "Analysis", "Introduction to Mathematical Analysis", "Differential Analysis", etc. Perhaps the influence of French and Soviet Mathematicians who taught at my University. The books are also either Smirnov, Demidovich, Pontryagin, Piskunov, or Dieudonné, so the style taught is kind of different).

I digress. For me, it doesn't matter. I love Mathematics. Not the way many people say they love it. It is something on my top list of things to do and I am self studying currently. Why? First, emotional reason: in retrospect, the happiest years of college where in the "Common Core" (two years of core knowledge (mainly Physics and Maths) every Engineering student goes through before choosing a specialty (except Computer Science students, they only do freshman and then CS). Maths (and Physics) bring me happiness in a way that is hard to describe.

The second reason is practical: when my Maths skills were somewhat sharp, I had no problem understanding other things, because I had the necessary tools to strip down the things and go at it where it was called for (for example, modulation with a transistor and using Taylor series for the current). I then turned utterly incompetent and let everything rust and I'm paying the price right now. I have graduated but the blade is dull and I can't live with myself like that. I've come a long way, especially because I was clueless in time management and focusing my efforts and getting things done and I had to learn how to mitigate my proclivity to be all over the place. The one sure thing is that I'll never let go. I may not become a great mathematician, but every day I am less incompetent than the day before it. The point is, as in everything else: people don't learn because they lack the drive to do so, face the reality of sucking, learning the basics and fighting that embarrassing feeling of the inner voice that says "shouldn't you already know that stuff?".

<comment><USER27> I studied Engineering like you. I graduated six years ago and the rust has definitely set in for me. At my university our Math courses were split into three streams, Statistics, Calculus and Linear Algebra. coincidently I had a Russian Lecturer as well.

It was after those core subjects where I fell in love - Thermodynamics and Fluid Dynamics was the 'aha' moment for me. It made all that core stuff make sense when I finally got a sense of the applications. Bernoulli, Euler, Fermi, Dirac - The shoulders of giants...

<comment><USER31> Yeah. Whenever I had trouble with a topic, it was because my core knowledge was weak and rusty. I was in Instrumentation and Control. The stuff in Control Theory was all Pontryagin, Lyapunov, Bellman, Evans, etc. We were fortunate enough to have a very good Professor who taught a course that is state of the art (continuous control in 3rd year, and discrete control systems in 4th year. Optimal control and RST controllers).

For applying the stuff, it's sort of easy. But I know that I'll never appreciate the beauty of it until I understand the concept of stability from a "Calculus" stand point and how it relates to systems.

Anyhow, I'm also acquiring Russian. Many Russian books have been translated but a lot have not.

</comment></comment></comment><comment><USER32> A Hitchhiker's Guide to Mathematics would probably just be the number 42 over and over again.

<comment><USER33> With alternating pages printed "What is 6 x 7"

<comment><USER32> No! Remember the book:

      "What do you get if you multiply six by nine?"

<comment><USER34> And because everything is base 10, 6 times 9 is 42. In base "A", it's 54, but we're using base 10, not base "A"

<comment><USER35> All bases are base 10</comment></comment></comment></comment></comment>


Postgres gets support for upsert - <AUTHOR> http://git.postgresql.org/gitweb/?p=postgresql.git;a=commit;h=168d5805e4c08bed7b95d351bf097cff7c07dd65 <comment><USER2> I think I squealed with delight when I read this title. This is really a great addition, and will simplify so much of my code once I'm able to upgrade.

I try not to think too deeply about all the DB internals (some people are much smarter than me about this stuff), but clearly there are a lot of things that need maintained internally to make this correct, so kudos to all the people who contributed to this.

<comment><USER3> A big +1 to this. Peter Geoghegan is the primary author and has been championing it for over 2 Postgres releases now. I know he's especially excited to see it land.

Also it couldn't have been possible without the support of Heikki and Andres (who actually submitted this to HN as his first ever submission).

<comment><AUTHOR> Hah, where from do you know that I don't have my sock puppet accounts ;)

</comment></comment></comment><comment><USER4> This is actually huge. A common problem that arises when you write applications is you want to INSERT if key does not exist else UPDATE. The right way of doing this without an upsert is using a transaction. However this will make life easier as you can do it directly in one SQL statement.

<comment><USER5> Just "using a transaction" is insufficient. You must be prepared to handle the case that neither the INSERT nor the UPDATE succeeds (with READ COMMITTED isolation), or that the transaction fails (with REPEATABLE READ isolation or better), by repeating the transaction. And if latency is at all a concern to you, you must wrap this all in a stored procedure to avoid the necessary round-trip-time between the commands.

Hence this is more than just saving typing a couple lines -- this saves writing entire stupid loops to do what is conceptually a simple (and very common) operation.

Postgres gets better and better.

<comment><USER6> ...and then you additionally have to verify that the specific reason both the update and the insert failed was due to concurrency, or you can make the mistake I did a couple years ago where I had transactions spinning against my database for weeks on end due to an unrelated constraint failure I had failed to notice with some transactions that I kept retrying in a loop as part of my hacked together upsert implementation (which is why my user id numbers jumped from ~8m to ~460m... it is a meaningless mistake, but one I am reminded of every day due to that "46" staring at me :/).

<comment><USER7> Is my conclusion correct that this implies sequence increments are run outside of the transaction?

<comment><USER8> Sequences in Postgres are updated even if a transaction is rolled back. It's pretty easy to see why that's important, especially when dealing with concurrent transactions.

You can't give out the same number to multiple transactions without causing a bunch of unique constraint violations. And you can't retroactively decrease a sequence number without affecting all the other transactions currently executing. You could in theory go back and plug in the gaps in a sequence, but it's more important to guarantee that a single thread of execution will always see increasing numbers than it is to guarantee that all numbers will eventually be handed out to a successful transaction.

</comment></comment></comment></comment></comment><comment><USER9> I love that it's there now. I've been waiting for it for a long time, but one thing I don't get is... why does every single implementation have to have their own slightly different syntax?

pgsql -> on conflict

mysql -> replace / on duplicate

oracle -> merge

mssql -> merge

sqlite -> insert or replace

firebird -> merge / update or insert

<comment><AUTHOR> MERGE is actually a separate feature. We're still considering implementing it. But it's syntax is too cumbersome for many usages. There's also some unrelated complexity - so implementing merge support does get easier by the infrastructure in here, but a fair amount of work remains.

<comment><USER10> It's great that you guys implemented UPSERT but I hope MERGE support won't be considered as insignificant because of this feature.

<comment><AUTHOR> I'm pretty sure it's not (going to be) considered insignificant.

Unfortunately that does not equate to resources (i.e. time by somebody sufficiently crazy^Wdetermined) for implementing it being available. A large part of postgres development is driven by individuals. Some of it on company time, but usually not most of it.

</comment></comment></comment></comment><comment><USER11> Yet another reason why I absolutely love Postgres. It might have taken four years, but we finally got what many of us have been asking for. Nothing ruined my day more than having to write insert-update loops, I am beyond ecstatic for this.

<comment><USER12> You love Postgres because it takes four years to get requested features?

<comment><USER5> Because the devs pay attention, and take the time to do things right. See [https://wiki.postgresql.org/wiki/UPSERT](https://wiki.postgresql.org/wiki/UPSERT) for a glimpse into the design that needed to go behind this.

I feel a lot of Postgres fans (myself included) put their "money" on Postgres circa late version 7 or early version 8, back when MySQL was the more featureful and performant of the two, while Postgres had the reputation for being more, shall we say, _robust_. (Remember, those were the days before InnoDB was the default in MySQL.)

The payout for investment in Postgres the past few years has been substantial -- native replication, true serializable transactions, foreign data wrappers, index-only scans, native JSON support, updatable views, and materialized views are among the features added to Postgres in the last 4 years. It's matured from being an "entry-level" RDBMS with few features that all work reliably, to a much more enterprise-friendly RDBMS with many features that _still_ work reliably.

<comment><USER13> _> late version 7 or early version 8, back when MySQL was the more featureful and performant of the two_

MySQL was never more featureful. The reason I started migrating back in the 7.1 time frame (when TOAST tables were added and you could finally store more than 8K of text in a TEXT column) was the lack of sub-selects in MySQL.

Even aside of that, Postgres was far ahead when considering basic SQL support: stored procedures, views, subselects, check constraints, triggers, actually enforcing foreign key constraints and so on.

It was significantly slower than MySQL, but it also scaled much better under load. Back then, when you had low load, MySQL would be about twice as fast as PostgreSQL but then as the load increases, MySQL's performance would drop sharply and Postgres would stay consistent.

By now, MySQL has mostly caught up feature-wise, but there's still stuff left that Postgres just does better. Also, even plain ideological reasons (community project vs. oracle open-core project) would want me to stay with postgres.

I also have anecdotal evidence that MySQL still has serious issues in the robustness department which I've yet to see with postgres.

<comment><USER14> MariaDB, WebScaleSQL, Percona are just three MySQL forks that are completely open source in every single way. Likewise MySQL-Server is GPL so it definitely should be considered as open source.

And if MySQL wasn't robust then YouTube, Facebook, Twitter, Alibaba, LinkedIn etc wouldn't be using it for core parts of their infrastructure. It's definitely robust.

No doubt that PostgreSQL is better at MySQL in many areas though and probably could do with a self contained, single download PostgreSQL Cluster edition.

<comment><USER15> > And if MySQL wasn't robust then YouTube, Facebook, Twitter, Alibaba, > LinkedIn etc wouldn't be using it for core parts of their infrastructure. > It's definitely robust.

Popularity is not an argument for quality. See: crocs, Justin Bieber, PHP.

Those companies you mentioned, like many others, probably use it because they're locked in that technology, not because it's a superior one. Just like banks still use COBOL.

If you're interested on knowing why using MySQL isn't a good idea, there's this (highly opinionated, obviously) post about it: [http://grimoire.ca/mysql/choose-something- else](http://grimoire.ca/mysql/choose-something-else)

<comment><USER14> I never said popularity was an argument for quality. You did.

What I am saying is that if MySQL wasn't robust then those companies simply wouldn't be using it. Since at their scale any bug or weakness will manifest at a level far greater than say at a startup. And they have the skills, time and money to choose any technology they wan't so I don't buy your argument that they are "locked in". Some like Facebook and LinkedIn have even created their own databases.

And I never said MySQL was a good idea. You did. I am saying that to claim it is not robust flies in the face of available evidence.

<comment><USER16> " never said popularity was an argument for quality. You did."

I'd tend to think your statement:

"And if MySQL wasn't robust then YouTube, Facebook, Twitter, Alibaba, LinkedIn etc wouldn't be using it for core parts of their infrastructure. It's definitely robust."

..points out that many (popular) sites use it, and a prerequisite is robustness, which thesaurus-wise, sounds a lot like quality.

I get that

a) you didn't actually say it, and

b) you could mean something much more specific, such as "companies with many highly-starred, complex open source projects which have also re-written major parts of their tech stack, but chose to leave mysql in place".

I know thats a bit more verbose, but it just seems to obviously close to the other author's interpretation with the ambiguity of the statement.

<comment><USER17> > many (popular) sites use it

threeseed's argument is not

        this handful of popular companies use it, therefore it is good

but rather

        this handful of companies, each of which is considered by some/many to be 'excellent' or 'top', use it, therefore it is good

</comment></comment></comment></comment></comment></comment></comment></comment></comment><comment><USER18> The documentation covering the new feature has now built: [http://www.postgresql.org/docs/devel/static/sql- insert.html](http://www.postgresql.org/docs/devel/static/sql-insert.html)

<comment><USER19> Postgres's documentation is one of the best I've seen. Thanks to everybody working on it!

</comment></comment><comment><USER20> For the backstory on why this has taken so long: [http://www.depesz.com/2012/06/10/why-is-upsert-so- complicate...](http://www.depesz.com/2012/06/10/why-is-upsert-so-complicated/)

</comment><comment><USER21> I know it doesn't matter to some developers out there, but for me, the only thing left is the ability to add a column after another one. Even if it's only logical order, it's easier for me to read a table structure when fields are ordered nicely.

<comment><USER22> Unsure what you mean. A subset of columns can be selected in any order, across tables. Were you referring to the select order used by "*"?

<comment><USER23> Yes, I think it's about changing the default presentation order. That could be a feature of the client, psql, pgcli, pgadmin3, but it won't be portable across them unless it's some metadata stored inside the db. Is there any db with that feature?

Edit: munro answered that. PostgreSQL does it but it's expensive because it has to rewrite the table.

<comment><USER21> Yes, MySQL: ALTER TABLE t1 ADD COLUMN `count` SMALLINT(6) NOT NULL AFTER `lastname`

</comment></comment></comment></comment><comment><USER24> So excited about upserts in Postgres!

This was the most requested feature in PostgreSQL: [https://postgresql.uservoice.com/forums/21853-general](https://postgresql.uservoice.com/forums/21853-general)

</comment><comment><USER25> Which version of Postgres will this land in?

<comment><USER26> 9.5, that is, upcoming!

<comment><USER27> Which will hopefully drop in September.

</comment></comment></comment><comment><USER28> I am extremely happy to hear about this. We currently insert, catch the exception and update if necessary. That can lead us to race conditions in concurrent transactions. Thank you very much for your excellent work on Postgres. It continues to be an amazing product to work with.

</comment><comment><USER29> I want to give a giant digital hug to all PG committers who made this happen.

</comment><comment><USER30> Who decided the syntax? Why not use MERGE ... INTO ... USING ... ON ... WHEN like Oracle and MSSQL and apparently ANSI SQL [1]? I'm not aware of ON CONFLICT being standard.

[1] [http://en.wikipedia.org/wiki/Merge_(SQL)](http://en.wikipedia.org/wiki/Merge_\(SQL\))

<comment><USER31> There's some extended discussion of the issue on the wiki[1]. A future full MERGE implementation might have slightly different semantics or performance characteristics than ON CONFLICT (which might also differ from other DBMS).

[1] [https://wiki.postgresql.org/wiki/UPSERT#SQL_MERGE_syntax](https://wiki.postgresql.org/wiki/UPSERT#SQL_MERGE_syntax)

</comment></comment><comment><USER32> Does the description seems like it's a two-pass operation to anyone else? Can anyone with more detailed knowledge tell why can't this be a single-pass "INSERT in a unique index, UPDATE if it fails" operation?

<comment><AUTHOR> To understand the problem one first has to know that for better or worse postgres does not use index organized tables. Additionally this ought to work not only on a primary key but also on additional unique constraints. Including partial unique indexes, potentially over expressions (say lower(username) or something).

That makes some approaches harder.

The bigger problem is that 'update if fails' is easier than it sounds. When you find a conflicting row in the index, that row's creating transaction might not yet have committed. So you need to first wait for it to commit, and then retry. But that implies that the transaction could actually roll back or delete the row again. And suddenly there's not actually a row to UPDATE. Thus you need to retry, right?

There's also some issues around avoiding deadlocks. If you "naively" just acquire a 'value lock' on the to-be-inserted row, you can easily get into deadlock territory if more than one row is inserted in one transaction. Such value locks have to be released after an attempt to update. Which then necessitates a looping attempt...

Does that start to explain the problems a bit?

</comment></comment><comment><USER33> I remember about 10 years ago using UPSERT in a Teradata database to do delta- diff updating of tables.

Basically, I was updating a table from an external file, and I wanted to add what was new, leave what was the same, and delete any records no longer in the external table. Using UPSERT to insert new rows, update a timestamp on existing rows, and then afterwards delete any rows who's timestamp was before the UPSERT operation. Because of the nature of the file, doing an external patch/diff approach was difficult, but the UPSERT operation worked like a breeze!

</comment><comment><USER34> Damnit, I just wrote a sweet gem that handles this... ah, okay, but it requires an index constraint, whereas I needed to merge on arbitrary conditions. So I didn't totally waste my time.

Way to go Postgres team!

<comment><USER35> Is the gem open source? I checked your github and didn't see it, I think people would still find it useful!

</comment></comment><comment><USER36> I've been looking forward to this for ages! The solution we use at the moment for upsert is a create or replace rule on the table but it isn't great as you need to remember to update the rule whenever you change the table schema...

      CREATE OR REPLACE RULE "replace_row" AS
      ON INSERT TO my_table
      WHERE EXISTS(SELECT 1 FROM my_table WHERE id=NEW.id)
      DO INSTEAD
      (UPDATE my_table SET
      col1=NEW.col1,
      col2=NEW.col2,
      col3=NEW.col3,
      WHERE id=NEW.id);

<comment><AUTHOR> That unfortunately is not safe on several fronts.

For one, the WHERE EXISTS() will not see concurrent insertions by transactions that are still in progress, which means you'll still get constraint violation errors.

For another, rules generally have very surprising behaviour. E.g. with this RULE you'll get into trouble if "id" isn't passed in as a explicit value, but uses a non-deterministic DEFAULT or directly passed in expression. Every reference to NEW.id will not be replaced by the result of that expression, but rather with the expression itself. For the common serial id column (aka autoincrement) you'll have a separate nextval() call in each reference. Which can make this explode in pretty damn confusing ways.

<comment><USER36> Unfortunately, there's no good all round solution for upsert in current postgres versions. For our use case (one logger daemon feeding results off a message queue in to a table in an idempotent way) this approach was the simplest and worked the best.

<comment><AUTHOR> Can't you just do the "usual" looping pattern around a subtransaction? If events are inserted by a single callsite that shouldn't prove to onerous?

C.f. [http://www.postgresql.org/docs/9.4/static/plpgsql-control- st...](http://www.postgresql.org/docs/9.4/static/plpgsql-control- structures.html)

<comment><USER36> Could have done but coming from a mysql background (i.e. where replace statement is supported), just crafting a simple replace rule was easier/faster to grok and implement. Also helped that the table schema was very unlikely to change much in the future - so we didn't have to worry too much about remembering to keep the rule up to date (and the fact the id column was defined as VARCHAR NOT NULL and had no default assigned to it).

</comment></comment></comment></comment></comment><comment><USER37> Does anyone know when this is expected to be released?

<comment><USER27> New versions of Postgres usually ship in September. Except 9.4, which slipped to December.

</comment></comment><comment><USER38> As someone that's been using PostgreSQL way before it was cool (since 2005), I'm super happy to see this. Previously I had to do EXISTS, IF FOUND UPDATE ELSE INSERT. Now I can just do upsert, so nice.

Very thankful to all the hard working Postgres devs. PostgreSQL was really good when I first started using it, and just keeps getting better. Glad to see the general community finally getting on the PostgreSQL bandwagon.

Now let's see if I can get more people to use internal PostgreSQL functions instead of constantly writing statements directly in their code.

</comment><comment><USER39> This reminds me of a common idiom in the Pick-style databases that I've worked on...

        READU REC FROM FILE,KEY ELSE
            REC<1> = 'stuff'
            * Set other fields as appropriate for a brand new record - this is the 'insert'
        END
        REC<1> = 'blah'
        * Set other fields as appropriate for an update
        WRITE BLAH TO FILE,KEY ;* Write record and release locks

This can, of course, be wrapped in a transaction.

</comment><comment><USER40> Seriously, IIRC the wiki page for this feature was like 5 years old. I'll have some statements I'll be very glad to update as soon as this is released.

I was so surprised when I ran into the lack of this feature as I'd had it on both MSSQL and MySQL.

</comment><comment><USER41> Upserts are generally useful tools for applications that run in active-active multi-datacenter configurations. Have you guys explored the implications of this feature for such deployments on top of Postgres yet?

</comment><comment><USER42> I come from a SQL Server background and recently jumped over to a Postgres shop. It seems like I came in at a really exciting time.

I was really missing SQL Server's powerful MERGE statement, so this is welcome news.

</comment><comment><USER43> So to simplify, what I understand is

1\. INSERT .. ON CONFLICT is an implementation UPSERT technique

2\. MERGE is something meant to be different from UPSERT but has UPSERT feature?

<comment><USER44> Basically true except for many implementations of MERGE not being concurrency safe, which means you would have to loop anyway on unique violation making it no better than the CTE based techniques that already exist in 9.4.

</comment></comment><comment><USER45> For anyone who has ever had to write scripts to import incremental batch data, CSV, etc this seems huge.

</comment><comment><USER46> When is Postgres going to get built in fault tolerant multi-master support in any form?

</comment><comment><USER47> I found engineering around the lack of this feature very _upserting_.

I'll get my coat.

</comment><comment><USER48> Does it handle locking (aka no deadlocks) correctly?

<comment><USER26> There is a section in UPSERT wiki about this - [https://wiki.postgresql.org/wiki/Value_locking#.22Unprincipl...](https://wiki.postgresql.org/wiki/Value_locking#.22Unprincipled_Deadlocking.22_and_value_locking)

EDIT: rather the whole "Value Locking" page - [https://wiki.postgresql.org/wiki/Value_locking](https://wiki.postgresql.org/wiki/Value_locking)

</comment></comment><comment><USER49> I think an UPSERT keyword would be more clear.

</comment><comment><USER50> Oh happy day. PostgreSQL is the best.

</comment><comment><USER51> Say goodbye to CRUD. Because 'create' and 'update' are syntactically and semantically the same, right?

</comment><comment><USER52> its about F __ __*G time

</comment><comment><USER53> Yes! it's about bloody time.

</comment><comment><USER54> Great! I'm just like it!

</comment><comment><USER43> Slightly OT, why do we see these errors with Oracle's MERGE statements?

ORA-01779: cannot modify a column which maps to a non key-preserved table

ORA-30926: unable to get a stable set of rows in the source tables

<comment><USER55> Asking a support question on a feature announcement post for a completely unrelated product is EXTREMELY off topic.

<comment><USER43> It was not meant as a support question. It was in the context of MERGE/UPSERT was I asking. I wanted to know if similar thing happens in PG.

</comment></comment></comment><comment><USER56> In my crazy-ass opinion, if you need upserts, there are problems with your application logic and problems with your database.

Get off my lawn, etc.

<comment><USER57> Why bother posting if you're not going to explain yourself? It's 2015, nobody's going to be impressed by mere contrarianism.</comment></comment>


Snowden: US hacks Chinese mobile phone companies, steals SMS data - <AUTHOR> http://www.scmp.com/news/china/article/1266821/us-hacks-chinese-mobile-phone-companies-steals-sms-data-edward-snowden <comment><USER2> I don't understand why everyone thinks it's terrible if a government surveils its own citizens but it's totally a-ok if it does foreigners. Because if you think about the reflexivity of it all, that means that it's a-ok if China surveils all American citizens and America all Chinese citizens. The only way that makes sense is if you think one government (America or China, depending on who you are) is privileged to violate the privacy rights of citizens of the other.

I would hope that Hacker News would be more cosmopolitan. So much of our work, especially, involves interacting with foreigners. If our government thinks that their communications with us deserve no protections because they're some suspicious other, and their government thinks that our communications with them deserve no protections because we're some suspicious other... then no one ends up with anything.

ETA: And when you think about it, that principle creates a giant loophole: country X surveils country Y's populace, country Y surveils country X's populace, and they have a mutual agreement to share data about each other's dissidents. Just goes to show that if you undermine a universal right for even one person, that right disappears.

<comment><USER3> The reason people consider it different is that they realise the world is a complex place with complex actors. In an ideal world, no country would spy on another. But they do. How could we stop China from doing so? We can't. So we spy back.

However, what our democratically elected government does to us is very much within our own power to change, and we have the ability to hold them to account.

<comment><USER4> If you can't stop China from spying on you, how is "spying back" any more justified than ANY OTHER reaction? After all, it does not change anything.

What _could_ change something, if one were to actually think freely, would be to have no secrets. Then suddenly what was stealing, is now just someone wasting their own time. BOOM. But of course, you can't have slavery and sitting fat on doing nothing without secrets, so that's not within the framework of allowed options.

<comment><USER5> I'd be very surprised if any government could survive without holding secrets. Lets take something innocuous--medical records. Surely you'd have to keep those secret. And surely a foreign power would want to access those secrets as well, for blackmail if nothing else.

But even if a government tried holding no secrets, it would be at an information disadvantage compared to more secretive rivals. For instance, most people are unsatisfied with the notion of abandoning their own privacy (which is a personal form of holding secrets) as a defense against government invasions of privacy, yet this is the same solution you propose that governments attempt against rival governments.

<comment><USER4> _Lets take something innocuous--medical records._

Those are secrets of people, not of nations.

 _But even if a government tried holding no secrets, it would be at an information disadvantage compared to more secretive rivals. For instance, most people are unsatisfied with the notion of abandoning their own privacy (which is a personal form of holding secrets) as a defense against government invasions of privacy, yet this is the same solution you propose that governments attempt against rival governments._

There is a huge difference between organizations and people, I never proposed such a thing when it comes to people vs. nations.

<comment><USER5> > Those are secrets of people, not of nations.

Yes, now please bother to read the next two sentences of my original post and then think about it, because I'm not in the mood to repeat myself.

</comment></comment></comment></comment></comment></comment><comment><USER6> Snowden's choice of Hong Kong as a hiding spot becomes more and more interesting. I suspect the timing of this particular revelation is timed deliberately just after the US files charges against him seeking his arrest by Hong Kong authorities.

A recent article suggested that China was already inclined to "solve" this problem (from a diplomatic and political standpoint) by doing what it does best: simply dragging its feet. This seems incredibly easy to do when the Hong Kong legal system is inclined to move slowly anyway, any extradition will go through a number of appeals and the process for applying for asylum is being revamped putting all such cases on hold (not that Snowden has applied for asylum yet).

It is an somewhere between widely suspected and an open secret that China engages in concerted intelligence efforts against the US government and US corporations. Many cyberattacks originate in China (and there is strong evidence that at least some are state-sponsored). And China is widely believed to have stolen nuclear secrets [1].

But this revelation goes the other way. I really can't predict how China will take this. I suspect they'll be more disinclined to hand Snowden over (or at least do it in any kind of timely fashion). To paraphrase Ned Flanders "We've tried nothing and we're all out of ideas!"

Who knew in 2008 that during the Obama administration it may get to the point of people wishing for the good ol' days of George W. Bush? Well maybe not that far but it's really not that far off. The war for intellectual property, Federal prosecutorial overreach (eg the Y12 "terrorists", Aaron Swartz), the relentless pursuit of whistleblowers and the end-run around the Fourth Amendment are simply stunning, particularly from an allegedly Democratic administration.

[1]: [http://www.nytimes.com/1999/03/06/world/breach-los-alamos- sp...](http://www.nytimes.com/1999/03/06/world/breach-los-alamos-special- report-china-stole-nuclear-secrets-for-bombs-us-aides.html?pagewanted=all)

<comment><USER7> _> Who knew in 2008 that during the Obama administration it may get to the point of people wishing for the good ol' days of George W. Bush? _

I only wish that Snowden would have been in a position to leak this stuff about a week before Obama was reelected. At least then voters would have been able to make an informed decision. Romney may have been equally as evil as Obama in terms of surveillance ambitions and disregard for the constitution, but the difference is that part of Obama's election pitch was that he was going to get rid of these types of programs, while Romney never said anything of the sort. I don't like the fact that a man who clearly lied about his fundamental political views just to get elected is sitting in the Oval Office right now. Bush, and all Presidents before him, may have lied about many things, but at least you knew where most of them stood politically.

<comment><USER3> _I don 't like people that would lie about their fundamental political views just to get elected sitting in he Oval Office._

A more charitable reading is that Obama didn't lie, he was just an idealist and changed position when he was confronted with the realities of being President.

That's not to defend it, but I don't think Obama knowingly and cynically lied his way into office.

<comment><USER8> I'm fiercely opposed to this newly unveiled surveillance state that US of A is... and yet I find myself not blaming Obama much for this.

I think people have misinformed notions about how much power the president really has. Obama's stated core focus was and has been on improving the situation for the poor/middle class. That's a big task on its own, I don't think you can expect one president to take on and dominate such variety of large tasks.

But now that things are in the open, support a candidate who in clear terms promises challenging these NSA programs for the next election cycle. Yes, Obama said he would do this in his own campaign trail, but it was unclear probably to him what he was up against exactly and how difficult it would have been to resist it alone. My optimistic take is now that we all know to a much greater degree what's going on, we're in better position to do something about it.

<comment><USER9> And yet he lies to cover it up. If he really wanted to oppose it, he could simply expose what's happening on his own. He could issue executive orders shutting it down. The DOJ, FBI and NSA are all part of the Executive Branch, after all.

<comment><USER10> At the very least he can explain _why_ he changed his mind. I get it. I just can't trust him if he doesn't come as clean as he can about his reversal on Gitmo and surveillance and whatnot. I totally understand getting into office, seeing the real data, and saying, "I can't be the one that pulls the plug and causes people to die."

But he owes it to us to at least tell us that.

<comment><USER11> How has he reversed his position on Gitmo?

<comment><USER12> The Associated Press's Elizabeth White reports on a speech by Presidential hopeful Barack Obama, then a junior Illinois Senator, to a crowd in Texas. "We're going to close Guantanamo. And we're going to restore habeas corpus," Obama says. "We're going to lead by example—not just by word but by deed. That's our vision for the future." [0]

[0]: [http://m.theatlanticwire.com/global/2013/01/obama-closing- gu...](http://m.theatlanticwire.com/global/2013/01/obama-closing-guantanamo- timeline/61509/)

<comment><USER11> And then there's this, from May 1st _this year_ : [http://www.usatoday.com/story/news/politics/2013/04/30/obama...](http://www.usatoday.com/story/news/politics/2013/04/30/obama- news-conference-100-days/2123391/)

</comment></comment></comment></comment></comment></comment></comment></comment></comment><comment><USER13> Not saying China is innocent, but this is why stuff like this needs to be transparent and the public needs to know about it, and if the government isn't willing to make it public, then whistleblowers like Manning and Snowden need to make the public aware of it, no matter what "laws" they have to break to do it. But then it's the public's _responsibility_ to protect them against the government, for doing them that service.

Because you have _no idea_ what the government is doing in your name, and what kind of conflicts they are creating, and then "all of the sudden" you end up with another war on your hands, and the US government propaganda machine tells you it's _their_ fault and they are the aggressor against US, when in fact it could be the opposite, and the attack may simply be retaliation for USA's own actions.

<comment><USER14> I agree. Even if this sort of thing happens behind the scenes, there are repercussions that will come back to affect people and businesses, even those without direct ties to China.

</comment></comment><comment><USER15> As someone who, five years ago, tried to get commercial bulk SMS connectivity out of these same main Chinese mobile carriers (China Mobile and China Unicom), and watched them largely quash spammy SMS broadcasting with automatic SMSC service suspension after a certain threshold outbound rate, this is interesting. The guys I talked to inside of these carriers led me to believe their SMSCs were just a Linux box.

Also, SMSCs appear to be provincial entities rather than national ones, so these compromises are likely only in a subset of cities. (Further evidenced online, eg. [http://www.smsclist.com/downloads/default.txt](http://www.smsclist.com/downloads/default.txt))

Note also that this article is poorly concluded: _US President Barack Obama says the NSA is not listening in on phone calls or reading emails unless legal requirements have been satisfied._ That's apparently only for US citizens inside the US, IIRC.

<comment><USER16> And the NSA's explicit mission is the surveillance of foreign powers to the benefit of the US.

This is where Snowden is sinking his message because literally nothing they do in this arena is illegal in the US, and confirming it or giving out details definitely is undermining US interests by the trust placed in him by the NSA to start with.

<comment><USER17> Agreed. His first little bit of knowledge about PRISM and what have you was very noble and a worthwhile cause. This information is just straight up harmful, and getting more into the "traitor" category of leaking information.

Also, people will complain about what the US is doing, but sure, go ahead and tell yourself that any other country isn't doing the same. China, N. Korea, Iran, UK, France, Israel, Australia, Germany, Russia, come on, it's a spy vs. spy world out there, and information is KING.... This leak of information slightly undermines the US bargaining power.

<comment><USER18> Why is it harmful? Are you naive enough to think that the Chinese gov't is learning anything new? It's only "harmful" from a "US is such a fine upstanding freedom loving country" public relations standpoint.

<comment><USER17> >> public relations standpoint.

^This is exactly what I meant when I said "This leak of information slightly undermines the US bargaining power." Politics and bargaining are a very sensitive game, and a very powerful game.

<comment><USER18> "This leak of information slightly undermines the US bargaining power."

That's hardly treasonous. Inconvenient, as it should be. If it helps, consider what our gov't has been using that bargaining power for lately.

<comment><USER19> exactly, in the US treason carries a very specific definition.

"Treason against the United States, shall consist only in levying War against them, or in adhering to their Enemies, giving them Aid and Comfort."

If the US decides to characterize Snowden's actions as treason, that means they are also CHOOSING to characterize civilians of, say, China, as enemies.

</comment></comment></comment></comment></comment></comment></comment><comment><AUTHOR> This story was published in HK only 5 minutes before this posting onto HN, by the only other journalist in direct contact with Snowden himself - Lana Lam.

SCMP is a top HK newspaper, and along with the Guardian, the primary publishing source for direct news from Snowden himself.

<comment><USER20> The only mention of SCMP being a "top newspaper" that I could find is in reference to it being the most profitable newspaper (per reader) in the world.

However, according to Wikipedia, its independence from the Chinese government influence is suspect.

[http://en.wikipedia.org/wiki/South_China_Morning_Post#Editor...](http://en.wikipedia.org/wiki/South_China_Morning_Post#Editorial)

<comment><AUTHOR> Quite the opposite - SCMP is renowned for publishing freedom articles against the CCP party line, which is one of the reason Snowden chose SCMP to publish to.

<comment><USER20> Interesting. I won't pretend to understand Snowden's motives, but the SCMP Wikipedia page does bring into question SCMP's independence from China, so I would appreciate it if you could comment on the issues noted on the wikipedia page.

</comment></comment></comment></comment><comment><USER21> "And the former National Security Agency contractor claims he has the evidence to prove it."

Well... where is it?

<comment><USER22> If your evidence is weak, I think it's better to leak a potentially controversial allegation well in advance of the evidence. Maybe it's conjecture from a bullet point in a PowerPoint slide?

</comment></comment><comment><USER23> ARGHHH....WHY WOULD HE DO THIS?

All he is doing is muddying the waters.

How can I claim to defend what he has done, if he is giving sensitive intelligence data to the CHINESE!!!!

He shouldn't muddy the waters. Just keep it focused on how the USGov't is taking away US civil liberties and privacy.

I mean...I understand why he is doing this - self preservation - but now he is looking more like a "spy" than a "whistleblower".

</comment><comment><USER24> Oof. We're moving quickly out of whistleblower territory and into actual espionage.

</comment><comment><USER25> This is all a bit reminiscent of Frederick Forsyth's "Fourth Protocol". A few words about the plot: the book is set in the early years of Margaret Thatcher's tenure as Prime Minister, in the time of the Reagan Rearmament. Elements in the Politburo devise a secret plan to explode an atomic bomb near RAF Bentwaters, blame this on a malfunction of American nuclear ordnance and cause a leftist government to be elected and Britain to fall into the USSR's sphere of influence. This plan was to be kept secret even from the KGB.

Both MI5 and KGB get wind of this, and they work in concert to stop it _while they remain enemies_ , because either side feels that if it succeeded they would open Pandora's box and have no institutional knowledge to deal with this new world.

It is my sincere hope that the world's intelligence agencies have the institutional knowledge to deal with the widespread knowledge on spying on its own citizens.

</comment><comment><USER26> The great firewall of China is seeming less totalitarian and more pragmatic. The US at least has to hack to get chinese SMS messages; allowing its users visit Facebook/Gmail/etc is like giving data away.

</comment><comment><USER3> Excuse me if I am a little dubious of the South China Morning Post. According to the article, he gave them an exclusive interview 10 days ago that they are finally publishing now. Interesting that Snowden chose to completely ignore his contacts at the Guardian etc. for this.

Are there any other sources to back this up? Or evidence?

<comment><USER27> The SCMP is the main English language newspaper in HK. It's been around for years and has reporters on the ground.

So what's a more "reliable" source? CNN? HuffingtonPost? Buzzfeed?

The Guardian is focusing on stories related to the UK, while the SCMP is focusing on stories related to HK and China.

<comment><USER3> If Snowden granted them an interview ten days ago they've done a pretty bad job handling an internationally significant story. The one sole piece of evidence is this quote attributed to Snowden:

"The NSA does all kinds of things like hack Chinese cell phone companies to steal all of your SMS data."

No actual evidence this time. Just a quote from a guy no one can call to verify.

<comment><USER18> Which is why we should mostly ignore it. He's using the press to curry favor with HK residents. Nothing shocking about that.

</comment></comment></comment></comment><comment><USER28> The USA use electrnoic surveilance on China... China uses electronic surveilance on the USA... its the nature of things.

Seriously, countries spy on countries and China sure is not innocent in hacking foreign countries.

</comment><comment><USER29> Because of all this bad press, the US will rally around a fake-libertarian or Republican in 2016. Rinse, wash and repeat.

<comment><USER30> I agree. It is always (somewhat) interesting to see who the elites place in the oval office every four years via near absolute control of the news media and a corporate owned congress, full of people who can switch talking points on demand.

</comment></comment><comment><USER31> Looks like he is bordering on treason. Be careful snowden.

<comment><USER19> "Treason against the United States, shall consist only in levying War against them, or in adhering to their Enemies, giving them Aid and Comfort. No Person shall be convicted of Treason unless on the Testimony of two Witnesses to the same overt Act, or on Confession in open Court."

Not that the US follows the constitution, but you know, in theory nothing he's done is remotely close to treason, and even if what he did qualifies, it might be difficult to get two Witnesses.

</comment></comment><comment><USER32> Accusing China of human rights violations and large scale cyber warfare is going to get more difficult in the future...

<comment><USER33> Why? Those accusations would be dependent on whether they were doing them or not. China has been "accusing" the U.S. of cyber warfare activities as well, and anyone who knew what the NSA was before 2013 knew that China was probably being truthful about that in general.

</comment></comment><comment><USER34> Is it legal for a foreign government such as the UK to spy on Americans and then hand that data to the US and vice versa? I bet they are doing that.

<comment><USER35> They are: [http://en.wikipedia.org/wiki/UKUSA_Agreement](http://en.wikipedia.org/wiki/UKUSA_Agreement)

</comment></comment><comment><USER36> Now he's just becoming a tool.

<comment><USER37> Like all those egyptian,iranian,chinese,russian,syrian ones?

</comment></comment><comment><USER38> What a disgusting circle-jerk of comments.

So far we have -

1\. Only China should be able to hack and spy on other nations.

2\. The USA should just leave everyone alone!

3\. Ohh, and this guy is not committing treason and espionage by releasing things like this, he's a freedom fighter. No more secrets!

Anyone else stopped reading HN these last couple of weeks?

I stopped right around when people were beginning to suggest that random government employees should be shot in protest, and as I looked as some of the posters names, I recognized a bunch of them.</comment>


Dear Mark Zuckerberg: Facebook Is an Engine of Anti-Muslim Hate. Don't You Care? - <AUTHOR> https://theintercept.com/2019/12/07/facebook-mark-zuckerberg-muslims-islamophobia/ <comment><USER2> It's also an engine of hate toward Christians, Donald Trump, an Republicans. Why should Muslims get special treatment?

<comment><USER3> Muslims in the US are a small, often despised, and occasionally persecuted minority. However, a majority of people in the US still identify as Christian, the US military is positively _infested_ with Christian dominionists, and in most of the US you haven't got a prayer of getting elected to public office if you don't at least pay lip service to Christianity.

<comment><USER2> "Muslims in the US are a small, often despised, and occasionally persecuted minority"

I still don't see your point. If your point is that someone in a majority can't be persecuted for their beliefs, you are wrong.

"most of the US you haven't got a prayer of getting elected to public office if you don't at least pay lip service to Christianity"

Really? Rashida Tlaib is just short of calling for the extermination of non- muslims and is in public office. There are also plenty of other muslims in government positions all around the US.

I also find it odd that you say this. If I was in government in a country that was predominately Muslim and didn't pay 'lip service' as you say TO Islam, I wouldn't last long.

Shouldn't government officials mostly represent the beliefs and wants of their constituents?

"infested"

The word you chose to use here isn't lost on me. IT shows your bias, which means I can't really take much of what you say seriously.

<comment><USER3> Deal the beam in your eye before concerning yourself with the mote in mine. You were the one whining about how Christians, Republicans, and Trump are also subject to hate.

But if you want to talk bias, let me put my cards on the table. I'm an atheist and an anarchist. I find Christianity and Islam equally loathsome, and since I live in the US I find Christians the greater threat simply because there are more of them and because statistics show that most domestic terrorism is perpetrated by white Christians who buy into far-right ideology.

I have no obligation to take what you say seriously, since you've made it plain that your sympathies lie with white supremacists, dominionists, and fascists.

<comment><USER2> "most domestic terrorism is perpetrated by white Christians who buy into far- right ideology"

This isn't true and has been debunked multiple times. A person that happens to be Christian and commits an act of terrorism is completely different than one that commits an act of terrorism in the name of Islam.

If you want to look at mass shootings, there are more gang-related mass shootings of 3 or more people than any other kind. This is conveniently left out to create a nice narrative that you describe.

Even if you left the above out, there are still very few white/Christian right-leaning acts of Domestic Terrorism. Much less per/capita, number of people killed, and overall totals than other groups.

If you pay attention to the press, if it is a white person that did the shooting, they are immediately demonized as a right-leaning white-nationalist before anybody knows the true facts.

This latest case of the navy base shooting? They need to investigate, even through there is plenty of evidence already of the contrary.

"sympathies lie with white supremacists, dominionists, and fascists"

Yeah, well, I deal in reality, not fantasy. White supremacists are a joke and few and far between.

I honestly have never met anyone in person like this. However, I've met plenty of hard-left antifa Fascists. Especially on college campuses that have no problem with violence.</comment></comment></comment></comment></comment>


The infamous "AI gaydar" study was repeated – couln't be reproduced - maze-le https://www.theregister.co.uk/2019/03/05/ai_gaydar/ <comment><AUTHOR> This title ('The infamous "AI gaydar" study was repeated – couln't be reproduced') is very incorrect; it does not correctly summarize either the linked article or the actual paper ([https://arxiv.org/pdf/1902.10739.pdf](https://arxiv.org/pdf/1902.10739.pdf)) that the article is discussing.

From the abstract on the actual paper:

> Using a new dataset of 20,910 photographs from dating websites, the ability > to predict sexual orientation is confirmed... While demonstrating that > dating profile images carry rich information about sexual orientation these > results leave open the question of how much is determined by facial > morphology and how much by differences in grooming, presentation and > lifestyle. The advent of new technology that is able to detect sexual > orientation in this way may have serious implications for the privacy and > safety of gay men and women.

So, the study was repeated, and _could_ be reproduced. However, it's still not clear if there's any merit to the facial morphology hypothesis, vs. the (less worrying) hypothesis that the models are just picking up on differences in grooming and presentation.

</comment><comment><USER2> This article mentions the words accuracy/accurate/accurately 10 times, e.g.

"when using his own dating-site-sourced dataset, was accurate at predicting the sexuality of males with 68 per cent accuracy – better than a coin flip"

But accuracy seems like a poor measure for something like this, when the population is highly unbalanced. It's trivial to create a classifier with high accuracy: just outputting 'heterosexual' every time would yield ~90% accuracy on faces of the general population.

<comment><USER3> This is true, which is why machine learning has long since learned to not even think of what you describe as a meaningful measure of accuracy. If you look at the linked paper [0], you'll find that the author uses the "ROC AUC" metric [1]:

>The ROC AUC score represents the probability that when given one randomly chosen positive instance and one randomly chosen negative instance, the classifier will correctly identify the positive instance

[0] [https://arxiv.org/pdf/1902.10739.pdf](https://arxiv.org/pdf/1902.10739.pdf) [1] [https://en.wikipedia.org/wiki/Receiver_operating_characteris...](https://en.wikipedia.org/wiki/Receiver_operating_characteristic)

<comment><USER2> Thanks. That makes more sense.

The article didn't mention AUC, so I assumed they were talking about accuracy in the sense people normally mean it, which also matches the definition in the sidebar of the wikipedia link you shared:

(TP + TN) / (P + N)

</comment></comment></comment><comment><USER4> People don't usually look at others completely naked and shaved and devoid of any other things like fashion and style. Why is this not allowed to be used by the AI?

</comment><comment><USER5> What's really astonishing about this is how opaque the behavior of the CNN is. We have no idea why it is doing what it is doing, and multiple papers and orthogonal investigations are required to figure it out.

</comment><comment><USER6> I'd be interested to see a class activation map to see if there's anything in it, probably not...

<comment><USER7> "class activation map" is a new term to me, so I looked it up very briefly.

Does it mean simply "what part of the image (or in aggregate, images) carried the strongest signal for categorization"? Or can it also tell you something about the feature set?

Would you suggest and good introductory example to the concept? This was the best I found in my search [https://jacobgil.github.io/deeplearning/class- activation-map...](https://jacobgil.github.io/deeplearning/class-activation- maps)

<comment><AUTHOR> From the original paper ([http://cnnlocalization.csail.mit.edu/Zhou_Learning_Deep_Feat...](http://cnnlocalization.csail.mit.edu/Zhou_Learning_Deep_Features_CVPR_2016_paper.pdf)):

"A class activation map for a particular category indicates the discriminative image regions used by the CNN to identify that category"

So, it's mostly what you said, but the maps you get are specific to the particular trained network you're using; the class activation map tells you, for a given network, for a given image, for a given class, which image regions did that network consider relevant to that class.

For learning more about interpreting neural nets I would highly recommend the posts on distill.pub; [https://distill.pub/2018/building- blocks/](https://distill.pub/2018/building-blocks/) and [https://distill.pub/2019/activation- atlas/](https://distill.pub/2019/activation-atlas/) are probably most pertinent.

</comment></comment></comment><comment><USER8> I assume in both cases the dataset was balanced. Otherwise metrics such as accuracy can be meaningless.</comment>


RollFWD A rolling prediction list of future headlines. What's yours? - <AUTHOR>

We're keeping a rolling list of headline predictions, for a reason yet unknown.
Future Headlines as of 03/10/13:
1. Wyoming Joins Texas as Second State to Issue Own Currency In Wake of U.S. Dollar Collapse- matterhorn
2. Following Dennis Rodman Visit, Switzerland Declares War on U.S.- matterhorn
3. Bloomberg Bans Bottled Water: Claims It Robs Our "Precious Bodily Fluids"- matterhorn
4. Patent Troll Claims Rights to Inhalation of Air- matterhorn
5. Demands Licensing Fees From All Breathers- sixQuarks
6. 155 year old sentenced to juvenile detention for tampering with town's singularity machine- devonbarret
7. Outrage as pedestrian killed by self driving car- devonbarret
8. 90% of car accidents caused by self driving system errors.- snowfox
9. Google glass banned in McDonalds- smewpy. <comment><USER2> Scientific study shows that highly developed neocortex in software developers is still responding to reptilian complex's commands. Hence, dark clouds in sight. (Inspired by cloudout's headline)

</comment><comment><USER3> Number 6 is not mine.

Nitpicking: My name has another t on the end.

Another headline: Infra-hats become number one selling product of the year.

</comment><comment><AUTHOR> Dark Clouds; internet becomes more vulnerable as 80% of all data now hosted in 20% of the world's data centers.

</comment><comment><USER4> It is now illegal to marry your clone.</comment>


Encrypted Backups to S3 for GitHub Enterprise - <AUTHOR> https://github.com/trailofbits/ghe-backup

<comment><AUTHOR> Feedback welcome! Interested to hear how others do GHE backups.</comment>